NewClient doesn't properly report client connection errors due to invalid headers

[jlewi]

I called NewClient with invalid headers in the configuration. Specifically, I set the OpenAPIKey header but the value was invalid because it had newlines in it. This results in an invalid request and thus NewClient times out waiting for Weaviate to be ready. However, the error isn't properly reported; the only error we get is a timeout waiting for weaviate. As a result, its extremely difficult to root cause the failure.

Here are more details:

NewClient ends up timing out trying to establish a connection. Here

weaviate-go-client/weaviate/weaviate_client.go

Line 116 in 6eaaf85

if err := con.WaitForWeaviate(config.StartupTimeout); err != nil {

If we step into code we see the REST request made here is failing with an error

weaviate-go-client/weaviate/connection/rest.go

Line 79 in 6eaaf85

var isReady bool

The error is
"net/http: invalid header field value for "X-Openai-Api-Key""

So the error message indicates the root cause. The response is also nil. I think this combination implies that the request never gets made and thus its a permanent error and retrying is pointless.

When the retry eventually times out the only error that's returned is that it timed out

weaviate-go-client/weaviate/connection/rest.go

Line 95 in 6eaaf85

return fmt.Errorf("weaviate did not start up in %s. Either the Weaviate URL %q is wrong or Weaviate did not start up in the interval given in 'startupTimeout'", startupTimeout.String(), con.basePath)

So the root cause isn't reported. This makes it extremely hard to debug.

I suggest two fixes

1. Errors should be classified as permanent or retryable; permanent errors should not be retried as that just slows things down
2. In the event of a timeout; return the most recent error to help identify the issue