Understand the semantics of ssh client in case the command fails

[vamossagar12]

While working on : #1096 , the code tries to access the /healthz end point SSHing into the a master node:

perf-tests/clusterloader2/pkg/measurement/common/api_availability_measurement.go

Line 62 in d26909c

sshResult, err := measurementutil.SSH(command, host+":22", provider)

The semantics that we need to understand is what happens if the command fails internally with error 500 for the Curl Request. In that case, what corrective actions need to be taken?

Currently, only err and sshResult.Code is checked:

https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/pkg/measurement/common/scheduler_latency.go#L317-L322

However, in some of my local testing, it was found that when the Curl request for /healthz end point fails, then Stderr in SshResult would be populated. In case of a proper response, Stderr would be empty and Stdout would be populated. The findings have been noted here:

#1162 (comment)

The intention is to find the semantics to verify this behaviour and document it accordingly.

[vamossagar12]

@wojtek-t I did some more experiments on this and also posted a question on #kubernetes-dev slack channel. the response i got is more or less in line with what I had found previously:

1) if ssh itself failed, then you'll get an err!=nil response from SSH()
2) if curl failed you'll get the (non-zero) exit code in sshResult.Code
3) if the HTTP response was an error code (4xx or 5xx) then  (as written) you'll have no idea, unless you can guess based on the body of the HTTP response.

1 suggestion was to use -f option with curl. The man page mentions the following:

 -f, --fail
              (HTTP) Fail silently (no output at all) on server errors. This is mostly done to better enable scripts etc to better deal with failed attempts. In nor-
              mal  cases when an HTTP server fails to deliver a document, it returns an HTML document stating so (which often also describes why and more). This flag
              will prevent curl from outputting that and return error 22.

              This method is not fail-safe and there are occasions where non-successful response codes will slip through, especially when authentication is  involved
              (response codes 401 and 407).

I tested by throwing 500 error and accessing an invalid url in a web server running in a pod. on both occasions the error code returned is 22.

I believe this is better than checking stdout and stderr as we might see differing behaviour. And hopefully, response codes 401 and 407 shouldn't be something that we should be worrying about.

[wojtek-t]

Thanks for digging into it - it's really useful.

[vamossagar12]

Closing this issue.