Cannot login when OAuth endpoint is using a customCA

[X-dark]

Describe the bug
I get a 500 error about certificate validation after authenticating on the oauth endpoint. I have tried to inject the CA bundle using [1] but the system CA seems to be ignored.

I suspect this is related to this: jupyterhub/oauthenticator#411 but I would be happy to get feedback on my investigation.

[1] https://docs.openshift.com/container-platform/4.6/networking/configuring-a-custom-pki.html#certificate-injection-using-operators_configuring-a-custom-pki

[vpavlin]

Based on the linked issue it seems there might be 2 issues

1. The self-signed certificates might be handled incorrectly by default tornado backend, which should be fixed by [OpenShift] add allowed_groups and admin_groups jupyterhub/oauthenticator#410 when merged and updated in this repo
2. IIUC your issue the problem is that we use the serviceaccount/ca.cert by default, which causes the injected custom CA to be ignored. Correct?

[X-dark]

1. From what I understood, tornado is using CA bundle from certifi. I have tried to overwrite that bundle (/opt/app-root/lib/python3.6/site-packages/certifi/cacert.pem with no impact). I am not sure to understand the impact of the PR you linked on my topic but I will be happy to try when merged.
2. Yes. That file contains the kube-apiserver and the ingress certificate (used by the oauth endpoint) but not the CA chain of the ingress certificate

[wseaton]

@X-dark For context, I added a conditional import to that PR that defaults to the pycurl powered tornado backend if available, which should bypass certifi entirely as under the hood it uses libcurl. At that point it should be much more straightforward to debug cert issues, and for me that fixed issues with the self-signed certs I was seeing (where they 'just worked' with standard curl). In regards to a custom PKI bundle I'm not sure what the impact would be for this change.

There's still some work that happens to get pycurl compiled for RHEL's UBI into the hub images, but I think @vpavlin and team are working on that.

[X-dark]

Yes I think this could be great. pycurl should use default system CA certificates and using the certificate injection mechanism it is easy to inject custom CA bundle at that level.

[akchinSTC]

Hi All,
I just installed the latest operator (v1.0) on openshift (v4.5) on IBM Cloud with the latest tagged release of the odh-manifests (v1.0.3).
On startup of jupyterhub, after accepting the permissions, Im greeted with the following:
instead of the spawner page, which I assume is the issue at large here, or hopefully related? At this time, is there a recommended workaround for this problem?
Thanks in advance!

[nobusue]

I got into same situation with @akchinSTC
OpenShift 4.7.2 on AWS
ODH operator 1.0.0
odh-manifests v1.0.0

[X-dark]

@wseaton so I see your PR has been merged 3 months ago but the latest odh-jupyterhub still uses a commit from Feb 10 of oauthenticator. Is there anything specific blocking this?

[wseaton]

@X-dark are you still seeing this issue in the latest Z-stream version of odh-manifests? odh-jupyterhub was configured to vendorize the fix in a fork, so the issue should be resolved AFAIK.

(cc @vpavlin)

[X-dark]

I am using 0.2.8 image of odh-jupyterhub from quay with manifests from latest HEAD from https://github.com/opendatahub-io/odh-manifests. So I guess I should be up to date. I am still seeing HTTP 599: Peer's Certificate issuer is not recognized from tornado.curl_httpclient.CurlError with CA injected into /etc/pki/ca-trust/extracted/pem.

[lucianolacurcia]

I am having the same problem, any workaround? I am using image 0.3.5, getting: " tornado.curl_httpclient.CurlError: HTTP 599: Peer's Certificate issuer is not recognized."

[shalberd]

#137 might offer some hints. There, the same type of configmap injection mechanism was used by me, then the configmap data section referenced in a volume and added to a spawned container volume mount programmatically. Unsure how this works here with Tornado, but still, might be worth a look in general.