Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.7/edge bundle cannot be deployed due to the circular dependency #605

Closed
beliaev-maksim opened this issue May 24, 2023 · 6 comments
Closed
Labels
bug Something isn't working Kubeflow 1.8 This issue affects the Charmed Kubeflow 1.8 release

Comments

@beliaev-maksim
Copy link
Member

running tests for bundle discovered a bug with circular dependency
https://github.com/canonical/bundle-kubeflow/actions/runs/5063062459/jobs/9102852996

istio gateway waits for istio pilot
but istio pilot waits for relation data from OIDC

but to configure OIDC I need istio gateway URL

 istio-ingressgateway/0*       waiting      idle   10.1.245.18                     Waiting for istio-pilot relation data, deferring event
istio-pilot/0*                waiting      idle   10.1.245.16                     List of <ops.model.Relation ingress:9> versions not found for apps: oidc-gatekeeper
oidc-gatekeeper/0*            blocked      idle   10.1.245.50                     public-url config required

but in order to set public URL on OIDC we might need to parse it from istio-ingressgateway

public_url = f"http://{gateway_svc.status.loadBalancer.ingress[0].ip}.nip.io"
@NohaIhab NohaIhab added the bug Something isn't working label May 29, 2023
@NohaIhab
Copy link
Contributor

This change was introduced in canonical/oidc-gatekeeper-operator#78, however, it should not be in 1.7/edge rather latest/edge. @beliaev-maksim Is this deployed with 1.7/edge not latest?

@ca-scribner
Copy link
Contributor

@beliaev-maksim I think you mentioned a check in the OIDC charm rewrite that you re--enabled and caused this problem, am I remembering correctly? Can you point to this in the code?

@beliaev-maksim
Copy link
Member Author

It is both edges

@ca-scribner
Copy link
Contributor

I see a manual release of the charm in main to 1.7/edge but if we want to update ch:1.7/edge we should update gh:track/1.7 and CI will publish the charm to edge. I've reverted ch:1.7/edge to match the gh:track/1.7 code by running this publish action

@NohaIhab NohaIhab added the Kubeflow 1.8 This issue affects the Charmed Kubeflow 1.8 release label Jul 12, 2023
@NohaIhab
Copy link
Contributor

I was not able to reproduce this with the following channels:
dex-auth 2.31/edge
istio-ingressgateway 1.16/edge
istio-pilot 1.16/edge
oidc-gatekeeper latest/edge

istio gateway is not waiting for istio pilot, status:

App                   Version  Status   Scale  Charm            Channel      Rev  Address         Exposed  Message
dex-auth                       active       1  dex-auth         2.31/edge    224  10.152.183.117  no
istio-ingressgateway           active       1  istio-gateway    1.16/edge    551  10.152.183.203  no
istio-pilot                    waiting      1  istio-pilot      1.16/edge    551  10.152.183.198  no       installing agent
oidc-gatekeeper                waiting      1  oidc-gatekeeper  latest/edge  233  10.152.183.215  no       installing agent

Unit                     Workload  Agent  Address      Ports  Message
dex-auth/0*              active    idle   10.1.205.76
istio-ingressgateway/0*  active    idle   10.1.205.78
istio-pilot/0*           waiting   idle   10.1.205.74         Execution handled 2 errors.  See logs for details.
oidc-gatekeeper/0*       blocked   idle   10.1.205.77         public-url config required

and the istio gateway service was created, I can get the IP with:

kubectl get svc istio-ingressgateway-workload -nkubeflow -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
10.64.140.43

from my understanding of the istio charms, istio pilot should send the data to istio gateway, without waiting for the ingress-auth relation data from oidc.
cc: @ca-scribner

@NohaIhab
Copy link
Contributor

NohaIhab commented Aug 4, 2023

tested with latest/edge bundle, did not hit this issue. Closing as the bug is currently not in any of charmed kubeflow bundle's channels.

@NohaIhab NohaIhab closed this as completed Aug 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Kubeflow 1.8 This issue affects the Charmed Kubeflow 1.8 release
Projects
None yet
Development

No branches or pull requests

3 participants