Blockers and Issues from CX Team

[tomhuang12]

This discussion thread is to collect use cases and the issues the CX team has run into.

[steve-fraser]

Dependencies Review for Helm

I believe Helm dependencies are for charts that need to be together in order to work, but not a hard requirement to install

Helm Installation Ordering

Helm does not do ordered installation of dependent charts currently, instead it re-orders according to Kubernetes resource KIND. Please review to this documentation for a more detailed explanation: https://stackoverflow.com/questions/51957676/helm-install-in-certain-order.

I have validated this by a chart with these dependencies

servicemesh chart.yaml

dependencies:
- name: cert-manager
  version: "1.6.0"
  repository: "https://charts.jetstack.io"
  condition: cert-manager.enabled
- name: linkerd2
  version: "2.11.0"
  repository: "https://helm.linkerd.io/stable"

example chart.yaml

dependencies:
- name: servicemesh
  version: "0.0.1"
  repository: "file://../servicemesh"
- name: kube-prometheus-stack
  version: "19.2.2"
  repository: "https://prometheus-community.github.io/helm-charts"
  condition: kube-prometheus-stack.enabled

The following values:

kube-prometheus-stack:
  enabled: true
  prometheus:
    annotations:
      linkerd.io/inject: enabled
  grafana:
    podAnnotations:
      linkerd.io/inject: enabled

upgrade/install log

client.go:501: [debug] Looks like there are no changes for Service "linkerd-proxy-injector"
client.go:501: [debug] Looks like there are no changes for DaemonSet "observ-example-prometheus-node-exporter"
**client.go:501: [debug] Looks like there are no changes for Deployment "observ-example-kube-promet-operator"**
**client.go:501: [debug] Looks like there are no changes for Deployment "observ-example-cert-manager-cainjector"**
client.go:501: [debug] Looks like there are no changes for Deployment "observ-example-cert-manager"
client.go:501: [debug] Looks like there are no changes for Deployment "observ-example-cert-manager-webhook"

This resulted in the ordering of resources as described in the above StackOverflow post all LIKE resources will be installed together. DeamonSets will be installed first then deployments, but will NOT validate that daemon sets are up and healthy. Please refer to here for the current order: https://github.com/helm/helm/blob/eeac83883cb4014fe60267ec6373570374ce770b/pkg/releaseutil/kind_sorter.go#L31

Custom Resource Definition Example/Issue

trying to install a certificate with a dependency of cert-manager:

cert-manager: 
  enabled: true
  installCRDs: true

dependencies:
- name: cert-manager
  version: "1.6.0"
  repository: "https://charts.jetstack.io"
  condition: cert-manager.enabled

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: linkerd-trust-anchor
  namespace: linkerd
spec:
  ca:
    secretName: linkerd-trust-anchor

Error:

Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: no matches for kind "Issuer" in version "cert-manager.io/v1"]

This condition fails because the helm cli prepares and checks installations KINDs before installed, in which case the cert-manager chart is not actually installed

Workaround:
Helm v3 offers the ability to create a crds folder that you can manually manage the CRDs
https://helm.sh/docs/topics/charts/#custom-resource-definitions-crds

Values.yaml

cert-manager: 
  enabled: true
  installCRDs: false

Chart.yaml

dependencies:
- name: cert-manager
  version: "1.6.0"
  repository: "https://charts.jetstack.io"
  condition: cert-manager.enabled

crds/cert-manager-crds.yaml

# Source: cert-manager/templates/templates.out
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: issuer.cert-manager.io

template/issuer.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: linkerd-trust-anchor
  namespace: linkerd
spec:
  ca:
    secretName: linkerd-trust-anchor

While this workaround does function, we now have a condition where we have to disable CRDs installed in one location while also enabling them in another. This could be very confusing to the end-user. This solution also requires a fork of the CRDs.

Using Helm post-render functionality could elevate this use-case by applying the custom objects post-deployment, but would be a requirement

[tomhuang12]

@darrylweaver I was thinking about the same thing, having HelmReleases and Kustomizations in a chart and utilize dependsOn feature to resolve dependency issues. In that case, we are recreating the original ProfileDefinition functionalities with helm chart. I will give it a try.

[tomhuang12]

Okay I did various experiments. it is nearly impossible to achieve the same result as the servicemesh chart using HelmRelease and Kustomizations without also adding the Linkerd Issuer, Certificate, and secret yaml in the gitops repo as well. I was trying to use Kustomization CR to create those resources using kustomize patch feature but it requires the files to exist to be patched on. That would break the pattern we are trying to achieve with profiles, having a single packaged solution that can be deployed and usable using one operation. The original ProfileDefinition and pctl would take care of this nicely.

IMO, we should keep the original ProfileDefinition and pctl to generate the manifests and leveraging dependsOn feature in HR and KS CRDs. Helm's dependency functionality isn't granular enough.

[darrylweaver]

Thanks for the analysis, that's really helpful.

See the fidelity project, Kraan: https://github.com/fidelity/kraan
This is a project that dealt with the dependencies by implementing layers, which might be a model we could use.

[bigkevmcd]

@tomhuang12 Can you say a bit more about what you tried, and why it doesn't work?

[tomhuang12]

This is what I am testing with: https://github.com/weaveworks/profiles-catalog/tree/servicemeshv2/charts/servicemeshv2. This chart doesn't work due to KS expecting files in the git repo but we don't want to add the files separately as that breaks the pattern. Steve's example works but requires a fork of the CRDs that we need to manage which we should avoid.

[steve-fraser]

helm charts that use namespaces as values issues

I want to open the discussion about how both Istio and Linkerd handle namespace values in chart.

Linkerd has a value in the chart called namespace

dependencies:
- name: linkerd2
  version: "2.11.0"
  repository: "https://helm.linkerd.io/stable"

linkerd2:
  namespace: linkerd

Same thing is expected for the istio chart

global:
    istioNamespace: default

If the namespace set in the chart values linkerd do not exist it breaks the installation because it expects those to exist.

log

Error: INSTALLATION FAILED: namespaces "linkerd" not found
helm.go:88: [debug] namespaces "linkerd" not found
INSTALLATION FAILED
main.newInstallCmd.func2
        helm.sh/helm/v3/cmd/helm/install.go:127
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:974
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:902
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:87
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_amd64.s:1581

Note: the --create-namespace parameter in helm install does not fix this because the namespace is a value

Linkerd and Istio not following the practice of using the release namespace could be at fault here.

Current list of hem charts that if this issue

• Linkerd
• Istio