Support App of Apps pattern for applications are generated dynamically rather than stored directly in Git

[dudell-bud]

We use the App of App pattern quite a lot within our environments - so we have an App that generates apps that generate our kube manifests. It would be great if the it could work out that the rendered manifests for an App were Applications and then also grab the configurations for those applications as part of the generated diffs.

At the moment to get around this I have to run the diff preview twice, using the output of base and target as the input for the second run - or try and generate that middleware set of Apps externally (however some of the differences I want to expose are at this layer).

[dag-andersen]

Hi @dudell-bud,

We use the App of Apps pattern at my workplace as well. However, all applications are stored in Git and not generated - So we do not have this problem.

Could you tell me a bit more about how those apps are generated? Perhaps you could share a code example? Depending on your needs, I might explore different solutions for implementation.

At the moment to get around this I have to run the diff preview twice, using the output of base and target as the input for the second run - or try and generate that middleware set of Apps externally (however some of the differences I want to expose are at this layer).

I am impressed you figured out how to do that since it is not documented 👍🏻

[sergeyshevch]

Recent changes with applicationset works great but in my case (multiple clusters, and multiple application) we use next scheme:

applicationSet -> root helm chart for each cluster -> multiple applications

Maybe it's possible to support app of apps in similar way as well?

For now I'm solving it with shell script like:

for envType in clusters/*; do
  if [ ! -d $envType ]; then
    continue
  fi

  envType=$(basename "$envType")
  for clusterName in clusters/$envType/*; do
    if [ ! -d $clusterName ]; then
      continue
    fi

    clusterName=$(basename $clusterName)
    echo "Processing $envType/$clusterName"
    helm template root root-helm-chart -f clusters/$envType/values.common.yaml  -f clusters/$envType/$clusterName/values.yaml --set clusterName=$clusterName --set envType=$envType >> target-branch/apps.yaml
  done
done

but it's not a perfect sollution

[dag-andersen]

@sergeyshevch

Thank you for showing interest in the project.

I like your solution to the problem.

I am not sure what ⬇️ means exactly :)

applicationSet -> root helm chart for each cluster -> multiple applications
Maybe it's possible to support app of apps in similar way as well?

How would you like the tool to work?

[sergeyshevch]

I guess when tool find kind:Application in generated manifests it should add it to Argo and template it

So it will work like a recursion

Same with appset. It will solve all issues around apps of apps

In my case I will just entirely remove my workaround

[dag-andersen]

@sergeyshevch

I agree. This would be a nice feature! 🚀

Over the weekend, I spent a few hours working on a solution. The main challenge is handling name collisions. If the generated Applications share a name with any other application, race conditions will occur, and the output will overwrite one another (since Kubernetes only allows a single resource with a given name).

Currently, I address this issue by appending a suffix to all applications with duplicate names, such as -1, -2, -3, and so on. I perform this as a one-time step before applying the applications to the cluster. However, dynamically managing this based on the generated applications is more complex.

The steps would look something like this:

1. Find all applications and ApplicationSets in the repository.
2. Patch the applications (redirect their targetRevision to the selected branch).
3. Ensure there are no duplicate names.
4. Generate applications from the ApplicationSets.
5. Patch the new applications.
6. Ensure there are no duplicate names (again).
7. Apply the applications to the cluster.
8. Extract the rendered manifests for each application:
   1. Find all new applications and ApplicationSets in the rendered manifests.
   2. Patch the new applications (again).
   3. Ensure there are no duplicate names.
   4. Generate applications from the ApplicationSets.
   5. Patch the new applications (again).
   6. Ensure there are no duplicate names.
   7. Apply the new applications.
9. Repeat the process until there are no nested applications.

(I probably missed a few steps.)

Currently, none of the substeps in 8. is implemented.

This process is quite challenging and will require many hours to implement.