Implement atmos packages to install 3rd party tools

[osterman]

Describe the Feature

Atmos should automatically add the packages base-path as a priority search PATH when executing any commands.

Expected Behavior

Packages are automatically installed and available to atmos.

Use Case

• Install versions of opentofu or terraform

• Install helmfile

• Install terraform-docs, tflint, etc.

• Support multiple concurrent versions

• Install any other binaries needed by workflows, custom commands, etc

Describe Ideal Solution

Atmos Commands

# Install all packages
atmos packages install

# Update packages 
atmos packages update

Atmos Configuration

# atmos.yaml
settings:
  packages:
    base-path: ./packages
    bin-path: ./packages/bin

# packages/
registry:
  #components: 
  # for examle only...

  # Configure the packages registries supported by Aqua
  packages:
  - name: aqua
    type: standard
    ref: v4.246.0 # renovate: depName=aquaproj/aqua-registry
  
  - name: foo
    type: github_content
    repo_owner: suzuki-shunsuke
    repo_name: private-aqua-registry
    ref: v0.1.0
    path: registry.yaml


# Define global packages not specific to any type component
# Use for subcommands
packages:
- name: hashicorp/[email protected]
- name: hashicorp/terraform
  version: v0.13.7
  command_aliases:
    - command: terraform
      alias: terraform-013


# Define packages required for a specific type of component
components:
  terraform:
    needs:
    - hashicorp/[email protected]
    - hashicorp/[email protected]

Stack Configuration

# stacks/_defaults.yaml
components:
  terraform:
    vpc:
      needs:
      - hashicorp/[email protected]

Alternatives Considered

No response

Additional Context

See https://github.com/suzuki-shunsuke/tfaction for example implementation

[suzuki-shunsuke]

There are several options:

1. Use a version manager like aqua
2. Run a version manager like aqua internally
3. Call a version manager like aqua as a Go Module
4. Develop the feature from scratch

The option 1 is simplest and easiest.
We don't need to develop anything.
We only need to encourage users to use a version manager.
But the drawback is that users need to install tools themselves.
By enabling atmos to manage tools, users don't need to install tools themselves.

The option 2 is not bad.
I think we should consider this option first.
I'm the author of aqua, which is a CLI version manager, so I use aqua here.
atmos can generate configuration files of aqua such as aqua.yaml, registry.yaml, aqua-checksums.json, and aqua-policy.yaml internally.
And atmos can execute aqua install internally to install tools.
By adding $(aqua root-dir)/bin to $PATH internally, users don't need to be aware of aqua.
aqua also provides a update command.
aqua provides Renovate config presets. We can utilize them or develop similar presets for Atmos.

This is an advanced topic, but one of problems is checksum verification in CI.
Checksum verification is a feature of aqua.
This feature is very important for security.
aqua recommends managing aqua-checksums.json by Git, otherwise checksums can be tampered.
If atmos hides aqua and doesn't manage aqua-checksums.json by Git, the checksum verification doesn't work enough.
But I'm aware that managing aqua-checksums.json by Git is a bit bothersome because we need to update checksums when we update tools.
Especially, when we update tools by Renovate, we also need to update checksums.
aqua provides a GitHub Action to update checksums automatically, but this requires a GitHub App.

The option 3 is good, but I guess most version managers are not Go Modules but CLI.
aqua is written in Go, and it doesn't use Go's internal packages, which means all packages are public.
But aqua doesn't intend that other tools depend on aqua's packages.
API isn't third party friendly, and API can be changed drastically, so I don't recommend using aqua as a library.

I don't recommend the option 4.
We should utilize existing tools rather than re-inventing the wheel.
We need to maintain something like aqua's registries ourselves.
aqua provides various features.
If other options isn't acceptable, we need to consider this option.

What do you think?

[osterman]

@suzuki-shunsuke

1. Use a version manager like aqua

My main concern is this: it defeats the purpose of the desired functionality, which is to avoid the need to manually install additional tools—including the tooling required to do so (besides Atmos itself). The goal is for everything to "just work" out of the box. I feel so strongly about this that I’d rather hold off until we find a solution that aligns with this principle.

2. Run a version manager like aqua internally

Same response as (1).

3. Call a version manager like aqua as a Go Module

Thank you for clarifying this point!

This was definitely my preferred approach. I was hoping this approach could work, as I noticed that the packages were not strictly internal.

However, I appreciate your warning about the unstable interface and its potential for changes without notice. This raises concerns and gives me pause. Let's see if there's a middle ground (see option 5).

4. Develop the feature from scratch

I would prefer to rule this out as well.

---

I’d like to propose two additional options based on your feedback.

Option 5: Call Aqua (Preferred)

We have an upcoming PR that introduces support for editorconfig.

• Add editorconfig validation #896

What’s notable about this implementation is that while many editorconfig packages are internal, we were able to call their public methods, achieving a near-native integration within Atmos.

What if we did something similar with Aqua?

In this model, if Aqua could expose a few stable APIs that provide command-line-like functionality from within Go, Atmos could call those directly. This way there's not a lot of work to maintain in Aqua, while providing other tooling the ability to leverage aqua. While this may sacrifice full flexibility, it offers the advantage of maintaining a native user experience within Atmos.

As part of this if we could programmatically pass a string containing aqua configuration, we can then work very well with parts of aqua. We could generate that configuration inside of atmos, so the package manafement can feel more integrated with things like stacks, workflows and custom commands.

Option 6: Go-based Aqua Bootstrapper (Less Preferred)

This is a riff on your #2.

If Aqua provided a method to bootstrap itself programmatically via Go, Atmos could invoke that as part of its process. This approach would enable similar functionality, but with a few limitations. Specifically, we’d be constrained to using Aqua’s configuration files without the ability to embed or extend those configurations directly within Atmos.

This limitation concerns me because we’re introducing radical improvements to Atmos configuration handling in an upcoming PR that adds support for remote configurations, local overrides, imports, and more. All of that provides a consistent configuration interface for atmos, including the underlying tooling. I’d like those improvements to be universally applicable across Atmos functionality.

• Supported the atmos.d, allowing automatic inclusion of configuration files from the atmos.d directory #808