-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial import of proposed infrastructure changes #80
Initial import of proposed infrastructure changes #80
Conversation
These changes are a proposal for how to manage infrastructure as code. The basic idea is as follows: 1. `env/` directory contains configuraiton specific to an environment. An 'environment' is any logical grouping of infrastructure that is specific to one particular use case, or location, or account, etc. Environments inherit configuration hierarchically. A component 'airflow' in a directory 'dev' in a directory 'sfo2' in a directory 'docean' will allow that 'airflow' component to inherit the configuration of those parent directories, so configuration never needs to be duplicated across environments. It also allows for simple perusal of configuration to find configuration values specific to an environment, or more global configuration. 2. A `terraform/` directory contains a `root` directory, where root modules are kept. These modules are passed `terraform.sh.tfvars` configuration files from an `env/`. The module is designed to absorb configuration and apply changes the same way in all accounts, with some logic embedded in the Terraform code to work around differences in different environments. In this way, the same root module can be used for multiple environments, keeping the code DRY. 3. A `terraform/` directory also contains a `sub` directory, which are sub-modules. Think of these as "libraries" of Terraform code. These are not always necessary, but sometimes the extra layer of abstraction is convenient. A well-designed root module is often enough for small projects without the need for sub-modules. 4. Purpose-built deployment scripts are used to pass the `env/` configuration to the components being deployed, which may live in other directories (for example the `terraform/` modules directories). In some cases, symlinking to an `env/` directory can prevent the need to write a custom deployment script. This proposal is not yet complete. Still to be completed: - Finish taking some existing Terraform modules and adapting them with new variables, and filling in the `env/` directories with new `.tfvars` files to carry the variables. - A guide to developing with this model - A guide to deploying with this model - Docker, GitHub Actions, and other code/configuration to execute deployment scripts using this model - Probably more... I'm done for the night :-)
See https://github.com/peterwwillis/treetracker-infrastructure/tree/reorganize-proposal-1/env/tt for the start of the See https://github.com/peterwwillis/treetracker-infrastructure/tree/reorganize-proposal-1/terraform/modules/root for the |
Added a bit more work. This PR has moved existing directories' files, which is conflicting with new changes (and will probably be way too big to merge all at once). Work needs to be done to restore the pre-existing files so as not to impact existing processes and infrastructure. Another change in the new configs here are things like new tfstate files based on an S3 prefix hierarchy to mirror that of the env/ directory. The existing tfstate files will need to be copied or moved to the new files as the new environments are imported/applied. Also, some consolidation or organization of S3 buckets is probably necessary. |
This PR is replaced by #82 |
These changes are a proposal for how to manage infrastructure as code.
The basic idea is as follows:
env/
directory contains configuraiton specific to an environment. An 'environment' is any logical grouping of infrastructure that is specific to one particular use case, or location, or account, etc. Environments inherit configuration hierarchically. A component 'airflow' in a directory 'dev' in a directory 'sfo2' in a directory 'docean' will allow that 'airflow' component to inherit the configuration of those parent directories, so configuration never needs to be duplicated across environments. It also allows for simple perusal of configuration to find configuration values specific to an environment, or more global configuration.A
terraform/
directory contains aroot
directory, where root modules are kept. These modules are passedterraform.sh.tfvars
configuration files from anenv/
. The module is designed to absorb configuration and apply changes the same way in all accounts, with some logic embedded in the Terraform code to work around differences in different environments. In this way, the same root module can be used for multiple environments, keeping the code DRY.A
terraform/
directory also contains asub
directory, which are sub-modules. Think of these as "libraries" of Terraform code. These are not always necessary, but sometimes the extra layer of abstraction is convenient. A well-designed root module is often enough for small projects without the need for sub-modules.Purpose-built deployment scripts are used to pass the
env/
configuration to the components being deployed, which may live in other directories for example theterraform/
modules directories). In some cases, symlinking to anenv/
directory can prevent the need to write a custom deployment script.This proposal is not yet complete. Still to be completed:
Finish taking some existing Terraform modules and adapting them with new variables, and filling in the
env/
directories with new.tfvars
files to carry the variables.A guide to developing with this model
A guide to deploying with this model
Docker, GitHub Actions, and other code/configuration to execute deployment scripts using this model
Probably more... I'm done for the night :-)