Infrastructure tools refactoring

• Preface
  • tl;dr
  • How to read this document
  • Why I created this document
  • What problem are we trying to solve
  • Analysis of current state
• Proposed structure of new tooling
• Plan of work
• FAQ
  • Why terraform?
  • How can I help?
• todos
• Global Reference

---

[todo(@bartsmykla)]: Remember to write section with suggestion to rewrite k/k8s.io/groups to custom terraform provider

Preface

I strongly believe in rule that before you'll start solving the problem with the code you should understand well what is the problem and solve it conceptualy first.

[IMPORTANT!] As you can assume, I've spent some time working on this document. I come to some conclusions and have some thoughts and ideas but they are just MY conclusions and just MY foughts and ideas. PLEASE don't feel obliged to agree with me just because I spent my time working on it. If you think I'm wrong or/and you think we could improve it PLEASE write about it. Even if it would completely scrap my ideas/suggestions/plans. Let's treat it as a working document and have fruitful discussion.

tl;dr

How to read this document

Why I created this document

What problem are we trying to solve

Analysis of current state

One could ask why this much effort to analyse all this components by hand and not to just use audit and take them drom there. This is a great question and the answer is I wanted to compare if all components from audit are being created by our tooling.

What I don't like is there are places like CIP Auditor which need some scripts being run in a specified sequence for it to be able to correctly provision resources. We need to make it more atomic.

• CIP Auditor
  • Preface
  • CIP Auditor Env
    • Terraform resources for CIP Auditor Env
    • CIP Auditor Env / Components
    • Yaml representation of CIP Auditor Env Components
  • CIP Auditor Deploy
    • Terraform resources for CIP Auditor Deploy
    • CIP Auditor Deploy / Components
    • Yaml representation of CIP Auditor Deploy Components
  • Reference
• Conformance Storage
  • Terraform resources
  • Variables used in Components
    • [BUCKETS]
  • Components
  • Yaml representation of Components
• GSuite
  • Terraform resources
  • Components
  • Yaml representation of Components
  • Reference
• Main Project
  • Flow Chart
  • Terraform data sources and resources
  • Components
  • Yaml representation of Components
• Namespaces
  • Terraform resources
  • Variables used in Components
    • [PROJECTS]
  • Components
  • Yaml representation of Components
• Prod Storage
  • Flow Chart
  • Terraform resources
  • Variables used in Components
  • Components
  • Yaml representation of Components
  • Notes
  • Reference
• Prod Storage GCLB
  • Terraform resources
  • Prod Storage GCLB / Components
  • Yaml representation of Components
  • Reference
• Release Projects
  • Terraform resources
  • Variables used in Components
    • [PROJECTS]
  • Components
  • Yaml representation of Components
  • Reference
• Releng
  • Terraform resources
  • Components
  • Yaml representation of Components
• Staging Storage
  • Flow Chart
  • Terraform resources
  • Variables used in Components
    • [PROJECTS]
    • [RS_PROJECTS]:
  • Components
  • Yaml representation of Components

---

Proposed structure of new tooling

Plan of work

FAQ

Why terraform?

How can I help?

todos

• @listx:

  • [todo(@listx)]: I don't undestand why we first create dummy Cloud Run Service in ensure-env-cip-auditor.sh and then overwriting it in cip-auditor/deploy.sh. What stands behind the decision to do it that way? Can we eliminate the step for the dummy service?

  • [todo(@listx)]: Do we really need the [PROJECT] to be a variable? Is CIP Auditor currently deployed for other than k8s-artifacts-prod projects too?

Global Reference

• ^G1 The name of components and structure of Yaml was used in conjuction to terraform components
• ^G2 Remember the IAM Binding resources used are authoritative (they will remove all other IAM Bindings from the resource) and are used here just as an example (they are easier to read). When components needed in our infrastructure will be described with different archetypes in mind ("project" will be an archetype instead of i.e. "prod storage") we should write terraform modules in that maner, that by default archetypes will get some IAMs granted by authoritative IAM Bindings and if archetype needs unusuall ones they will by granted by IAM Member resources which are non-authoritative. (for more information you can look for example at notes of Terraform's Google Storage Bucket IAM resource documentation.