- Securely Automating Infrastructure in the Cloud
- All about me...
- In the industry since 1994
- Currently Sr. Architect & Director, Managed Services, Olson Digital.
- Find me on twitter: @sj_sadowski
- Or linkedin: http://www.linkedin.com/in/sjsadowski
- Or email me: [email protected]
- Or email me at my less than professional address: [email protected]
- Expectations
- I hope you will all walk away saying "well, I already knew that," because it means you already have at least the knowledge
- For those who don't, I my hope is that there are at least no surprises.
- This is not a "how-to" guide so much as a "how we did" guide
- Everything revolves around the idea of "minimum necessary access" and "do it securely or don't do it"
- How did we start?
- Worked to understand our security space
- Knowing how we could manage our infrastructure was key
- Further, understanding our automation tools from the bottom up
- What we knew we needed: auditable, repeatable process that integrated as fully as possible in to our environment
- What could we keep, what could we throw away - and what did we need that was new?
- Tools we had: Configuration Management (Chef11), Monitoring (Zabbix), Centralised logging (ELK - single node), Alerting (PagerDuty)
- Tools we needed: Better monitoring, better logging, infrastructure automation, better configuration management, auditing, code management
- Tools we picked:
- Git for SCM/GitLab for UI & Access Management
- Jenkins for CI/CD
- Terraform & Packer for IaC
- InSpec for configuration testing
- Chef12 for Configuration Management
- ELK - full HA cluster for logging
- Sensu for Monitoring
- PagerDuty for Alerting
- What about the details?
- Treat every engineer like a developer
- Treat every object in the infrastructure like code
- That's great, but how?
- Every environment starts with a new group in GitLab and projects initialized with default configurations and build pipelines
- Environments are then configured with the appropriate access credentials for the provider; AWS is currently our only true first class citizen, with Azure rapidly maturing. Second class citizens are currently RackspaceCloud, GCE, and On-prem we also support vSphere and OpenStack, but both have tertiary support
- Infrastructure is defined first (terraform), then created, bootstrapped, and tested with InSpec.
- Instances are defined with appropriate configurations in config management (roles in Chef) that are tied to appropriate checks (Sensu) and logging (ELK)
- Finally, all instances are registered for periodic scanning by our security team
- Takeaways: all of our communications are locked down, https where required, ssh by default.
- Defined security policy - with backing of our security team
- Any exceptions related to client-environment for the clients have to be signed off on, my team allows no internal exceptions. "Lock it down or turn it off"
- Again: Minimum necessary access: devs have access to development environments only, all other changes/deployments handled through Jenkins for infrastructure, or other defined build tool for code/applications
- Q: Where's the security? it's only explicitly mentioned once!
- A: It's inherent in every step.
- Base images built to a hardening standard
- Machines are scanned for compliance along the build pipelines
- Communication is secured (TLS/SSH/SCP)
- Keys are encrypted (GPG) with passwords stored separately
- Data objects for configuration management are encrypted
- So... why not tool X? It does A, B, C better...
- Tools that had the best internal support (KB, understanding)
- Tools that were most compatible with other parts of the org
- Tools that had the best trade-offs
- Represented the best overall needs for OUR org
- My advice for you: do the same. Find your tools, and run with them
- What are some of our problems?
- Lack of client buy-in
- Developer demands
- Signal v. Noise (Monitoring/Alerting)
- Knowledge sharing
- THE FUTURE (none of which is technical... really)
- Better education for both sales org and clients
- Reduction of noise by tuning our alerts
- Better compliance and inspection