-
Notifications
You must be signed in to change notification settings - Fork 4
Kubernetes 101
Also published to medium at https://medium.com/@wyvern8/kubernetes-vms-and-docker-101-4e4736239b38
TL;DR (or you prefer manga) https://cloud.google.com/kubernetes-engine/kubernetes-comic/
An accompanying slide deck: http://slides.com/andrewvaughan/intro-to-kubernetes#/
This is an opinionated view of the past, present and future of managed compute resources, and how Kubernetes is uniquely positioned to play a significant role in the realisation of the goals of the agile manifestos, in particular those related to challenges in the 'devops' space.
Note that the so called 'serverless' and event driven microservices architectures using AWS Lambda, SQS, Kinesis and similar services represent a view of where true cloud native architectures are headed, and some approaches addressed by container management now may be superceded or augmented by new features as both organisations and streaming frameworks mature.
As an example, it is possible to deploy serverless capabilities within k8s such as via https://github.com/kubeless/kubeless which may be a reasonable approach for certain types of Lambda deployments.
Either way, it is highly likely that enterprises will need to support monolithic applications into the foreseeable future alongside modern architectures, so thinking about ways to decompose these into manageable units of functionality incrementally - with a strategic architecture in mind - is going to be an ongoing focus for forward thinking organisations.
In the beginning, servers were glorified PCs used to share data, then:
- rooms of networked servers were used as data centres (and for friday night doom sessions)
- these rooms were outsourced to hosting providers
- hosting providers realised they could cut costs by sharing resources allocated to customers, as they were not always fully utilised.
- new generations of hosting providers leveraged virtualisation to provide unprecedented virtual network capabilities and availability levels and the 'cloud' was born.
- organisations started moving local/offshore DCs into the cloud.
Kubernetes is based on the ideas and learnings of the development of "Borg", the Google internal container management system used to serve google services for 10+ years.
Virtually everything google does runs on this platform or variants thereof.
Kubernetes was released as a new open source project to the community/industry to add value to the Docker ecosystem, by providing solutions to the challenges Google had already dealt with internally when managing containers at scale.
It has since grown into a comprehensive and flexible platform for container scheduling and deployment management.
Kubernetes has been criticised as being complex, particularly by PaaS vendors. The new concepts involved are there for reasons that may not be immediately apparent until more complex deployments are required. This quote comes to mind.
"as simple as possible but no simpler"
Rather than answering the 'what is k8s?' question directy, let's further summarise some of the history and concepts that led us here:
- what are virtual machines?
- what is docker? (and other container types)
- what are the problems that each try to solve?
On the way through, the aspects of k8s that are relevant to each area will be touched on, and this should give an overall view of why Kubernetes exists and how it stands to play an important part of platforms evolution.
Virtual machines are primarily a hardware abstraction layer and scheduling engine.
-
VMs allow software designed to run on a specific architecture or operating system to run on another operating system/platform,
- This is really useful for sharing servers and apps to others without having to provide detailed instructions to get things going, and alleviates the 'it works on my machine' frustration.
-
VMs also allow a single physical host to appear as many logical hosts.
- This is the key approach used by 'cloud' providers to sell units of compute time etc
These concepts have existed in *nix for many years, but it was when desktop and hosted server virtualization hit the mainstream that things really took off and paved the way to the cloud services we take for granted today.
There are two types of hypervisors for abstraction layers and resource scheduling:
- Type 1: runs directly on hardware wthout a host OS. Also referred to as native or 'bare metal'.
- eg. AWS custom hardware with minimal kernel and Xen/KVM running EC2 instances
- Type 2: runs on a host os.
- eg. Windows 10 running Virtualbox launching an Ubuntu VM.
Virtualization is the foundation of the 'cloud' provider compute resources we see today such as AWS EC2, Azure, GCE, Digital Ocean. These services generally now use Type 1 hypervisors such as Xen, KVM, or custom hypervisors, with the aim to minimise the impact of virtualization to the guest processes as much as possible.
In addition to compute virtualisation, these services compete via ancillary services such as network/security infrastructure, storage and specialized services such as vms optimized for machine learning.
Performance overheads are becoming less of an issue in modern hypervisors, particularly in cloud infrastructure, and VMs will continue to service important use cases both locally and in the cloud.
In the context of a large scale platform being able to execute packaged units of code directly and at scale, VMs present a number of challenges. They require:
- images including a full OS and virtualized hardware to be booted on demand
- services to be configured and started in a way that encourages stateful monolithic deployments
- a way to trigger an execution of internally deployed code from outside the VM
- a way for this execution to behave in the context of a broader orchestration.
In the case of a container, each unit of code:
- is ready to execute in the context of a broader deployment without custom wiring up of services - docker and k8s have conventions for this.
- makes use of the host kernel and does not require an environment to be 'booted'
Notice that these differences imply that containers work best as single responsibility microservices, and this encourages best practice design by making 12 factor design https://12factor.net/ the path of least resistance. Wins on all sides!
A side area to consider is that each vendor approach to virtualise hardware is different, often proprietary, and opaque in the case of cloud providers which each have their own vendor specific apis to manage platform resources.
VMs can also be huge to transfer once built/baked which is not a great developer experience.
There are open standards for local virualized hardware that will continue to play a role. https://www.hashicorp.com/ in particular created Packer and Vagrant for vm abstraction, and Terraform as a crossplatform DSL to describe cloud service resources. There are many challenges that are vendor specific and change over time that will continue to make this a difficult approach.
Docker is currently the defacto standard for 'containers' although k8s is not locked to this and also supports the competing rkt containers.
Docker is based on the concept of LXC 'cgroups' (control groups), which is a linux kernel feature facilitating allocation of resources (cpu/memory/network etc) on a linux system, to sets of privileged tasks in a hierarchical access controlled structure.
Network and resource namespacing is also an important kernel capability that completes the picture of a fully sandboxed execution environment required for sharing infrastructure safely.
This kernel feature was arguably underutilized until Docker TM built tools using this foundation to achieve the benefits of virtual machines, without the overhead of emulating the hardware in addition to the desired application environment.
Docker consists of services and tools to help manage cgroup segmented resources as 'containers' in a way that hierarchies of images can share.
The key to this is that all linux distributions use the same kernel (within versions). For example, a docker image can be created from another docker image as a base, such as the signed Ubuntu image.
Subsequent docker images can also be created 'from' this Ubuntu image, but as you already have the ubuntu base image and you already have a local kernel - it does not need to be downloaded for each image based on ubuntu.
This is a huge time/cost saving, and can subsequently be published for others to use and build upon.
Kernel (host) -> Ubuntu image (local image) -> Ubuntu NGINX (pulled from registry)
Because docker instances use the local kernel and dependencies to run natively, this is a game changer for performance and security. Additional instances can be started in seconds - as long as the host machine has the resources to support this.
Docker 'compose' is a simple way to describe a set of containers and how they are wired together and expose resources. This is a similar notion to a k8s pod to a degree, and a great way to get started learning about docker.
As docker users started requiring horizontally scalable deployments and managed service deployments spanning multiple nodes, Docker 'swarm' was released to provide similar capabilities to k8s.
Docker has since announced native support for k8s https://techcrunch.com/2017/10/17/docker-gives-into-invevitable-and-offers-native-kubernetes-support/
Given k8s is supported as a managed service by all cloud providers, this is a safer bet than swarm, and allows much of the complexity of managing a kubernetes master to be handled by the cloud provider.
The k8s Helm project provides mechanisms for defining releases, deploying, rolling back etc.
Yet another abstraction layer! ...but one tightly coupled to the linux kernel using proven mechanisms.
Speaking with Kelsey Hightower https://twitter.com/kelseyhightower from Google who kindly reviewed this document and helped clarify some areas:
"it helps to think of k8s as the hypervisor for containers such as docker, as k8s does a similar conceptual job as VM hypervisors do for virtual machines".
In order to deal with different cloud provider resource management approaches, this homogeneous abstraction layer still uses specific types of virtual machines provided by AWS or other providers as compute 'nodes'.
Once these k8s nodes are provisioned and attached to a k8s cluster node however, we quickly start to care far less where it is hosted and how it is managed.
It is a similar story for other resource types such as:
- load balancing
- persistent storage
- secrets management
- service discovery
Each provider has a solution for each of these, but they are somewhat different, and therefore k8s abstracts these capabilities for each provider type. If you have a platform that k8s does not know how to abstract the extensibility of k8s allows you to add this yourself.
K8s is built from the ground up to have pluggable extensibility.
It is accepted that containers:
-
are the build output unit that should encapsulate the code asset to be deployed to a homogeneous network.
-
should not not describe how it should be deployed, but can declare what it expects and provides.
Kubernetes does not treat containers as the smallest deployable unit. Instead it uses the concept of a 'pod'.
A pod descriptor generally contains one or two closely related containers, and allows groups of containers to be scaled as a whole - ie, we dont scale containers, we scale pods.
Container instances in a given pod are guaranteed to be on the same node, although pods instances can be scaled out across multiple nodes.
Kubernetes containers in deployments can have defined minimum and maximum system resource requirements and limits (cpu/ram).
This means that pods can be stacked efficiently into nodes, allowing cluster nodes to be autoscaled to the optimum level, and prevent misbehaving containers from impacting other deployments.
There are many other areas to be covered separately such as:
- namespacing
- labels and selectors
- port and dns conventions
- ingress controller types
Lets keep this for the next session.
Rather than spelling out how a service should be deployed and exposed, k8s encourages the approach of specifying the desired outcome, and letting the k8s and helm tiller process work out how things should happen.
The deployment descriptor referencing the pods should control how many instances will be created, along with any dependency types and their scaling patterns.
This can be simply achieved via k8s manifests deployed via kubectl, however typically the 'Helm Chart' approach is used for in order to allow for versioned releases and rollbacks.
Kubernetes has a number of object and controller types for a range of purposes. Please read https://kubernetes.io/docs/concepts/
Increasingly we are seeing efforts by cloud providers to focus on container based and event driven serverless strategies.
- they are faster, cheaper, and simpler to operate
- customers do not want to lock in to vendor specific approaches, when a community led initiative such as k8s is becoming a defacto standard.
To date, all cloud providers as well as Docker have recognised that Kubernetes is the best way to deal with cross context container scheduling complexities, and have released managed k8s offerings. (AWS this year)
It is awesome and interesting.
Most devs are familiar with Docker and generally like it, however in many organisations there is a disconnect between the cool stuff being done by engineers locally, and what happens when code moves through to dev/test/prod.
Kubernetes represents the best bet on moving to a container scheduling operating model within existing approved platforms.
By building with docker in mind, and running k8s locally, teams can work on building deployments that are directly reproducible through to production, without having to build out AMIs etc to support the service deployment.
This creates far less dependency on ops teams, and gives control to engineers building solutions.
No-one likes staying back for releases. They should not be events, they should just happen automatically when certain criteria are met. Continuous deployment is where we are headed as an an industry, and k8s is a proven key enabler on this journey.
From a commercial perspective, under-utilised cloud vms can be stacked efficiently with workloads rather than having dedicated vms per services set and guestimating resource usage. Autoscaling can occur both at the node host provider level, and at the pod level.
From a security perspepctive, docker uses a git-like hierarchy of image hashes, and images can either be pulled from public image repositories or private / hosted repos and validated by hash. This means confidence in deployed asset integrity.
Thanks again @kelseyhightower for review and feedback.