More Related Content
What's hot (20)
Similar to Getting Deep on Orchestration - Nickoloff - DockerCon16 (20)
More from allingeek (6)
Recently uploaded (20)
Getting Deep on Orchestration - Nickoloff - DockerCon16
- 1. Getting Deep on Orchestration Jeff Nickoloff Engineer, Author, and Consultant
- 2. • Author of Docker in Action • Blog about Docker at https://medium.com/on-docker • Professional engineering and containerization consultant • Work with startups and fortune 100 companies • Training • http://allingeek.com • Heaps of orchestration experience at Amazon About me
- 3. What is Orchestration? Before Time Examples Abstractions OSS Agenda Picking Apart Platforms Abstractions Architecture Components Failures Demo: Entropy About failure Architecture Break stuff
- 4. Orchestration What is it anyway?
- 5. Remember what it was like to SCP deployment artifacts? How about just checking out from SVN right into prod? Perl wizardry… How about the luxurious experience of deploying via app server? <2005 … and the following years of pain
- 6. “The first time you encounter real orchestration - having lived in its absence - you may experience the urge to laugh, or cry, or both.” — Jeff Nickoloff
- 8. • Amazon Apollo (Code Deploy) • Deployment as an entity • Amazon Marketplace Feeds and Reports • Clustering, resource aware scheduling, state management It was like living in a different world and there was no going back for anyone. How many proprietary orchestration platforms do you think there are? Proprietary Orchestration
- 9. An orchestration platform is: a system that provides control of high-level abstractions which imply certain deployment and lifecycle management semantics. Abstractions: • Force multipliers for communication • Often required to cope with complexity or variability An Observational Description
- 10. In the configuration management OSS world we have: • Puppet, Chef, Ansible, Packer/Terraform, etc Cloud specific • CloudFormation, ElasticBeanstalk, ECS, etc In the container platform OSS world we have: • Swarm, UCP, Kubernetes, Mesosphere, Rancher, etc OSS Orchestration Reading this list gives me the feels…
- 12. Container Orchestration • Docker + Libnetwork + Swarm + Compose + etcd / consul / zk • Docker + Flannel + Systemd + etcd + kubelet + kube (api, scheduler, controller-manager, proxy, DNS) • Docker (sometimes) + Mesos + Marathon + Calico + zk Breaking down Swarm, Kubernetes, and Mesosphere
- 13. Force multiplying ideas: higher level examples • Container (an isolated context for processes) • Composition / Pod (containers w/ shared lifecycle) • Service (long running event handler) • Replication Controller (managed scale) • Job (process with a linear lifecycle) • Deployment (pushing and starting containers) Abstractions
- 14. What features should we expect? • Frontend • Ingest descriptions of desired state • Provide system visibility (and integration) • Backend • Manage a compute resource pool • Manage logical networks, routes, and names • Service discovery and load balancing • Manage storage attachment, distribution, and replication • Manage containers and maintain the desired state • AA infrastructure integration Architecture
- 15. How do you control the system? • One or a collection of APIs • Command Line Program • Web or Native GUI What do these interfaces interact with on the backend? An Interface
- 16. Where do they maintain the authoritative state? Databases provide accounting for entities and their state. Centralized or decentralized and commonly provide: • KV semantics • Distributed locks • HA with strong consistency (Paxos / Raft) • Record observation (watches) • Update / Delete semantics with fencing tokens System of Record
- 17. Control loops are the platform’s automata. • React to state changes on notification or polling • Calculate deltas with desired state and apply • Might require leadership election for HA Agents / Control Loops How do they maintain the desired state of the system?
- 18. Event driven agents that coordinate changes in the system • Container (re)scheduler • Cluster node registrar • Service node registrar • Local supervisor / init • Service controller • Job controller Agents / Control Loops
- 19. Event driven agents that coordinate changes in the system • State Observation • Feedback for control loops • Entity Lifecycle Graph • Registration and Discovery • Route to an IP • Engine in a cluster • Replica of a service • Endpoint of a service Patterns
- 21. What Breaks? How am I going to fix it?
- 22. • Still need high-low entropy service discovery and routing • This is becoming less of a problem • Bugs/failures you’d experience without orchestration or clustering Common Issues
- 23. • System of Record is a big SPoF • Need durable distributed DB • “Something isn’t showing up!“ • Everything uses the same registration/discovery pattern Clustering Issues
- 24. • N machines and M containers on an overlay or virtual network • Container attachment / detachment • Node attachment / detachment • Heartbeats / node monitoring Networking Stress
- 25. • Upgrade mechanics • Careful dance? • Is rollback possible? • Blue/Green? Platform Lifecycle
- 27. The Future Where do we take these systems from here?
- 28. Maybe a few new abstractions? • Feed (Job with an input document) • Report (Job with an output document) • Request (Job with input and output documents) • Cron Job (run a periodic job) • User / User collection maybe? • Failure Injection Extensions
- 29. Putting your money where your mouth is… Build confidence in complex distributed systems by injecting realistic failures and comparing operations against a steady state. • http://principlesofchaos.org • Failure mode and effects analysis • Netflix / SimianArmy Not much in the container space yet… Failure Injection
- 30. An orchestration abstraction for failure injection Features: • Probabilistic failure injection policies • Failure modes • Latency, partition, GC pause, etc • Applied to existing containers filtered by label • An event stream • Notifications • Integrates with the Docker API Project: Entropy (PoC)
- 31. Demo Slide $ entropy create --failure latency --frequency 10 --probability .50 --image allingeek/gremlins --criteria service=PingGoogle