Getting Deep on Orchestration: APIs, Actors, and Abstractions in a Distributed System by Docker Captain Jeff Nickoloff
5 likes12,808 views
The document discusses orchestration in software engineering, emphasizing its role as a system providing organization and planning abstractions for managing complex software systems. It covers various orchestration platforms like Docker, Kubernetes, and Mesosphere, detailing components and agents that facilitate communication and state management. Additionally, it introduces the concept of 'entropy' as a tool for failure injection to enhance system resilience and offers a demo of its implementation.
Getting Deep on Orchestration: APIs, Actors, and Abstractions in a Distributed System by Docker Captain Jeff Nickoloff
- 1. Getting Deep on Orchestration Jeff Nickoloff Engineer, Author, and Consultant
- 2. What is orchestration? Abstractions Examples Agenda Components and patterns APIs System of Record Agents Demo: Entropy About failure Architecture Break stuff
- 3. High quality abstractions are force multipliers for communication and reasoning about more complex (or potentially variable) ideas and systems. Using a computer or programming is an act of communication. <tba> (Software that provides such an abstraction gives a user amazing power.) Software like Docker. About Abstractions
- 4. Orchestration What is it anyway?
- 5. “Orchestrate (v): to organize or plan (something that is complicated).” — Merriam-Webster
- 6. “Orchestrating (tv): to arrange or combine so as to achieve a desired or maximum effect.” — Merriam-Webster
- 7. (for orchestral music uses see Podium) A system that provides organization and planning abstractions for other abstractions. In the OSS world we have: • Swarm • Kubernetes • Mesosphere Orchestration Platform (Software)
- 8. (for orchestral music uses see Podium) A system that provides organization and planning abstractions for other abstractions. Orchestration Platform (Software) In the OSS world we have: • Docker + Swarm + Compose + etcd / consul / zk • Docker + Flannel + Systemd + etcd + kubelet + kube (api, scheduler, controller-manager, proxy, DNS) • Mesos + Marathon + Calico + zk
- 9. Force multiplying ideas: a few lower level examples • Engine (runs containers) • Cluster (nodes as an engine) • Network (routing, overlay, etc) • and network name resolution • Volume (named sharable storage) Abstractions
- 10. Force multiplying ideas: higher level examples • Service (long running event handler) • Job (process with a linear lifecycle) • Feed (Job with an input document) • Report (Job with an output document) • Request (process a single request) • Cron Job (run a periodic job) Abstractions
- 12. … probably many! Users and other software compositions need some means to interact with the orchestration platform. • One or a collection of APIs • Command Line Program • Web or Native GUI An Interface
- 13. Managed state of the platform Accounting for entities (container, pod, service, volume, etc) and their state. Commonly provide: • KV semantics • Record observation (watches) • Update / Delete semantics with fencing tokens • Distributed locks • HA with strong consistency (Paxos / Raft) System of Record
- 14. API’s provide interfaces between the platform and users or other tools. Control loops are the platform’s automata. • Passively react to state changes • Manage system state based on active monitoring • Might require leadership election for HA Agents / Control Loops
- 15. Event driven agents that coordinate changes in the system • Container (re)scheduler • Cluster node registrar • Service node registrar • Low entropy network registrar • Local supervisor / init • Service controller • Job controller • Report workflow controller • Feed workflow controller • Distributed cron controller Agents / Control Loops
- 16. Event driven agents that coordinate changes in the system • State Observation • Feedback for control loops • Entity Lifecycle Graph • Registration and Discovery • Route to an IP • Engine in a cluster • Replica of a service • Endpoint of a service Patterns
- 17. Getting dirty <insert illustration of a Swarm cluster> <insert illustration of a Kubernetes cluster> <insert illustration of a Mesosphere cluster> Deep Dive: Cluster
- 18. Even more examples <tba> Deep Dive: Service
- 19. An Example Demo a new abstraction: Entropy
- 20. A powerful and complex idea Build confidence in complex distributed systems by injecting realistic failures and comparing operations against a steady state. • http://principlesofchaos.org • Failure mode and effects analysis • Netflix / SimianArmy Failure Injection
- 21. … and container based platforms All container platforms (clustered or not): • enable high-entropy systems • add new components and failure modes • provide new mechanisms for handling failure __ extension point, failure injection interface (the kernel), cross cutting concern Failure Injection
- 22. An orchestration abstraction for failure injection Features: • Probabilistic failure injection policies • Failure modes • Latency, partition, GC pause, etc • Applied to existing containers filtered by label • An event stream • Notifications • Integrates with the Docker API Project: Entropy
- 23. An orchestration abstraction for failure injection Components: • Microservice API • CLI • Policy manager (control loop) • Failure injection agents (pluggable) On GitHub: __ insert URL Project: Entropy
- 24. Demo Slide $ # Container platform is running. Start a service. $ docker-compose up -d -p election -f election.yml $ # Start the entropy platform $ docker-compose up -d -p entropy -f entropy.yml $ # Define a policy $ entropy -H $DOCKER_HOST run -f 1s -p .1 -t servicename=voter recv_drop