The challenge of application distribution - Introduction to Docker (2014 dec 03)

The challenge of application
distribution
Introduction to Docker

Credits & references
Intros slides borrowed from
• dotCloud’s « Docker introduction » slideshow
• AWS re:Invent 2014 session APP303, « Lighting Fast Deploys with
Docker Containers and AWS »

Agenda
• The application distribution challenge
• The current solutions
• Introduction to Docker, Containers, and the Matrix from Hell
• Why people care: Separation of Concerns
• Technical Discussion
• Ecosystem, momentum
• How to build Docker images
• How to make containers talk to each other, how to handle data
persistence
• Demo 1: isolation
• Demo 2: real case - installing Go Math!, tail –f containers, unit
tests

The challenge: apps have fundamentally changed
~2000 2014
Long lived Development is iterative and
constant
Monolithic and built on a
single stack
Built from loosely coupled
components
Deployed to a single server Deployed to a multitude of
servers

Everybody already heard/said it:

The challenge: apps have fundamentally changed
Static website
User DB
Redis + redis-sentinel
Web frontend
Queue Analytics DB
Background workers
API endpoint
nginx 1.5 + modsecurity + openssl + bootstrap 2
postgresql + pgv8 + v8
hadoop + hive + thrift + OpenJDK
Ruby + Rails + sass + Unicorn
Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv +
nodejs + phantomjs
Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client
Development VM
QA server
Public Cloud
Disaster recovery
Contributor’s laptop
Production Servers
Multiplicity of
Stacks
Multiplicity of
hardware
environments
Production Cluster
Customer Data Center
Do services and
apps interact
appropriately?
Can I migrate
smoothly and
quickly?

The Matrix From Hell
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
Dev VM QA Server
Single Prod
Server
Onsite
Cluster
Public Cloud
Contributor’s
laptop
Customer
Servers

Current solutions
• ‘Configuration’ tools like Ansible, Chef, Puppet, …
• VM tools : Vagrant
• Container tools: Google’s LMCTFY

Goods
Multiplicity of
transporting/storing
Multipilicity of
methods for
Do I worry about
how goods
interact (e.g.
coffee beans next
to spices)
Can I transport
quickly and
smoothly
(e.g. from boat to
train to truck)
Cargo Transport Pre-1960

Also a matrix from hell
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?

Multiplicity of
Goods
Multiplicity of
methods for
transporting/storin
g
Do I worry about
how goods
interact (e.g.
coffee beans next
to spices)
Can I transport
quickly and
smoothly
(e.g. from boat to
train to truck)
Solution: Intermodal Shipping Container
…in between, can be loaded
and unloaded, stacked,
transported efficiently over long
distances, and transferred from
one mode of transport to
another
A standard container that is
loaded with virtually any
goods, and stays sealed until
it reaches final delivery.

Containers applied to software

Static website User DB Web frontend Queue Analytics DB
Dev VM
QA server Public Cloud
Contributor’s
laptop
Docker is a shipping container system for code
Multiplicity of
Stacks
Multiplicity of
hardware
environments
Production
Cluster
Customer Data
Center
Do services and
apps interact
appropriately?
Can I migrate
smoothly and
quickly
…that can be manipulated
using standard operations and
run consistently on virtually any
hardware platform
An engine that enables any
payload to be encapsulated
as a lightweight, portable,
self-sufficient container…

Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Dev VM QA Server
Single Prod
Server
Onsite
Cluster
Public Cloud
Contributor’s
laptop
Customer
Servers
Docker eliminates the matrix from Hell

Good for developers
• Build once…(finally) run anywhere*
• A clean, safe, hygienic and portable runtime environment for your app.
• No worries about missing dependencies, packages and other pain points during
subsequent deployments.
• Run each app in its own isolated container, so you can run various versions of
libraries and other dependencies for each app without worrying
• Automate testing, integration, packaging…anything you can script
• Reduce/eliminate concerns about compatibility on different platforms, either your
own or your customers.
• Cheap, zero-penalty containers to deploy services? A VM without the overhead of a
VM? Instant replay and reset of image snapshots? That’s the power of Docker
* Anywhere = any x86 server running a modern Linux kernel (3.2+ generally. 2.6.32+ for RHEL 6.5+, Fedora, &
related)

Good for (Dev)Ops
• Configure once…run anything
• Make the entire lifecycle more efficient, consistent, and repeatable
• Increase the quality of code produced by developers.
• Eliminate inconsistencies between development, test, production, and customer
environments
• Support segregation of duties
• Significantly improves the speed and reliability of continuous deployment and
continuous integration systems
• Because the containers are so lightweight, address significant performance, costs,
deployment, and portability issues normally associated with VMs

Why it works—separation of concerns
• Dan the Developer
– Worries about what’s “inside” the
container
• His code
• His Libraries
• His Package Manager
• His Apps
• His Data
– All Linux servers look the same
• Oscar the Ops Guy
• Worries about what’s
“outside” the container
• Logging
• Remote access
• Monitoring
• Network config
• All containers start, stop,
copy, attach, migrate, etc. the
same way

Under the hood
• To learn more about how it works, some background:
–Linux namespaces
• PID
• IPTables
• UTS (~hostname)
• Mount
• IPC (inter-process)
–CGroups
–LXC Containers

Containers vs. VMs
App
A
Bins
/
Libs
Bins
/
Libs
Bins
/
Libs
Hypervisor (Type 2)
Host OS
Server
Guest
OS
App
A’
Gues
t
OS
App
B
Gues
t
OS
App A’
Docker
Host OS
Server
App A
Bins/Libs
Bins/Libs
App B
App B’
App B’
App B’
VM
Container
Containers are isolated,
but share OS and, where
appropriate, bins/libraries
Guest
OS
Guest
OS
…result is significantly faster
deployment, much less overhead,
easier migration, faster restart

Why are Docker containers lightweight?
VMs Containers
App
A
Bins
/
Libs
Original App
(No OS to take
up space, resources,
or require restart)
App
Δ
Bins
/
App
A
Bins
/
Libs
App
A’
Bins
/
Libs
Gues
t
OS
Modified App
Copy on write
capabilities allow us to
only save the diffs
between container A
and container A’
VMs
App
A
Bins
/
Libs
Guest
OS
Every app, every copy of an
app, and every slight modification
of the app requires a new virtual server
App
A
Copy of
App
No OS. Can
Share bins/libs
Guest
OS
Guest
OS

Terminology
Docker concept OOP analogy
Dockerfile Source code
Image Class
Container Instance

What are the basics of the Docker system?
Dockerfile
For
A
Source
Code
Repositor
y
Docker Engine
Docker
Container
Image
Registry
Build
Docker
Host 2 OS (Linux)
Container A
Container B
Container C
Container A
Push
Search
Pull
Run
Host 1 OS
(Linux)

Changes and Updates
App
A’’
Bins
/
Libs
Docker Engine
Docker
Container
Image
Registry
Push
Update
Docker Engine
App
A
Bins
/
Libs
App
Δ
Bins
/
Base
Container
Image
Host is now running
A’’
Container
Mod A’’
App
Δ
Bins
/
App
A
Bins
/
Libs
Bins
/
Host running A wants to upgrade to A’’.
Requests update. Gets only diffs
Container
Mod A’

A new open source project… or the next big thing?
• Compatible with Jenkins, Travis, Chef, Puppet, Vagrant and OpenStack…
• Ecosystem: lots of tools, even by serious contributors (Google –
Kubernetes, cAdvisor - NetFlix …), new IAAS provider emerge (like
Tutum),
• Supported by Amazon, Google, Microsoft, Rackspace, Digital Ocean,
Joyent, RedHat, …
• Momentum... See pull requests numbers and frequency to get an idea
• Used in production… even by big players (HP, Ebay, New Relic, Etsy,
Yelp, Spotify, …)

Ecosystem Support
• Operating systems
– Virtually any distribution with a 2.6.32+ kernel
– Red Hat/Docker collaboration to make work across RHEL 6.4+, Fedora, and other members of the family (2.6.32
+)
– CoreOS—Small core OS purpose built with Docker
• OpenStack
– Docker integration into NOVA (& compatibility with Glance, Horizon, etc.) accepted for Havana release
• Private PaaS
– OpenShift
– Solum (Rackspace, OpenStack)
– Other TBA
• Public PaaS
– Deis, Voxoz, Cocaine (Yandex), Baidu PaaS
• Public IaaS
– Native support in Rackspace, Digital Ocean,+++
– AMI (or equivalent) available for AWS & other
• DevOps Tools
– Integrations with Chef, Puppet, Jenkins, Travis, Salt, Ansible +++
• Orchestration tools
– Mesos, Heat, ++
– Shipyard & others purpose built for Docker
• Applications
– 1000’s of Dockerized applications available at index.docker.io

Use Cases
• Ted Dziuba on the Use of Docker for Continuous Integration at Ebay Now
– https://speakerdeck.com/teddziuba/docker-at-ebay
– http://www.youtube.com/watch?feature=player_embedded&v=0Hi0W4gX--4
• Sasha Klizhentas on use of Docker at Mailgun/Rackspace
• http://www.youtube.com/watch?feature=player_embedded&v=CMC3xdAo9RI
• Sebastien Pahl on use of Docker at CloudFlare
• http://www.youtube.com/watch?feature=player_embedded&v=-Lj3jt_-3r0
• Cambridge HealthCare
• http://blog.howareyou.com/post/62157486858/continuous-delivery-with-docker-and-
jenkins-part-i
• Red Hat Openshift and Docker
• https://www.openshift.com/blogs/technical-thoughts-on-openshift-and-docker

Use Cases—From the Community
Use Case Examples Link
Clusters Building a MongoDB cluster using docker http://bit.ly/1acbjZf
Production Quality MongoDB Setup with Docker http://bit.ly/15CaiHb
Wildfly cluster using Docker on Fedora http://bit.ly/1bClX0O
Build your own PaaS OpenSource PaaS built on Docker, Chef, and Heroku Buildpacks http://deis.io
Web Based
JiffyLab – web based environment for the instruction, or lightweight use of,
Environment for
Python and UNIX shell
Instruction
http://bit.ly/12oaj2K
Easy Application
Deployment
Deploy Java Apps With Docker = Awesome http://bit.ly/11BCvvu
How to put your development environment on docker http://bit.ly/1b4XtJ3
Running Drupal on Docker http://bit.ly/15MJS6B
Installing Redis on Docker http://bit.ly/16EWOKh
Create Secure
Sandboxes
Docker makes creating secure sandboxes easier than ever http://bit.ly/13mZGJH
Create your own SaaS Memcached as a Service http://bit.ly/11nL8vh
Automated Application
Deployment
Multi-cloud Deployment with Docker http://bit.ly/1bF3CN6
Continuous Integration
and Deployment
Next Generation Continuous Integration & Deployment with dotCloud’s
Docker and Strider
http://bit.ly/ZwTfoy
Testing Salt States Rapidly With Docker http://bit.ly/1eFBtcm
Lightweight Desktop
Virtualization
Docker Desktop: Your Desktop Over SSH Running Inside Of A Docker
Container
http://bit.ly/14RYL6x

How to build an image?
Two ways to create an image:
• You work in a container, then commit it for exploratory work
• You build a new image from a Dockerfile for automated,
repeatable process

1 container ?= (1process|1task|other) ?
Best practices appear
• Do not force you to 1 process per container
• Ideal case: 1 ‘task’ per container, with separation of
concerns.
• The task might require 1 single process
• Or 0 (for data-only containers)
• Or several (ex: nginx+php-fpm, monitoring, …)

Containers are isolated…
then how to make them talk to each other?
• Private network between containers
• Linking to make them easy to talk to a friend container
• Expose ports
• Mount volumes (rw or ro)
• Advanced/not so trivial other solutions
–Service discovery
–Enter into the container’s namespace

How to handle data persistence
• Data only containers, sharing data volumes
• Mounting this data volumes in the running containers

How to handle data persistence
How data volume works

Demo time!
• Quick look at the Docker commands
• Isolation demo
• Multi containers demo: installing and launching Go Math!
Academy in seconds.
• Running unit tests in a container

The challenge of application distribution - Introduction to Docker (2014 dec 03)

Warning
•Docker is great
–It reduces dependencies, provides isolated and
controlled context
•But it brings new tools, and new technologies, hence
a potential complexity increase
–libswarm, systemd, etcd, ambassadord, fleet, etc

Some references
• Start by the online tutorial and documentation, they are
great!
• Here’s some links kept while learning… but so much
available!
–Eight Docker Development Patterns
–Panamax and Docker and Some Best Practices
–Tutum blog
–The 5 Most Important Things I’ve Learned From Using
Docker
–Understanding Docker Volumes
–Docker's Youtube videos

More references
First steps
• dotScale 2013 - Solomon Hykes - Why we built Docker (video)
• Twitter University – Solomon Hykes – Introduction to Docker (video)
Going further
• Docker layers cost (blog), on how to optimize your images layer size.
• Twitter University - Docker at Spotify (video)
• Top 10 open-source Docker developer tools (blog, incomplete but gives ideas)
• Docker In-depth: Volumes (blog)
Food for thought / Stimulate your imagination
• AWS re:Invent 2014 - Lightning fast deploys with Docker containers and AWS (video,
slides)
• FutureStack14 - Docker and the Future of Modern Software (New Relic blog, video)
• AWS re:Invent 2014 - Infrastructure as Code (video, more related to Cloudformation
than to Docker but very inspiring if you consider the two technologies)
• Alternative? Rocket by CoreOS… started a debate (like Joyent answer)

Tools for dev, day-to-day
• For everyone
• On Mac
– Kitematic UI to manage your images/containers
• On Windows
– Try the new XShell5 beta to easily deal with all your current containers
• So many others… and new ones every week
• Fig (Fast isolated environments, by Docker) repo, site
• A Better Development Environment with Docker and Fig (blog)
• Lots of tools to manage multi containers apps: Kubernetes, Shipyard,
Panamax,

The challenge of application distribution - Introduction to Docker (2014 dec 03)

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to The challenge of application distribution - Introduction to Docker (2014 dec 03) (20)

Recently uploaded (20)

The challenge of application distribution - Introduction to Docker (2014 dec 03)

Editor's Notes