AtlasCamp 2015 Docker continuous integration training

Continuous Integration with
Docker and Bamboo
STEVE SMITH • DEVOPS ADVOCATE • ATLASSIAN • @TARKASTEVE
https://bitbucket.org/ssmith/atlascamp2015-docker-ci

Warning:
Bad analogies coming up…

VMs vs Containers
Huffington Post

Virtual Machine
Virtual HW
BIOS/EFI
Guest FS / Kernel
Applications
Physical Hardware
Host FS / Kernel
VM Hypervisor
BIOS/EFI
Virtual HW
BIOS/EFI
Guest FS / Kernel
Applications

Containers / Docker
Physical Hardware
Host FS / Kernel
Docker
Engine
BIOS/EFI
Guest FS
Application
Guest FS
Application Virtual HW
BIOS/EFI
Guest FS / Kernel
Applications
Physical Hardware
Host FS / Kernel
VM Hypervisor
BIOS/EFI
Virtual HW
BIOS/EFI
Guest FS / Kernel
Applications

Using features of the Linux kernel
Docker
libcontainer
systemd
nspawn
libvirt
cgroups
CPU accounting
Namespaces
Posix capabilities
btrfs
OverlayFS
LXC

Containers take this further…
Network
Abstraction
Process
Isolation
Filesystem
Isolation
Resource
Limits

Docker takes this to the next level…
Build Tools Packaging
Lifecycle Communication Dev (boot2docker)
Sharing

boot2docker
Virtual Hardware
Host FS / Kernel
BIOS/EFI
Guest FS
Application
Guest FS
Application

Docker containers are like OO classes…
Inheritance
Sourcefile
(Dockerfile) Compile
Dep. Injection Declarative Deps

FROM ubuntu:trusty
RUN apt-get update &&
apt-get install -y nginx &&
apt-get clean
EXPOSE 80
EXPOSE 443
CMD nginx -g daemon off;
Single inheritance
Add member
Override parent behaviour
Exposed linking points
Dockerfile ~= Sourcefile

Inheritance resolved at runtime
Ubuntu
Add nginx
Add run command
(sorta like JVM class assembly)
Union Filesystem
- AUFS
- BTRFS
- Devicemapper
- OverlayFS

‘run’ instantiates a new container
Ubuntu
Add nginx
Add run command
Writable
Immutable
(kinda like ‘new’)
Runtime layer

[ssmith] $ docker build -q --tag=mynginx docker/mynginx/
Sending build context to Docker daemon 12.72 MB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:trusty
---> 8eaa4ff06b53
Step 1 : RUN apt-get update && apt-get install -y nginx && apt-get clea
---> Running in f05a4a034fe3
---> 1e84ac75dd9a
Removing intermediate container f05a4a034fe3
Step 2 : EXPOSE 80
---> Running in 77602b8eb9ec
---> 4076f56233ff
Removing intermediate container 77602b8eb9ec
Step 3 : EXPOSE 443
---> Running in b627b8c1d328
---> f5a3ff140964
Removing intermediate container b627b8c1d328
Step 4 : CMD nginx -g daemon off;
---> Running in 132573fe4fc9
---> c1df0408cf70
‘build’ compiles to binary

Inspecting the stack
[ssmith] $ docker history mynginx
IMAGE CREATED CREATED BY
606f0f611cb2 6 days ago /bin/sh -c #(nop) CMD [nginx -g daemon off;]
27d0fad8620d 6 days ago /bin/sh -c #(nop) EXPOSE map[443/tcp:{}]
dc26b50e9806 6 days ago /bin/sh -c #(nop) EXPOSE map[80/tcp:{}]
7a04b014a0a7 6 days ago /bin/sh -c apt-get update && apt-get install
b39b81afc8ca 9 days ago /bin/sh -c #(nop) CMD [/bin/bash]
615c102e2290 9 days ago /bin/sh -c sed -i 's/^#s*(deb.*universe)$/
837339b91538 9 days ago /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic
53f858aaaf03 9 days ago /bin/sh -c #(nop) ADD file:ca5b63647c6b7a419e
511136ea3c5a 19 months ago

Linked Containers
(dependency injection; sort-of…)
Docker host
postgresql client
--link postgresql:pg
can refer to
hostname pg
in commands

Volumes
Docker host
/var/volume1
DATA
/var/volume2
/var/volume1
client1
/var/volume2
--volumes-from DATA
(haven’t got a class analogy for this one)

The example project
?
https://bitbucket.org/ssmith/atlascamp2015-docker-ci

The example project
Transfer
Trigger / Async
Data
Data
(See http://bit.do/postgres-es for details)

How to test…
Vagrant?
Dependencies
everywhere?
Docker?

Docker testing
Reuse images Startup speed Idempotent tests
Dev / Test match Deploy to Docker

Elasticsearch Dockerfile
FROM java:openjdk-7-jre
ENV ES_VER 1.3.4
ENV ES_URL https://download.elasticsearch.org/elasticsearch/elasticsearch/
elasticsearch-${ES_VER}.tar.gz
WORKDIR /opt/
RUN adduser --system elasticsearch
RUN curl ${ES_URL} -o elasticsearch.tgz &&
tar xzf elasticsearch.tgz &&
ln -s elasticsearch-${ES_VER} elasticsearch &&
mkdir /opt/elasticsearch/data/ && chown elasticsearch /opt/elasticsearch/data/ &&
mkdir /opt/elasticsearch/logs/ && chown elasticsearch /opt/elasticsearch/logs/ &&
rm elasticsearch.tgz
EXPOSE 9200
EXPOSE 9300
USER elasticsearch
CMD /opt/elasticsearch/bin/elasticsearch -f

Elasticsearch build
[ssmith] docker build -q --tag=elasticsearch docker/elasticsearch/
Sending build context to Docker daemon 2.56 kB
Step 0 : FROM java:openjdk-7-jre
---> b797b3ba124d
Step 1 : ENV ES_VER 1.3.4
---> Running in 4842001c48d0
---> 2717f23208b7
Removing intermediate container 4842001c48d0
Step 2 : ENV ES_URL https://download.elasticsearch.org/elasticsearch/elasticsearch/
elasticsearch-${ES_VER}.tar.gz
---> Running in b2f574153a18
---> a292b608b854
Removing intermediate container b2f574153a18
Step 3 : WORKDIR /opt/
---> Running in 0f138382e071
---> aa2ebf2e8061
Removing intermediate container 0f138382e071
Step 4 : RUN adduser --system elasticsearch
---> Running in 0800878ae8c0
---> 2b0c2619f328
Removing intermediate container 0800878ae8c0
Step 5 : RUN curl ${ES_URL} -o elasticsearch.tgz && tar xzf elasticsearch.tgz &&
ln -s elasticsearch-${ES_VER} elasticsearch && mkdir /opt/elasticsearch/data/ &&
chown elasticsearch /opt/elasticsearch/data/ && mkdir /opt/elasticsearch/logs/ &&

Postgresql Dockerfile
FROM postgres:9.2
ENV POSTGRES_USER customer
COPY testschema.sql /
COPY init-test-db.sh /docker-entrypoint-initdb.d/

Postgresql init file
#!/bin/bash
echo "#### Create test DB ####"
gosu postgres postgres --single <<EOF
CREATE USER customer PASSWORD 'customer';
CREATE DATABASE customer OWNER customer;
GRANT ALL PRIVILEGES ON DATABASE customer TO customer;
EOF
echo "#### Initialise test DB ####"
gosu postgres postgres --single customer -j < /testschema.sql
echo
echo "#### Test DB initialised ####"

Postgresql build
[ssmith] docker build -q --tag=postgres docker/postgres/
Sending build context to Docker daemon 5.12 kB
Step 0 : FROM postgres:9.2
---> 898ba6de4072
Step 1 : ENV POSTGRES_USER customer
---> Running in 932957e34d5e
---> 91b679d67fd9
Removing intermediate container 932957e34d5e
Step 2 : COPY testschema.sql /
---> 65b67419fe54
Removing intermediate container 54298baaa088
Step 3 : COPY init-test-db.sh /docker-entrypoint-initdb.d/
---> b6d84236c1f7
Removing intermediate container f9000db9da92
Successfully built b6d84236c1f7

Transfer Dockerfile
FROM java:openjdk-7-jre
# Install netcat for waitport
RUN apt-get update &&
apt-get install -y netcat-openbsd &&
apt-get clean
COPY docker/waitport /usr/local/bin/
RUN curl https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -o /
usr/local/bin/lein &&
chmod a+rx /usr/local/bin/lein
RUN adduser --disabled-password --gecos '' transfer
ADD . /code
RUN chown -R transfer.transfer /code
USER transfer
WORKDIR /code
RUN lein deps
CMD waitport elasticsearch 9200 &&
waitport postgres 5432 &&
lein test-out junit

Manual run
[ssmith] docker run -d —name=postgres postgres
181470ed78836c9285664c18f0d5b4c6a9069d28c4f71b9d621d4c24762ad938
[ssmith] docker run -d --name=elasticsearch elasticsearch
5718e1ba9b7caf0e25f53d803a128d2a3d4d518ae8d747b843cc2cbdd78620ce
[smith] docker run --link postgres:postgres --link elasticsearch:elasticsearch transfer
Waiting for TCP connection to elasticsearch:9200...OK
Waiting for TCP connection to postgres:5432...OK
Received notification for accountupdate for 55d3290b-5baa-4ca6-b353-cacf5b104c60
Performing update of 55d3290b-5baa-4ca6-b353-cacf5b104c60
Indexing row 55d3290b-5baa-4ca6-b353-cacf5b104c60 Mr. 55d3290b 55d3290b Esq.
Dispatch complete

Manual cleanup
[ssmith] docker ps
CONTAINER ID IMAGE COMMAND CREATED …
5718e1ba9b7c elasticsearch:latest "/bin/sh -c '/opt/el 5 minutes ago …
181470ed7883 postgres:latest "/docker-entrypoint. 6 minutes ago …
[ssmith] docker kill elasticsearch postgres
elasticsearch
postgres
[ssmith] docker ps
CONTAINER ID IMAGE COMMAND CREATED …
[ssmith] docker ps -a
CONTAINER ID IMAGE COMMAND CREATED
ff6e5409d8ba transfer:latest "/bin/sh -c 'waitpor 11 minutes ago
5718e1ba9b7c elasticsearch:latest "/bin/sh -c '/opt/el 11 minutes ago
181470ed7883 postgres:latest "/docker-entrypoint. 11 minutes ago
[smith] docker rm transfer postgres elasticsearch
transfer
postgres
elasticsearch

Simplifying with Compose
feedyourskull.com

Compose
Declarative Deps Fetch Images Build Images
Link Containers

Our docker-compose.yml
transfer:
build: .
links:
- postgres
- elasticsearch
postgres:
build: docker/postgres
elasticsearch:
build: docker/elasticsearch

Compose Run
[ssmith] docker-compose up
Creating devweek15code_postgres_1...
Creating devweek15code_elasticsearch_1...
Creating devweek15code_transfer_1...
Attaching to devweek15code_postgres_1, devweek15code_elasticsearch_1, devweek15code_transfer_1
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
<SNIP> <SNIP> <SNIP> <SNIP> <SNIP>
transfer_1 | Waiting for TCP connection to postgres:5432...OK
transfer_1 | Received notification for accountupdate for dcf286bf-7011-40f5-abb0-153d94acde
transfer_1 | Performing update of dcf286bf-7011-40f5-abb0-153d94acde69
transfer_1 | Indexing row dcf286bf-7011-40f5-abb0-153d94acde69 Mr. dcf286bf dcf286bf Esq.
elasticsearch_1 | creating index, cause [auto(index api)], shards [5]/[1], mappings []
elasticsearch_1 | update_mapping [account] (dynamic)
transfer_1 | Dispatch complete
devweek15code_transfer_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)
Stopping devweek15code_elasticsearch_1...
Stopping devweek15code_postgres_1...

Compose Cleanup
[ssmith] docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------
devweek15code_elasticsearch_1 /bin/sh -c /opt/elasticsea ... Exit -1
devweek15code_postgres_1 /docker-entrypoint.sh postgres Exit 0
devweek15code_transfer_1 /bin/sh -c waitport elasti ... Exit 0
[ssmith] docker-compose rm —force
Going to remove devweek15code_transfer_1, devweek15code_elasticsearch_1, devweek15code_postgres_1
Removing devweek15code_postgres_1...
Removing devweek15code_elasticsearch_1...
Removing devweek15code_transfer_1…
--------------------------------------------------------------------------------
[ssmith]

Running With Bamboo
Plan
Stage
Job
Tasks
QA
Staging
Prod
Artifacts
Deployment
Environments
Git Repo

Bamboo Live
https://atlascamp2015.atlassian.net/

AWS cloud agents with Docker
With Docker
Or create
your own

Require Docker support
Bamboo will start
on-demand

Compose with BambooSouth Florida
Classical Review

Extracting ResultsAtlantic Sentinel

Fetching container files
--------------------------------------------------------------------------------
devweek15code_elasticsearch_1 /bin/sh -c /opt/elasticsea ... Exit -1
devweek15code_postgres_1 /docker-entrypoint.sh postgres Exit 0
devweek15code_transfer_1 /bin/sh -c waitport elasti ... Exit 0
[ssmith] docker-compose ps -q transfer
fd7e728e3e9a361d96c253f5aeadab1a3506538d1b0e19d27c82848d9bf48bf8
[ssmith] ID=`docker-compose ps -q transfer`
[ssmith] docker cp $ID:/code/testreports.xml .
[ssmith] file testreports.xml
testreports.xml: XML document text

Deployment Environments
Plan
Stage
Job
Tasks
QA
Staging
Prod
Artifacts
Deployment
Environments
Git Repo

Deployment Environments
QA
Staging
Prod
Artifacts
Deployment
Environments

2
3
Useful Links
https://www.docker.com/subscribe_newsletter/
1 https://www.docker.com/
4 http://www.nkode.io/2014/08/24/valuable-docker-links.html
http://boot2docker.io/

Thank you!
STEVE SMITH • DEVOPS ADVOCATE • ATLASSIAN • @TARKASTEVE

AtlasCamp 2015 Docker continuous integration training

Recommended

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to AtlasCamp 2015 Docker continuous integration training (20)

More from Steve Smith (10)

Recently uploaded (20)

AtlasCamp 2015 Docker continuous integration training