Docker to the Rescue of an Ops Team

Docker to the rescue
of an Ops Team
Rachid
Zarouali
C.I.O Synolia
Twitter / Slack : Xinity
rachid.zarouali@synolia.com

Agenda
This talk is about:
● Monitoring (a bit)
● Private Registry
● CI/CD
● Security
● Docker experience

Once upon a time
An ops team starts a new project

Rebuild everything !
June 2014
A monitoring system from scratch
“Microservices” oriented
Replaceable parts

Some rules first !
Simple
Efficient
Extendable
Python based

Components
Collectd
Collectd proxy
Graphite
Grafana
Cabot (alerting)

Test your might !
Python 2.6 along with python 2.7
Different version of “some” libraries
Whisper backend (I/O storm)
Upstream repositories issues

Docker to the rescue
Save our project

Docker ? way too soon !
Barely no skills
Used only to do some testing
Pretty serious concerns

Ok let’s gamble !
Grow our docker fu
Write some Dockerfiles
Build images locally
Spawn PoC platform

…..
RUN echo "deb http://mirror.debian.ikoula.com/debian wheezy-backports main" >> /etc/apt/sources.list
RUN apt-get -qq update
RUN apt-get -qqy dist-upgrade
RUN apt-get -qqy --force-yes install vim python-cairo gunicorn supervisor (...)
RUN pip install whitenoise txamqp whisper==0.9.13 carbonate
RUN pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/lib" carbon==0.9.13
RUN pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/webapp"
graphite-web==0.9.13
ADD conf/nginx.conf /etc/nginx/nginx.conf
ADD conf/supervisord.conf /etc/supervisor/conf.d/grafana.conf
ADD initial_data.json /var/lib/graphite/webapp/graphite/initial_data.json
ADD conf/local_settings.py /var/lib/graphite/webapp/graphite/local_settings.py
ADD conf/carbon.conf /var/lib/graphite/conf/carbon.conf
ADD conf/storage-schemas.conf /var/lib/graphite/conf/storage-schemas.conf
RUN mkdir -p /var/lib/graphite/storage/whisper
RUN touch /var/lib/graphite/storage/graphite.db /var/lib/graphite/storage/index
RUN chmod 0775 /var/lib/graphite/storage /var/lib/graphite/storage/whisper
RUN python /var/lib/graphite/webapp/graphite/manage.py syncdb --noinput --pythonpath=/var/lib/graphite/webapp/graphite
--settings=settings
RUN chmod 0664 /var/lib/graphite/storage/graphite.db
RUN chown -R www-data /var/lib/graphite/storage
…..

Container = OS …. Wait !
Too many layers ( 121+ layer issue)
Build time …. (20 to 30 minutes at best)
Huge Images (800+ Mo)
Unnecessary tools and libs

We can do better !
Apply best Practices (@abbyfuller)
Implement simple CI/CD
Dockerfile Linting
Build a private registry
Deal with security concerns

…
RUN echo "APT::Install-Recommends false;" >> /etc/apt/apt.conf.d/00recommends
&& echo "APT::Install-Suggests false;" >> /etc/apt/apt.conf.d/00recommends
&& echo "APT::AutoRemove::RecommendsImportant false;" >> /etc/apt/apt.conf.d/00recommends
&& echo "APT::AutoRemove::SuggestsImportant false;" >> /etc/apt/apt.conf.d/00recommends
ENV DEBIAN_FRONTEND noninteractive
ENV GRAPHITE_VERS 0.9.13
RUN apt-get -qqy update
&& apt-get -qqy install python-cairo gunicorn git python2.7-dev wget ca-certificates python-flup expect sqlite3 libcairo2
libcairo2-dev pkg-config nodejs sqlite3 memcached python-ldap make gcc libffi-dev
RUN wget https://bootstrap.pypa.io/get-pip.py
&& python get-pip.py
&& pip install --no-cache-dir --upgrade setuptools
&& pip install --no-cache-dir django django-admin-tools
&& pip install --no-cache-dir whitenoise txamqp whisper==${GRAPHITE_VERS} carbonate
&& pip install --no-cache-dir --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/lib"
carbon==${GRAPHITE_VERS}
&& pip install --no-cache-dir --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/webapp"
graphite-web==${GRAPHITE_VERS}
RUN apt-get purge gcc make python2.7-dev libcairo2-dev libffi-dev python2.7-dev pkg-config -qqy
&& apt-get clean
&& apt-get autoremove -qqy
&& rm -rf /root/.cache /var/lib/apt/lists/* /tmp/* /var/tmp/*
...

Best practices :)
Few image layers ( < 20 )
Small image (~ 400Mo)
Lower footprint (100Mo)
Faster build time (~5Min)

CI/CD Recipe
build:
image:
registry.synolia.com/synolia/dockerunitest:latest
publish:
docker:
repo: synomon_datastor
tag: $${BRANCH/master/latest}
file: Dockerfile
insecure: true
when:
repo: synolia/systeam-monitoring_datastor
branch: [develop, master]
notify:
hipchat:
from: "synoci"
room_id_or_name: "$$ROOM_ID"
auth_token: "$$AUTH_TOKEN"
notify: true
when:
success: false
failure: true
Dockerfile Linting
Build
Push
Notify (fail only)

Docker to the rescue II
The return of the hero moby

This isn’t over yet !
Docker udp issues
Tricky iptable filtering
Unstable data volume
Configuration management

Round 2: FIGHT !
Metric proxy (Collectd) on the
host!!
Simplify iptables rules
Mount directories (metrics)

A new path opens
To a brighter future

Epic loots !
No more dependency issues
Replaceable and movable parts
Greater security level
Clustering ready (Swarm/K8S)

Lessons learned
Caution when using udp IPv4
Config files out of the container
Don’t use env variables (security)
Use (abuse) automation

What’s next ?
Greater Docker challenges

Many rooms to grow
Reduce (even more) Image size
Sign Images (notary to the rescue)
Vulnerability scanning
Implement rolling upgrades

New docker based projects
Migrate Development platform (2015)
Swarm clustering (*)
Full scale Docker (*)
(*) Work In Progress

Thank You DockerCon !
PS: don’t forget to rate my talk :)

Docker to the Rescue of an Ops Team

Recommended

More Related Content

What's hot (20)

Viewers also liked (16)

Similar to Docker to the Rescue of an Ops Team (20)

More from Docker, Inc. (20)

Recently uploaded (20)

Docker to the Rescue of an Ops Team