Building a Docker Swarm cluster on ARM by Dieter Reuter and Stefan Scherer

Jun 27, 20161 like1,577 views

In this training you'll learn how to build a physical Docker Swarm cluster with Raspberry Pi's. We'll guide you through the setup process and you’ll learn how to use Docker and Docker Swarm to complete the cluster. You'll learn how to build and deploy a distributed application and ship it as Docker containers to your cluster. In the end you have built a portable datacenter which can be used for testing and live demos as well. - building a hardware cluster with Raspberry Pi’s - install and set up HypriotOS, a Debian-based Linux system - installing Docker and Docker Swarm to connect all cluster nodes - build a distributed application, the Docker voting app - ship the app with Docker Compose to your cluster - let’s run and test your app In this training you'll learn how to build a physical Docker Swarm cluster with Raspberry Pi's. We'll guide you through the setup process and you’ll learn how to use Docker and Docker Swarm to complete the cluster. You'll learn how to build and deploy a distributed application and ship it as Docker containers to your cluster. In the end you have built a portable datacenter which can be used for testing and live demos as well. - building a hardware cluster with Raspberry Pi’s - install and set up HypriotOS, a Debian-based Linux system - installing Docker and Docker Swarm to connect all cluster nodes - build a distributed application, the Docker voting app - ship the app with Docker Compose to your cluster - let’s run and test your app

Dieter Reuter @quintus23m
Senior Consultant, SEAL Systems
Stefan Scherer @stefscherer
Software Engineer, SEAL Systems
Building a Docker Swarm on ARM

Hardware
Pi’s
Switch
Power supply
Agenda
Software
SD card image
flash
docker-machine
Demo
docker-compose

• Education
• Learning cloud principles
• Hands-on cluster
Why?

Parts list
For a four node swarm.
• 4x Raspberry Pi 2/3
• 5 port switch with 5V
• 6x USB power supply
• Distant bolts 2.5mm
• 4x micro USB cable
• 1x 5.5mm barrel USB
• 4x flat patch cable
• 4x Micro SD cards
Build Hardware

Building a Docker Swarm cluster on ARM by Dieter Reuter and Stefan Scherer

HypriotOS
Get started with Docker
in 5 minutes

For Linux and Mac
Small helper script to do all these in just
one command:
• Download the SD card image
• Uncompress it
• Set hostname for device
• Set WiFi
Flash HypriotOS

Flash HypriotOS
$ flash
--hostname swarm01
--ssid wifipsk
--password wifipwd
https://downloads.hypriot.com/hypriotos-rpi-v0.8.0.img.zip
# repeat for swarm02..swarm04

Add your SSH key
$ ssh-keygen -R swarm01.local 
$ ssh-copy-id pirate@swarm01.local
# repeat for swarm02..swarm04

… some plumbing code
$ export TOKEN=$(for i in $(seq 1 32); do echo -n $(echo
"obase=16; $(($RANDOM % 16))" | bc); done; echo)
$ function getip() { (traceroute $1 2>&1 | head -n 1 |
cut -d( -f 2 | cut -d) -f 1) }

Create Docker Swarm manager
$ docker-machine create -d generic
--engine-storage-driver=overlay --swarm --swarm-master
--swarm-image hypriot/rpi-swarm:latest
--swarm-discovery="token://$TOKEN"
--generic-ip-address=$(getip swarm01.local)
swarm01

Create Docker Swarm nodes
$ docker-machine create -d generic
--engine-storage-driver=overlay --swarm
--swarm-image hypriot/rpi-swarm:latest
--swarm-discovery="token://$TOKEN"
--generic-ip-address=$(getip swarm02.local)
swarm02
# repeat for swarm03..swarm04

Control your Docker Swarm
$ eval $(docker-machine env --swarm swarm01)
$ docker info

• Self made cluster
• Pico Cluster
• BitScope Blade
Your ideas?
Examples

• http://blog.hypriot.com
• http://picocluster.com
• http://bitscope.com/product/blade
Further Links

Thank you!
@quintus23m
@stefscherer
@HypriotTweets

This document discusses storage options for Docker containers and images. It introduces different storage drivers that can be used, such as AUFS, BTRFS, DeviceMapper, Overlay, and Overlay2. It also discusses using persistent storage for containers through creating named Docker volumes stored on NFS, BTRFS, or Gluster file systems. This allows stateful applications to have their data persisted even if containers are removed.

The Dockerfile Explosion and the Need for Higher Level Tools by Gareth RushgroveDocker, Inc.

Dockerfiles are great. They provide a zero-barrier-to-entry format for describing a single Docker image which is immediately clear to anyone reading them. But with that simplicity comes problems that become apparent as your adoption of Docker gathers pace. * Dockerfiles can inherit from other docker images, but images are not Dockerfiles * Dockerfile provides no built-in mechanism for creating abstractions, so as usage grows identical or similar instructions can be duplicated across many files * The Docker APi exposes a build endpoint, but the API is very course, taking Dockerfile as the transport rather than exposing the individual instructions * Dockerfiles are just that, files. So they can come from anywhere The one layer per line in a Dockerfile limitation can lead to an explosion of layers, which fail to take advantage of the promised space and performance benefits.

DockerCon 2016 Ecosystem - Everything You Need to Know About Docker and Stora...ClusterHQ

Docker volumes allow storing state from containers outside of the image layer for persistence. They can be local storage or enabled for external storage management using plugins. This provides high availability for data by allowing the data to move with containers during failures or maintenance. The document discusses key concepts around stateful vs stateless containers and volumes. It also demonstrates creating and managing volumes using Docker Volume plugins in UCP for shared storage and failover capabilities.

Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart Docker, Inc.

Riot builds a lot of software. At the start of 2015 we were looking at 3000 build jobs over a hundred different applications and dozens of teams. We were handling nearly 750 jobs per hour and our build infrastructure needed to grow rapidly to meet demand. We needed to give teams total control of the “stack” used to build their applications and we needed a solution that enabled agile delivery to our players. On top of that, we needed a scalable system that would allow a team of four engineers to support over 250. After as few explorations, we built an integrated Docker solution using Jenkins that accepts docker images submitted as build environments by engineers around the company . Our “containerized” farm now creates over 10,000 containers a week and handles nearly 1000 jobs at a rate of about 100 jobs an hour. In this occasionally technical talk, we’ll explore the decisions that led Riot to consider Docker, the evolutionary stages of our build infrastructure, and how the open source and in-house software we combined to achieve our goals at scale. You’ll come away with some best practices, plenty of lessons learned, and insight into some of the more unique aspects of our system (like automated testing of submitted build environments, or testing node.js apps in containers with Chromium and xvfb).

You Don't Have to Start Over! A Practical Guide for Adopting Docker in the En...Docker, Inc.

So you are looking to adopt docker, but receive feedback and commentary such as "our development pipeline won't support containers" or "the applications aren't micro services, so I don't see a benefit." You are not alone, these and other statements are common misconceptions when considering using docker in the enterprise. Perhaps having a real enterprise use case with some tips on objection handling would support your goal of adopting docker in your organization? In this presentation, Chris Ciborowski, CEO and Principal Consultant at Nebulaworks and Docker Captain will discuss ways that you can leverage docker in existing enterprise environments providing tangible benefits to both developers and operations teams and accelerate DevOps adoption. He will also provide a few insider tips on objection handling learned while working on enterprise container adoption in enterprise clients.

DockerCon EU 2015: Speed Up Deployment: Building a Distributed Docker Registr...Docker, Inc.

This document discusses how to build a distributed Docker registry at scale. It describes the basic Docker registry setup and how to scale globally by using multiple registry instances across different locations, load balancing requests between them, and replicating registry data between instances using SnapMirror for high availability and disaster recovery. The full Nginx configuration for load balancing registry requests is also provided.

Docker serverless v1.0Thomas Chacko

Serverless computing is a cloud-native paradigm where developers build and run applications without managing infrastructure. It involves short-running, stateless functions that are triggered by events. With serverless, applications automatically scale up or down based on usage, and customers only pay for the compute time used. The document discusses serverless offerings from various cloud providers, demos serverless architectures using Docker containers, and notes serverless is well-suited for event-driven workloads like mobile backends and IoT but not long-running stateful processes.

DockerCon SF 2015: Orchestration for SysadminsDocker, Inc.

What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea LuzzardiMike Goelzer

Docker 1.12 introduces several new features for managing containerized applications at scale including Docker Swarm mode for native clustering and orchestration. Key features include services that allow defining and updating distributed applications, a built-in routing mesh for load balancing between nodes, and security improvements like cryptographic node identities and TLS encryption by default. The document also discusses plugins, health checks, and distributed application bundles for declaring stacks of services.

The Good, the Bad and the Ugly of Networking for Microservices by Mathew Lodg...Docker, Inc.

Ugly connectivity challenges in your development and production environments. The good: Advances in container networking in the past year, including the emergence of “Micro SDNs” as the way to simplify Docker deployments The bad: Understanding live network behavior and troubleshooting The ugly: Multicast, security, robustness and resiliency at scale In this talk you will learn how to weave Dockerized microservices together without tying yourself in knots or putting your head in a noose. You’ll learn how to effectively use micro SDNs, service discovery and request routing. You'll also see how to solve the bad and the ugly connectivity challenges in your development and production environments.

Configuration Management and Transforming Legacy Applications in the Enterpri...Docker, Inc.

Share the continuity of Société Générale's journey with Docker Enterprise from different points of view, from executives to devops, with CD platform as an enabler. Creating a Dockerfile that runs a container on a developer's laptop is pretty straightforward. But extending that to stacks of containers running on a dozen environments (development, integration, testing, staging, production, etc.) with different configuration and topologies can be a challenge. This talk will cover aspects of our journey to Docker Enterprise: What configuration should go in an image? Where to put different types of configuration? Images, environment variables, entrypoint, ...? How to store assets for building images and configuration for deployment in version control. We will discuss how Société Générale has implemented these, and what we plan next for Docker Enterprise deployment.

How to be successful running Docker in ProductionDocker, Inc.

John’s presentation will cover his lessons learned from running Docker in Production @ SalesforceIQ. Learn how to scale your registry using AWS and S3. Should you use Device Mapper or AUFS? Why run Swarm, Mesos, Kubernetes, or neither. Finally, know how persistent storage (Kafka, Cassandra, or SQL) can be run successfully with Docker in Production His team focuses on Docker based solutions to power their SaaS infrastructure and developer operations.

OSCON: Incremental Revolution - What Docker learned from the open-source fire...Docker, Inc.

Since Solomon Hykes unveiled Docker at the PyCon conference three years ago, containers have revolutionized how developers and ops teams build, ship, and run applications. Solomon explores the past, present, and future of our container ecosystem and shares lessons learned from managing successful open source projects across several dimensions: technology, people, products, and business. Sign up for the Docker for Mac and Windows beta: beta.docker.com

Taking Docker to Production: What You Need to Know and DecideDocker, Inc.

This document provides advice on taking Docker to production. It recommends starting simply by focusing on Dockerfiles and containerizing existing applications before complex orchestration. It also warns against common anti-patterns like using the "latest" tag or trapping data in containers. The document outlines sample swarm architectures and tech stacks and notes that outsourcing non-critical components can simplify operations. It closes by suggesting that an orchestrator may not always be needed and that running one container per VM is a valid approach.

Docker and CloudStackSebastien Goasguen

- Docker can be integrated with CloudStack in several ways, including running Docker in CloudStack virtual machine guests, packaging CloudStack as Docker containers, and using Docker orchestrators to manage containers. - CloudStack could potentially be re-architected to run its components in Docker containers and use Docker networking for isolation, with an orchestrator like Mesos or Kubernetes managing the CloudStack application. - There are open questions around whether CloudStack or other orchestrators should schedule virtual machines or containers as the primary compute resource in a private cloud data center.

Docker Container As A Service - March 2016Patrick Chanezon

This document discusses Docker Containers as a Service (CaaS). It begins by showing how Docker can be used to build a software layer on top of the internet hardware layer. It then discusses how Docker can be used with different cloud platforms and orchestration tools. It presents Docker CaaS as addressing the needs of both developers and IT operations by supporting the full application lifecycle across any infrastructure or operating system. Key characteristics of Docker CaaS include open APIs, pluggable architecture, and broad ecosystem support.

runC: The little engine that could (run Docker containers) by Docker Captain ...Docker, Inc.

With the announcement of the OCI by Solomon Hykes at last summer's DockerCon, a Docker-contributed reference implementation of the OCI spec, called runC, was born. While some of you may have tried runC or have a history of poking at the OS layer integration library to Linux namespaces, cgroups and the like (known as libcontainer), many of you may not know what runC offers. In this talk Phil Estes, Docker engine maintainer who has also contributed to libcontainer and runC, will show what's possible using runC as a lightweight and fast runtime environment to experiment with lower-level features of the container runtime. Phil will introduce a conversion tool called "riddler", which can inspect and convert container configurations from Docker into the proper OCI configuration bundle for easy conversion between the two environments. He'll also demonstrate how to make custom configurations for trying out security features like user namespaces and seccomp profiles.

Docker worshop @Twitter - How to use your own private registrydotCloud

DockerCon SF 2015: Orchestration for Devs (machine + compose)Docker, Inc.

This document discusses Docker Machine and Docker Compose. Docker Machine allows provisioning and management of Docker hosts on various cloud providers and virtualization systems. Docker Compose defines and runs multi-container applications using a YAML file. The latest versions of both tools feature improved performance, stability, and configuration options as well as experimental features like smart container recreation in Compose. Demostrations of Docker Machine and Compose are provided.

What's New in Docker 1.12?Ajeet Singh Raina

Docker Engine 1.12 can be rightly called ” A Next Generation Docker Clustering & Distributed System”. Though Docker Engine 1.12 Final Release is around corner but the recent RC3 brings lots of improvements and exciting features. One of the major highlight of this release is Docker Swarm Mode which provides powerful yet optional ability to create coordinated groups of decentralized Docker Engines. Swarm Mode combines your engine in swarms of any scale. It’s self-organizing and self-healing. It enables infrastructure-agnostic topology.The newer version democratizes orchestration with out-of-box capabilities for multi-container on multi-host app deployments.

ContainerDayVietnam2016: Django Development with DockerDocker-Hanoi

Cuong Tran presented on Django development with Docker. The presentation covered: 1. Introduction to the Django Docker stack including Nginx, Django, Postgres, and Redis. 2. How to run the Django stack using Docker Compose including building, starting containers, and migrating data. 3. Common activities like running commands in containers, updating code in Git, and rebuilding Docker images. 4. Problems and solutions like handling Docker stop signals gracefully, ensuring proper startup order, and optimizing the Docker build process. 5. Useful Docker snippets for stats, removing containers/images, and saving/loading images.

Container Days Boston - Kubernetes in productionMike Splain

Kubernetes in Production, From the Ground Up discusses setting up a Kubernetes cluster from scratch using CoreOS, etcd, Docker, and Terraform. The document outlines setting up etcd for high availability using an autoscaling group, configuring the Kubernetes master nodes to run the API server, scheduler, and controller manager as pods, and deploying worker nodes that run the kubelet to join the cluster. The process involves using cloud-init scripts, Terraform, and container images to automate the installation and configuration of all cluster components in a scalable and resilient way.

Kubernetes on CloudStack with coreOSSebastien Goasguen

Hashicorp: Delivering the Tao of DevOpsRamit Surana

Docker and stuffVarun Sharma

TIAD 2016 : Migrating 100% of your production services to containersThe Incredible Automation Day

The document summarizes BlaBlaCar's journey to migrating 100% of their production services to containers. It describes how they started with bare metal servers and evolved to using configuration management tools like Chef. They then standardized on CoreOS and rkt as their container platform. Key tools they developed include dgr for building container images and ggn for managing services running in containers. They also implemented service discovery using custom tools like Nerve and Synapse. The document shares many lessons learned from their large scale production container deployment.

DCSF19 CMD and Conquer: Containerizing the Monolith Docker, Inc.

Tony Lee & Nelson Wang, Splunk Modern microservice-oriented software architectures evangelize the principles of infrastructure-as-code and declarative directives to manage and run applications. At Splunk, we wanted to marry these ideals with the majestic monolith, Splunk Enterprise, to simplify the use of our product through containerization. Without rearchitecting the entire product from the ground-up, which can be a costly investment, we focused on incorporating a flexible configuration management layer on top of the core application. This has enabled us to make running Splunk in Docker act and behave as a true microservice, greatly reducing the friction of migrating towards more container-native software. We not only concentrated on making our open-source Docker image initiative user-friendly and production-ready, but we also wanted to seamlessly integrate it back into our internal engineering process. Join us for this session as we discuss migrating a traditional application into a microservice ecosystem, developing a containerization strategy for both external customer usage and internal development, as well as learning about our internal container platform at scale.

DockerCon EU 2015: The Glue is the Hard Part: Making a Production-Ready PaaSDocker, Inc.

PaaSTA is Yelp's open-source Docker-based PaaS that glues together common PaaS components like scheduling with Mesos/Marathon, service delivery with Docker, discovery with SmartStack, and monitoring with Sensu. The document discusses how PaaSTA provides a production-ready platform by using stable components, reducing single points of failure, enabling graceful degradation, making upgrades painless, and incorporating self-healing and alerting capabilities. Lessons learned include the importance of interfaces, maintaining the app-infra boundary, using the right abstractions, and making iterative improvements.

IThome DevOps Summit - IoT、docker與DevOpsSimon Su

How to build a Distributed Serverless Polyglot Microservices IoT Platform us...Animesh Singh

When people aren't talking about VMs and containers, they're talking about serverless architecture. Serverless is about no maintenance. It means you are not worried about low-level infrastructural and operational details. An event-driven serverless platform is a great use case for IoT. In this session at @ThingsExpo, Animesh Singh, an STSM and Lead for IBM Cloud Platform and Infrastructure, detailed how to build a distributed serverless, polyglot, microservices framework using open source technologies like: OpenWhisk: Open source distributed compute service to execute application logic in response to events Docker: To run event driven actions 6. Ansible and BOSH: to deploy the serverless platform MQTT: Messaging protocol for IoT Node-RED: Tool to wire IoT together Consul: Tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable. Kafka: A high-throughput distributed messaging system. StatsD/ELK/Graphite: For statistics, monitoring and logging

More Related Content

What's hot (20)