Your Auto-Scaling Bot - Volkan Tufecki

May 4, 20172 likes894 views

In this talk we will talk about a docker swarm architecture that scales automatically with the help of a Slack Bot. The talk will include - Docker Swarm and Docker Compose - Monitoring containers by cadvisor - Managing alerts with promotheus and alert manager - Running a slack bot that decides to deploy or undeploy services - Generating load with siege

Your Auto-Scaling Bot
Volkan Tufekci
DevOps Eng.,
eBay Turkey - GittiGidiyor

• DevOps Eng 2+ years
• Backend Dev. 6+ years
Who am I?

• 15M Subscriber
• 38M Visitor/Month
• In every 2 secs a product
is sold
eBay TR -
GittiGidiyor

• When we hit the thresholds
• When there is a push notification
• When we employ more services
Problem
We need to scale

• Container scaling
• Host(Node) scaling*
Problem
What kind of scaling?
* We can’t use public cloud services

We can dedicate someone to monitor the load on our services and
perform ”docker scale”
or
We can use a bot to do the stuff automatically for us
Solution

Ingredients
• Docker Swarm Mode
• Node Exporter
• cAdvisor
• Prometheus
• Alert Manager
• Slack + Python Bot
• Grafana(not mandatory but nice to have)
Recipe

Official Prometheus exporter
for node metrics
Node Exporter

Container resource usage and
performance metrics
cAdvisor

Time series data based
monitoring and alerting
solution
Prometheus

Prometheus
• scrape_interval
• evaluation_interval
• states:
• inactive
• pending
• firing
Under the Hood

For deduplicating, grouping,
and routing alarms to the
correct receiver integration
such as Slack
Alert Manager

Under the Hood
Alert Manager
• group_by
• group_wait
• group_interval

Open source software for
time series analytics.
Graphical visualization
solution.
Grafana

• Parse the notification
• If it is ”FIRING” alarm from a service
• Get current # of tasks
• Check if we are below upper
threshold
• If so, increase scale
• If not, notify
Algorithm
Scaling up the service

• Parse the notification
• If it is ”RESOLVED” notification
• For each task
• Get current # of tasks
• Check if we are above
lower threshold
• If so, decrease scale
Algorithm
Scaling down the services

Thank You!
@volkantufekci
@docker
#dockercon

Learn from the development team as we dive into some of the latest and upcoming features in Docker EE, our enterprise container management solution. We will focus on the architecture and configuration of the features and how they can be used with both modern apps and containerized legacy apps. Stay for some tips on monitoring and troubleshooting to help you prevent your production environment from going sideways.

Dell Trials and Triumphs using Docker on Client Systems by Sean McGinnis and ...Docker, Inc.

Docker for Ops - Scott Coulton, PuppetDocker, Inc.

In this talk, Scott Coulton will take you through Docker's cluster solution Swarm mode with his operations hat on. We will start from the beginning by describing what swarm mode is, what it does, and how it works behind the scenes. From there, we will look at very basic configurations of Swarm mode from the point of view of the operations team as well as a production-ready workflow including deployments of the cluster, logging and CD best practices. Attendees will be able to apply their learnings to their use cases.

DockerCon EU 2015: Placing a container on a train at 200mphDocker, Inc.

Presented by Casper S. Jensen, Software Engineer, Uber At Uber, we've been introducing Docker to give service owners more control over their environments. However, everything at Uber is moving very fast so we have had to do it a way such that Docker fitted into the existing infrastructure and services could be migrated seamlessly to Docker without any service interruptions. In this talk we will talk about the challenges we faced while doing this, such as handling both non-Docker and Docker builds, image replication, integration with our deployment systems and other challenges when deploying Docker at scale.

DCEU 18: Docker Containers in a Serverless WorldDocker, Inc.

Jules Testard - Software Engineer, Docker Inc Since the advent of AWS Lambda in 2014, the Function as a Service (FaaS) programming paradigm has gained a lot of traction in the cloud community. Since then, interest has increased for developers and entreprises to build their own open source solutions on top of Kubernetes. A number of competing frameworks in this space have been developed. In this talk, we will look at three specific frameworks (OpenFaas, Nuclio and FN) and for each framework we will: Show how to create, deploy, and invoke a function using that framework Show how Docker images and containers are used by each framework under the hood Investigate how the frameworks leverage KNative to build, ship and run applications on Kubernetes

Node.js Rocks in Docker for Dev and OpsBret Fisher

This document discusses best practices for building Node.js applications in Docker containers. It covers topics like using multi-stage Docker builds to avoid packaging devDependencies in production images, properly handling process shutdown in Node.js, and using Docker Compose for local development with features like auto-restart on file changes and dependency management between linked services. The overall goal is to help build production-ready Node.js images and development environments that are optimized for speed, size and security.

DCEU 18: App-in-a-Box with Docker Application PackagesDocker, Inc.

Michael Irwin - Application Architect, Virginia Tech Docker Application Packages is an experimental tool that makes it easy to share multi-service applications. Create a Compose file, package it in an image, and voilà! You now have an "app-in-a-box"! Not convinced yet? No worries! It took a while for me to be convinced too! In this session, we'll start off by diving into how Docker Application Packages actually works, which will help us understand the use cases. We'll see how dev environments can hook in to this app-in-a-box by replacing the service being worked on with a dev container. Then we'll move on to see how end-to-end functional tests are much easier to run. And, finally, we'll see how to maintain an "app-in-a-box" with the latest versions of each component in a CI/CD pipeline, allowing for a unique app-in-a-box for each feature branch under development. Lots of good material! And lots of live demos!

DCSF19 How To Build Your Containerization Strategy Docker, Inc.

Lee Namba, Docker The Docker Enterprise container platform helps organizations deploy and manage applications faster and it secures the application pipeline at a lower cost than traditional application delivery models. But it takes more than just great technology to achieve the desired results. The organization and culture of your enterprise directly impacts what you transform, how it’s done, and who does it. Success requires a strategy for how you will govern the container platform environment, how to assess your application estate, what your delivery pipeline will look like, and how to ensure developers, operators, security teams and others play nicely together. In this talk I will cover topics such as different types of workloads (legacy, microservices, FaaS, big data and more), how your org chart can influence whether you deploy CaaS (Containers as a Service) vs CLaaS (Clusters as a Service), how "shifting left" can determine if you can outsource, centralized vs distributed CI/CD and how containers play a role, transforming your pets into cattle, how giant whale balloons are used for onboarding, and a prescriptive and comprehensive methodology for successfully deploying containers into your enterprise.

Docker for .NET Developers - Michele Leroux Bustamante, SollianceDocker, Inc.

This document discusses how Docker can help .NET developers by allowing them to containerize their .NET applications. It provides benefits like running applications across Windows and Linux containers, improving development workflows through features of Docker for Windows like Linux containers on Windows, and facilitating continuous integration and deployment of containerized applications through automation. The document provides examples of Dockerfiles for different types of .NET applications and demonstrates how to add Docker support and containerize existing applications using Visual Studio tools.

Docker Online Meetup: Announcing Docker CE + EEDocker, Inc.

Docker Community Edition (CE) and Enterprise Edition (EE) are the best expressions of the Docker Platform to date. Whether you’re a developer, an ops team or a enterprise IT-team member, and no matter the infrastructure, Docker CE and EE gives you a way to install, upgrade and maintain Docker with the support and assurances required for your particular workload. Both Docker CE and EE are available on a wide range of popular operating systems (including Windows Server 2016) and cloud infrastructure. Developers and devOps have the freedom to run Docker on their favorite infrastructure without risk of lock-in. Michael Friis will give an overview of both editions and highlight the big enhancements to the lifecycle, maintainability and upgradability of Docker.

Taking Docker from Local to Production at Intuit JanJaap Lahpor, Intuit and H...Docker, Inc.

In this talk, we will share how a small team at Intuit moved Docker from local to production serving real and critical workloads. We will share how we addressed the organization challenges of running Docker at large enterprises by building a business case for a pilot project to prove the value of containers and its real world application. Next, we will share how we solved the technical challenges that present themselves when taking Docker from local to production in a corporate data center. We will share the blueprint for the business case and the associated pilot which laser focused on running stateless back-end services throughout the entire SDLC. Finally, we will highlight our crawl-walk-run approach that allowed us to make inexpensive mistakes before investing in the right areas as our Docker knowledge increased. We will share the major technical issues we encountered, how we overcame them and the lessons we learned.

Docker?!?! But I'm a SysAdminDocker, Inc.

Your developers just walked into your cube and said: "Here's the new app, I built it with Docker, and it's ready to go live." What do you do next? In this session, we'll talk about what containers are and what they are not. And we'll step through a series of considerations that need to be examined when deploying containerized workloads - VMs or Container? Bare Metal or Cloud? What about capacity planning? Security? Disaster Recovery? How do I even get started?

Reduce DevOps Friction with Docker & Jenkins by Andy Pemberton, CloudbeesDocker, Inc.

DockerCon SF 2015: Docker in the New York Times NewsroomDocker, Inc.

The document discusses how the New York Times uses Docker in its newsroom. Some key points: - Docker is used to deploy over 300 micro-applications across multiple servers for density, speed, and true versioning. - Docker provides consistency through immutable containers and distributed applications. It allows for horizontal scaling. - Common services like configuration, discovery, routing and scheduling are abstracted out into separate micro-applications on GitHub like Remora and Promise. - Docker containers provide consistency while applications are heterogeneous. Future areas of focus include improving notifications, secrets management, image registry, and automated building.

How to build your containerization strategyDocker, Inc.

The Docker Enterprise Edition platform helps customers deploy and manage applications faster and it secures the application pipeline at a lower cost than traditional application delivery models. But it takes more than just great technology to achieve the desired results. The organization and culture of your enterprise directly impacts what you transform, how it’s done, and who does it. Success requires a strategy for how you will govern the Docker EE container platform, how to assess your application estate, what your delivery pipeline will look like, and how to ensure developers, operators, security teams and others play nicely together. In this talk I will cover topics such as different types of workloads (legacy, microservices, FaaS, big data, ...), how your org chart can influence whether you deploy a CaaS (Containers as a Service) vs CLaaS (Clusters as a Service), how "shifting left" can determine if you can outsource, centralized vs distributed CI/CD and how containers play a role, transforming your pets into cattle, how giant whale balloons are used for onboarding, and a prescriptive and comprehensive methodology for successfully deploying Docker in your enterprise.

Docker presentationWes Eklund

Docker provides a standard way to package applications into containers that are portable and can run on any infrastructure. The containers isolate applications from one another and ensure that dependencies and configurations are included so applications always run the same. This allows developers to build applications once and deploy them anywhere without worrying about compatibility or missing dependencies. It also allows operators to configure applications once and run anything on their infrastructure efficiently by eliminating inconsistencies between environments.

Efficient Parallel Testing with Docker by Laura FrankDocker, Inc.

Fast and efficient software testing is easy with Docker. We often use containers to maintain parity across development, testing, and production environments, but we can also use containerization to significantly reduce time needed for testing by spinning up multiple instances of fully isolated testing environments and executing tests in parallel. This strategy also helps you maximize the utilization of infrastructure resources. The enhanced toolset provided by Docker makes this process simple and unobtrusive, and you’ll see how Docker Engine, Registry, Machine, and Compose can work together to make your tests fast.

Docker Store: The New Destination for Enterprise Software - Lily Guo and Alfr...Docker, Inc.

Docker Store is the place to find trusted community and enterprise content. Independent software vendors, startups and developers alike now have a marketplace to create and distribute Enterprise-ready content through the Store. Join Chinmayee and Alfred from the Docker Store team to learn how ISVs, big and small, are using Docker Store. They will cover: How to publish and distribute high quality, reusable containers and plugins; and How Enterprise customers can simplify procurement and management of their software assets using Docker Store.

Making it Easier to Contribute to Open Source Projects Using Docker Container...Docker, Inc.

Android Meets DockerDocker, Inc.

How does awesome Android benefit from dazzling Docker? With an introduction to the Docker way of continuous integration for Android, it would help you get a consistent Android development and build environment with minimal effort. Syllabus: * Background story - where the problem happened * Why Android dev should care? * How to provision an Android environment to build project usually? How Docker is gonna solve it? * Typical machine dependant failing unit tests examples, which could be revealed by using Docker * Philosophy, features, improvements, comparison against traditional approach * Limitations * Quick setup of Jenkins using AndroidSDK docker slaves (live demo)

Automated hardware testing using docker for spaceDocker, Inc.

Two things are for certain – space is hard, and Docker is not just for web content! Space software development traditionally lags behind state of the art software process for good reason – our missions are long (7+ years), we run on highly constrained embedded hardware, and the software cannot fail. Docker, along with a devops mentality, has helped us create a scalable, parallelizable and rapidly deployable test infrastructure for DART, NASA’s mission to hit an asteroid at 6 km/s. During the presentation, we will walk through how our dev cycle has changed from a human based testing system to an automated one. We will outline how we are using Docker (and NASA Goddard’s Core Flight Executive) for both our embedded development environment and our scalable test environment. Next, we will discuss what deployment means to us (and how different it is from web deployment). Lastly, we will explore lessons learned on how our hardware-centric testing approach was adapted into a software-based approach: what worked, what didn’t, what we wish we could do someday. How can you help? We are new to Docker. We are excited to share our experiences and hear from the Docker community on our use cases, technological hurdles that we faced, our solutions to these problems, and how we can harness Docker to the fullest extent.

Java in a world of containersDocker, Inc.

"Container technologies such as Docker are rapidly becoming the de-facto way to deploy cloud applications, and Java is committed to being a good container citizen. This session will explain how OpenJDK fits into the world of containers, specifically how it fits with Docker images and containers. The session will focus on the production of optimized Docker images containing a JDK. We will introduce technologies such as jlink, that can be used to reduce the size of the created image. The session will explain Alpine/musl support for an effective image and runtime. The session will also talk about and the inclusion of Class Data Sharing (CDS) archives and Ahead of Time (AOT) shared object libraries for improving startup time. The attendees will learn about the recent work that has gone into OpenJDK for interacting with container resource limitations."

Learning the Alphabet: A/B, CD and [E-Z] in the Docker Datacenter by Brett Ti...Docker, Inc.

What is the right balance between moving fast, innovating, experimenting with new technology, and protecting the personal data of our customers and interests of our stakeholders? How can we safely try new ideas in production without risking costly downtime? Does the utopia where developers are free from lock-in and operators enjoy the calm of a steadily running system exist in the real world? Is it possible to have open platforms with better security? At Kroger Digital we are still working through these questions every day but are redesigning our systems with the goals of true operational maturity and security. Discover how we are building capabilities for monitoring, A/B testing, and continuous delivery with Docker Datacenter, plugins, and open source building blocks such as NGiNX, ElasticSearch, and more.

Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment ModelDocker, Inc.

Micro Focus uses Docker Hub at scale to support its software delivery and deployment model. Some key points: - Docker Hub is used as the registry service for Micro Focus container images - It allows for optimized, secure, reliable and cost-effective software delivery through deployments and updates of container images to customers and partners - Micro Focus leverages features like private repositories, offline/online access, signing and scanning of images, and integration with CI/CD pipelines - Over 1,650 organizations, 450 repositories, and 18 teams are used on Docker Hub to manage access and deliver software from Micro Focus

Troubleshooting tips from docker support engineersDocker, Inc.

Docker makes everything easier. But even with the easiest platforms, sometimes you run into problems. In this session, you'll learn first-hand from someone whose job is helping customers fix these problems. In this session, you'll learn how to: Ask better questions & identify the real problems you want to solve Use command line tools to inspect internal Swarm and Kubernetes components Debug and troubleshoot interactions between Enterprise Edition components Troubleshoot Docker Enterprise Edition Windows nodes Apply these skills to troubleshooting Docker Enterprise Edition

Docker for Mac and Windows: The Insider's Guide by Justin CormackDocker, Inc.

Docker for Mac and Windows were released in beta in March, and provide lots of new features that users have been clamouring for including: file system notifications, simpler file sharing, and no Virtualbox hassles. During this talk, I will give the inside guide to how these products work. We will look at all the major components and how they fit together to make up the product. This includes a technical deep dive covering the hypervisors for OSX and Windows, the custom file sharing code, the networking, the embedded Alpine Linux distribution, and more.

DockerCon SF 2015: DHE/DTRDocker, Inc.

Docker Trusted Registry is a private registry for Docker images that provides key features like role-based access control, audit logs, and storage drivers. It has a modular architecture with components like registry servers, load balancers, and authentication servers. It can be deployed on any infrastructure and supports integration with directory services. Docker Trusted Registry comes in different editions for teams of various sizes and needs and pricing starts from free to $150 per month depending on the edition.

DockerCon EU 2015: Deploying and Managing Containers for DevelopersDocker, Inc.

Continuous Delivery: releasing Better and Faster at DashlaneDashlane

Detecting secrets in code committed to gitlab (in real time)Chandrapal Badshah

More Related Content

What's hot (20)

Docker for .NET Developers - Michele Leroux Bustamante, SollianceDocker, Inc.

Docker Online Meetup: Announcing Docker CE + EEDocker, Inc.

Taking Docker from Local to Production at Intuit JanJaap Lahpor, Intuit and H...Docker, Inc.

Docker?!?! But I'm a SysAdminDocker, Inc.

Reduce DevOps Friction with Docker & Jenkins by Andy Pemberton, CloudbeesDocker, Inc.

DockerCon SF 2015: Docker in the New York Times NewsroomDocker, Inc.

How to build your containerization strategyDocker, Inc.

Docker presentationWes Eklund

Efficient Parallel Testing with Docker by Laura FrankDocker, Inc.

Docker Store: The New Destination for Enterprise Software - Lily Guo and Alfr...Docker, Inc.

Making it Easier to Contribute to Open Source Projects Using Docker Container...Docker, Inc.

Android Meets DockerDocker, Inc.

Automated hardware testing using docker for spaceDocker, Inc.

Java in a world of containersDocker, Inc.

Learning the Alphabet: A/B, CD and [E-Z] in the Docker Datacenter by Brett Ti...Docker, Inc.

Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment ModelDocker, Inc.

Troubleshooting tips from docker support engineersDocker, Inc.

Docker for Mac and Windows: The Insider's Guide by Justin CormackDocker, Inc.

DockerCon SF 2015: DHE/DTRDocker, Inc.

DockerCon EU 2015: Deploying and Managing Containers for DevelopersDocker, Inc.

Docker for .NET Developers - Michele Leroux Bustamante, SollianceDocker, Inc.

Docker Online Meetup: Announcing Docker CE + EEDocker, Inc.

Taking Docker from Local to Production at Intuit JanJaap Lahpor, Intuit and H...Docker, Inc.

Docker?!?! But I'm a SysAdminDocker, Inc.

Reduce DevOps Friction with Docker & Jenkins by Andy Pemberton, CloudbeesDocker, Inc.

DockerCon SF 2015: Docker in the New York Times NewsroomDocker, Inc.

How to build your containerization strategyDocker, Inc.

Docker presentationWes Eklund

Efficient Parallel Testing with Docker by Laura FrankDocker, Inc.

Docker Store: The New Destination for Enterprise Software - Lily Guo and Alfr...Docker, Inc.

Making it Easier to Contribute to Open Source Projects Using Docker Container...Docker, Inc.

Android Meets DockerDocker, Inc.

Automated hardware testing using docker for spaceDocker, Inc.

Java in a world of containersDocker, Inc.

Learning the Alphabet: A/B, CD and [E-Z] in the Docker Datacenter by Brett Ti...Docker, Inc.

Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment ModelDocker, Inc.

Troubleshooting tips from docker support engineersDocker, Inc.

Docker for Mac and Windows: The Insider's Guide by Justin CormackDocker, Inc.

DockerCon SF 2015: DHE/DTRDocker, Inc.

DockerCon EU 2015: Deploying and Managing Containers for DevelopersDocker, Inc.

Similar to Your Auto-Scaling Bot - Volkan Tufecki (20)

Continuous Delivery: releasing Better and Faster at DashlaneDashlane

Detecting secrets in code committed to gitlab (in real time)Chandrapal Badshah

Kubernetes and Gitops at Quicksign - Koncrete Kube Oct 2018Cedric Vidal

Ship code like a keptnRob Jahn

Keptn is an open-source project that provides tools to enable continuous delivery and automation for modern applications using Kubernetes. It allows developers to focus on code and DevOps teams to focus on tools rather than building custom pipelines. Keptn provides automated multi-stage delivery pipelines, automated quality gates, self-healing deployments, and enables zero-touch toolchain integration and updates. It also supports automated problem remediation in production for continuous operations. Keptn follows cloud-native design principles and provides a common way for organizations to achieve autonomous delivery and operations.

Gocd – Kubernetes/Nomad Continuous DeploymentLeandro Totino Pereira

GitOps - Modern best practices for high velocity app dev using cloud native t...Weaveworks

Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016ShuttleCloud

In this talk, we'll explain our journey from having near-zero monitoring to having all of our infrastructure monitored with the necessary metrics and alerts. We will share with the audience some of the mistakes we did and what lessons we have learned. We currently have around 200 instances monitored with a comfortable cost-effective in-house monitoring stack based on Prometheus. We want to demonstrate that you don't need to have a big fleet to embrace Prometheus and that it is a non-expensive solution for monitoring. ---------- ShuttleCloud is a small startup specialized in email and contacts migrations. We developed a reliable migration platform in high availability used by clients like Gmail, GContacts and Comcast. For example, Gmail alone has imported data for 3 million users with our API and we process hundreds of terabytes every month. ------------- Follow us on Twitter: @ShuttleCloud: https://twitter.com/ShuttleCloud @ShuttleCloudEng: https://twitter.com/ShuttleCloudEng ShuttleCloud.com

Making Wallstreet talk with GO (GO India Conference 2015)Matthew Campbell

Cloud Native Apps with GitOps Weaveworks

GitOps provides a way for organizations to adopt continuous delivery practices using cloud native technologies like Kubernetes and Docker. It advocates storing all infrastructure configurations and code in a Git repository to enable automated deployment and management of applications. Weaveworks adopted GitOps and was able to recover from a complete system failure in just 45 minutes by having their entire system defined as code. GitOps follows the principles of continuous delivery and observability by integrating deployment pipelines, monitoring, and security policies to drive faster, more reliable software releases.

Big feature - small sprint Igor Goldshmidt

My talk presentation from SQA Days 21 conference. During last two years I've been working in Agile teams and quite often have come across the situation when the feature is pretty big, but they still want to release it at the end of the sprint. That is in spite of the fact, that there is a great risk of violation of the timing and poor product quality. So what should we, quality guards, do when eventually everything depends on us? My report presents the analysis of the development stages from the testing point of view in such a difficult situation and provides tips for success.

PAC 2019 virtual Bruno Audoux Neotys

This document describes a performance automation solution using load testing scripts to continuously monitor application performance. The solution uses scripts to test functionality, availability, response times, and end-to-end workflows. Load testing engines run the scripts on a periodic schedule and store results. An alerting system analyzes results and sends alerts if response times exceed thresholds or tests fail to run. The system is containerized using Docker for scalability. Potential customers include project managers who need regression testing, monitoring of production applications, and emergency alerts about degradations or failures.

Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...Lean IT Consulting

Continuous Lifecycle London 2018 Event KeynoteWeaveworks

Today it’s all about delivering velocity without compromising on quality, yet it’s becoming increasingly difficult for organisations to keep up with the challenges of current release management and traditional operations. The demand for developers to own the end-to-end delivery, including operational ownership, is increasing. A “you build it, you own it” development process requires tools that developers know and understand. So I’d like to introduce “GitOps”- an agile software lifecycle for modern applications. In this session, I will discuss these industry challenges, including current CICD trends and how they’re converging with operations and monitoring. I’ll also illustrate the GitOps model, identify best practices and tools to use, and explain how you can benefit from adopting this methodology inherited from best practices going back 10-15 years.

AWS re:Invent 2013 - MBL303 Gaming Ops - Running High-performance Ops for Mob...Eduardo Saito

This document summarizes lessons learned from operating high-performance mobile gaming applications on AWS. It covers incident management, change management, auto-scaling, and cloud optimization tools. For incident management, it recommends using automatic paging, big status dashboards, and incident reviews. For change management, it suggests tools like Puppet and Jenkins to automate and track changes by responsible teams. On auto-scaling, it provides examples of on-demand and scheduled scaling and reducing bootstrap times. It also lists tools for cloud optimization like Trusted Advisor and in-house solutions.

Scaling Pinterest's MonitoringBrian Overstreet

This document summarizes Brian Overstreet's talk on scaling Pinterest's monitoring system over time as the company and traffic grew. It describes how Pinterest started with just Ganglia for system metrics and no application metrics. They introduced Graphite but faced challenges with packet loss and metrics being dropped. They then introduced OpenTSDB which users were happier with due to its querying speed. Pinterest developed an agent-based pipeline using Kafka and Storm to address packet loss issues and allow over 1.5 million points per second to be ingested by OpenTSDB. Key lessons included the need to educate users, control incoming metrics, and ensure the monitoring system scales with engineers rather than just site users.

Scaling Prometheus on Kubernetes with ThanosThomas Riley

Redundant devopsSzabolcs Szabolcsi-Tóth

Spring insight what just happenedBoulder Java User's Group

Continuous Build To Continuous Release - ExperienceRaja Soundaramourty

Free GitOps Workshop (with Intro to Kubernetes & GitOps)Weaveworks

View this video on Youtube here: https://youtu.be/tK4S8y3j5TA In this info-packed and hands-on workshop we covered:  Introduction to Kubernetes & GitOps talk: We covered the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.  Get Started with GitOps: You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for: - an opinionated repo structure to minimize decision fatigue - disaster recovery using GitOps - Helm charts example - Multi-cluster example - all with free and open source tools mostly in the CNCF (eg. Flux and Helm). If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)

Continuous Delivery: releasing Better and Faster at DashlaneDashlane

Detecting secrets in code committed to gitlab (in real time)Chandrapal Badshah

Kubernetes and Gitops at Quicksign - Koncrete Kube Oct 2018Cedric Vidal

Ship code like a keptnRob Jahn

Gocd – Kubernetes/Nomad Continuous DeploymentLeandro Totino Pereira

GitOps - Modern best practices for high velocity app dev using cloud native t...Weaveworks

Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016ShuttleCloud

Making Wallstreet talk with GO (GO India Conference 2015)Matthew Campbell

Cloud Native Apps with GitOps Weaveworks

Big feature - small sprint Igor Goldshmidt

PAC 2019 virtual Bruno Audoux Neotys

Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...Lean IT Consulting

Continuous Lifecycle London 2018 Event KeynoteWeaveworks

AWS re:Invent 2013 - MBL303 Gaming Ops - Running High-performance Ops for Mob...Eduardo Saito

Scaling Pinterest's MonitoringBrian Overstreet

Scaling Prometheus on Kubernetes with ThanosThomas Riley

Redundant devopsSzabolcs Szabolcsi-Tóth

Spring insight what just happenedBoulder Java User's Group

Continuous Build To Continuous Release - ExperienceRaja Soundaramourty

Free GitOps Workshop (with Intro to Kubernetes & GitOps)Weaveworks

More from Docker, Inc. (20)

Containerize Your Game Server for the Best Multiplayer Experience Docker, Inc.

Raymond Arifianto, AccelByte and Mark Mandel, Google - We have been deploying containerized micro-services for our Game Backend Services for a while. Now we are tackling the challenge to scale up fleets of game dedicated servers in multiple regions, multiple data centers and multiple providers - some in bare metal, some in Cloud. So we leverage docker containerization to deploy Game Servers to achieve Portability, Fast Deployment and Predictability, enabling us to scale up to thousands of servers, on demand, without a sweat.

How to Improve Your Image Builds Using Advance Docker BuildDocker, Inc.

Nicholas Dille, Haufe-Lexware + Docker Captain - Docker continues to be the standard tool for building container images. For more than a year Docker ships with BuildKit as an alternative image builder, providing advanced features for secret and cache management. These features help to make image builds faster and more secure. In this session, Docker Captain Nicholas Dille will teach you how to use Buildkit features to your advantage.

Build & Deploy Multi-Container Applications to AWSDocker, Inc.

Lukonde Mwila, Entelect - As the cloud-native approach to development and deployment becomes more prevalent, it's an exciting time for software engineers to be equipped on how to dockerize multi-container applications and deploy them to the cloud. In this talk, Lukonde Mwila, Software Engineer at Entelect, will cover the following topics: - Docker Compose - Containerizing an Nginx Server - Containerizing an React App - Containerizing an Node.JS App - Containerizing anMongoDB App - Runing Multi-Container App Locally - Creating a CI/CD Pipeline - Adding a build stage to test containers and push images to Docker Hub - Deploying Multi-Container App to AWS Elastic Beanstalk Lukonde will start by giving an overview of how Docker Compose works and how it makes it very easy and straightforward to startup multiple Docker containers at the same time and automatically connect them together with some form of networking. After that, Lukonde will take a hands on approach to containerize an Nginx server, a React app, a NodeJS app and a MongoDB instance to demonstrate the power of Docker Compose. He'll demonstrate usage of two Docker files for an application, one production grade and the other for local development and running of tests. Lastly, he'll demonstrate creating a CI/CD pipeline in AWS to build and test our Docker images before pushing them to Docker Hub or AWS ECR, and finally deploying our multi-container application AWS Elastic Beanstalk.

Securing Your Containerized Applications with NGINXDocker, Inc.

How To Build and Run Node Apps with Docker and ComposeDocker, Inc.

Kathleen Juell, Digital Ocean - Containers are an essential part of today's microservice ecosystem, as they allow developers and operators to maintain standards of reliability and reproducibility in fast-paced deployment scenarios. And while there are best practices that extend across stacks in containerized environments, there are also things that make each stack distinct, starting with the application image itself. This talk will dive into some of these particularities, both at the image and service level, while also covering general best practices for building and running Node applications with database backends using Docker and Compose.

Hands-on Helm Docker, Inc.

Distributed Deep Learning with Docker at SalesforceDocker, Inc.

Jeff Hajewski, Salesforce - There is a wealth of information on building deep learning models with PyTorch or TensorFlow. Anyone interested in building a deep learning model is only a quick search away from a number of clear and well written tutorials that will take them from zero knowledge to having a working image classifier. But what happens when you need to deploy these models in a production setting? At Salesforce, we use TensorFlow models to help us provide customers with insights into their data, and we do this as close to real-time as possible. Designing these systems in a scalable manner requires overcoming a number of design challenges, but the core component is Docker. Docker enables us to design highly scalable systems by allowing us to focus on service interactions, rather than how our services will interact with the hardware. Docker is also at the core of our test infrastructure, allowing developers and data scientists to build and test the system in an end to end manner on their local machines. While some of this may sound complex, the core message is simplicity - Docker allows us to focus on the aspects of the system that matter, greatly simplifying our lives.

The First 10M Pulls: Building The Official Curl Image for Docker HubDocker, Inc.

James Fuller, webcomposite s.r.o. - Curl is the venerable (yet very modern) 'swiss army knife' command line tool and library for transferring data with URLs. Recently we (the Curl team) decided to build a release for Docker Hub. This talk will outline our current development workflow with respect to the docker image and provide insights on what it takes to build a docker image for mass public consumption. We are also keen to learn from users and other developers how we might improve and enhance the official curl docker image.

Monitoring in a Microservices WorldDocker, Inc.

Fabian Stäber, Instana - In recent years, we saw a great paradigm shift in software engineering away from static monolithic applications towards dynamic distributed horizontally scalable architectures. Docker is one of the key technologies enabling this development. This shift poses a lot of new challenges for application monitoring, ranging from practical issues (need for automation) to technical challenges (Docker networking) to organizational topics (blurring line between software engineers and operations) to fundamental questions (define what is an application). In this talk we show how Docker changed the way we do monitoring, how modern application monitoring systems work, and what future developments we expect.

COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...Docker, Inc.

Clemente Biondo, Engineering Ingegneria Informatica - When the COVID 19 pandemic started, Engineering Ingegneria Informatica Group (1.25 billion euros of revenues, 65 offices around the world, 12.000 employees) was forced to put their digital transformation to the test in order to maintain operational continuity. In this session, Clemente Biondo, the Tech Lead of the Information Systems Department, will share how his company is reacting to this unforeseeable scenario and how Docker-driven digital transformation had paved the path for work to continue remotely. Clemente will discuss learnings moving from colocated teams, manual approaches, email based-business processes, and a monolithic application to a mature DevOps culture characterized by a distributed autonomous workforce and a continuous deployment process that deploys backward-compatible Docker containerized microservices into hybrid multi cloud datacenters an average of twice a day with zero-downtime. He will detail how they use Docker to unify dev, test and production environments, and as an efficient and automated mechanism for deploying applications. Lastly, Clemente shares how, in our darkest hour, he and others are working to shine their brightest light.

Predicting Space Weather with DockerDocker, Inc.

The document discusses how NOAA's Space Weather Prediction Center transitioned from a monolithic architecture to microservices using Docker. It describes how they started with a small verification project, then replaced their critical GOES satellite data source. This improved developers' morale and delivery speed. They encountered some security issues initially but learned from them. The transition was very successful and allowed them to quickly expand their mission to forecast aviation impacts using scientists' models packaged as Docker services.

Become a Docker Power User With Microsoft Visual Studio CodeDocker, Inc.

Brian Christner, 56k + Docker Captain - In this session, we will unlock the full potential of using Microsoft Visual Studio Code (VS Code) and Docker Desktop to turn you into a Docker Power User. When we expand and utilize the VS Code Docker plugin, we can take our projects and Docker skills to the next level. In addition to using VS Code, we streamline our Docker Desktop development workflow with less context switching and built-in shortcuts. You will learn how to bootstrap new projects, quickly write Dockerfiles utilizing templates, build, run, and interact with containers all from VS Code.

How to Use Mirroring and Caching to Optimize your Container RegistryDocker, Inc.

Monolithic to Microservices + Docker = SDLC on Steroids!Docker, Inc.

Ashish Sharma, SS&C Eze - SS&C Eze provides various products in the stock market domain. We spent the last couple of years building Eclipse which is an investment suite born in cloud. The journey so far has been very interesting. The very first version of the product were a bunch of monolithic windows services and deployed using Octopus tool. We successfully managed to bring all the monolithic problem to the cloud and created a nightmare for ourselves. We then started applying microservices architecture principles and started breaking the monolithic into small services. Very soon we realized that we need a better packaging/deployment tool. Docker looked like a magical solution to our problem. Since its adoption, It has not only solved the deployment problem for us but has made a deep impact on different aspects of SDLC. It allowed us to use heterogeneous technology stacks, simplified development environment setup, simplified our testing strategy, improved our speed of delivery, and made our developers more productive. In this talk I would like to share our experience of using Docker and its positive impact on our SDLC.

Kubernetes at Datadog ScaleDocker, Inc.

Kubernetes networking can be complex to scale due to issues like growing iptables rules, but newer solutions are helping. Pod networking uses CNI plugins like flannel or Calico to assign each pod an IP and allow communication. Service networking uses kube-proxy and iptables or IPVS for load balancing to pods. DNS is used to resolve service names to IPs. While Kubernetes networking brings flexibility, operators must learn the nuances of their specific CNI plugin and issues can arise, but the ecosystem adapts quickly to new needs and changes don't impact all workloads.

Labels, Labels, Labels Docker, Inc.

Andy Clemenko, StackRox - One underutilized, and amazing, thing about the docker image scheme is labels. Labels are a built in way to document all aspects about the image itself. Think about all the information that the tags inside your clothing carry. If you care to look you can find out everything about the garment. All that information can be very valuable. Now think about how we can leverage labels to carry similar information. We can even use the labels to contain Docker Compose or even Kubernetes Yaml. We can even include labels into the CI/CD process making things more secure and smoother. Come find out some fun techniques on how to leverage labels to do some fun and amazing things.

Build & Deploy Multi-Container Applications to AWSDocker, Inc.

Lukonde Mwila, Entelect As the cloud-native approach to development and deployment becomes more prevalent, it's an exciting time for software engineers to be equipped on how to dockerize multi-container applications and deploy them to the cloud. In this talk, Lukonde Mwila, Software Engineer at Entelect, will cover the following topics: - Docker Compose - Containerizing an Nginx Server - Containerizing an React App - Containerizing an Node.JS App - Containerizing anMongoDB App - Runing Multi-Container App Locally - Creating a CI/CD Pipeline - Adding a build stage to test containers and push images to Docker Hub - Deploying Multi-Container App to AWS Elastic Beanstalk Lukonde will start by giving an overview of how Docker Compose works and how it makes it very easy and straightforward to startup multiple Docker containers at the same time and automatically connect them together with some form of networking. After that, Lukonde will take a hands on approach to containerize an Nginx server, a React app, a NodeJS app and a MongoDB instance to demonstrate the power of Docker Compose. He'll demonstrate usage of two Docker files for an application, one production grade and the other for local development and running of tests. Lastly, he'll demonstrate creating a CI/CD pipeline in AWS to build and test our Docker images before pushing them to Docker Hub or AWS ECR, and finally deploying our multi-container application AWS Elastic Beanstalk.

From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...Docker, Inc.

Elton Stoneman, Docker Captain + Container Consultant and Trainer How do you provide a SaaS offering when your product is a 10-year old Fortran app, currently built to run on Windows 10? With Docker and Kubernetes of course - and you can do it in a week (... to prototype level at least). In this session I'll walk through the processes and practicalities of taking an older Windows app, making it run in containers with Kubernetes, and then building a simple API wrapper to host the whole stack as a cloud-based SaaS product. There's a lot of technology here from a real world case study, and I'll focus on: - running Windows apps in Docker containers - building a .NET Core API which can run in Linux or Windows containers - running the stack in Kubernetes with Docker Desktop locally and AKS in the cloud - configuring AKS workloads in Azure to burst out to Azure Container Instances And there's a core theme to this session: Docker and Kubernetes are complex technologies, but they're the key to modern development. If you invest time learning them, they make projects like this simple, portable, fast and fun.

Developing with Docker for the Arm ArchitectureDocker, Inc.

This virtual meetup introduces the concepts and best practices of using Docker containers for software development for the Arm architecture across a variety of hardware systems. Using Docker Desktop on Windows or Mac, Amazon Web Services (AWS) A1 instances, and embedded Linux, we will demonstrate the latest Docker features to build, share, and run multi-architecture images with transparent support for Arm.

Sharing is Caring: How to Begin Speaking at ConferencesDocker, Inc.

For many, the idea of speaking in front of a bunch of strangers can be enough to prevent you from ever responding to a CFP. But don't let it! Speaking at conferences, meetups, or even at your own company is a fantastic way to share you knowledge, meet others, advance your career, and give back. Whether you are on the fence, have decided to take the plunge and submit a CFP, or have already even a few talks, this session is for you. Drawing on their experience reviewing DockerCon CFPs and prepping speakers (including themselves) for a variety of conferences, Jenny Burcio and Ashlynn Polini will outline tips and strategies for turning your idea into a winning proposal and ultimately a compelling talk. They will share how to submit and prepare for your first - or next - conference talk.

Containerize Your Game Server for the Best Multiplayer Experience Docker, Inc.

How to Improve Your Image Builds Using Advance Docker BuildDocker, Inc.

Build & Deploy Multi-Container Applications to AWSDocker, Inc.

Securing Your Containerized Applications with NGINXDocker, Inc.

How To Build and Run Node Apps with Docker and ComposeDocker, Inc.

Hands-on Helm Docker, Inc.

Distributed Deep Learning with Docker at SalesforceDocker, Inc.

The First 10M Pulls: Building The Official Curl Image for Docker HubDocker, Inc.

Monitoring in a Microservices WorldDocker, Inc.

COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...Docker, Inc.

Predicting Space Weather with DockerDocker, Inc.

Become a Docker Power User With Microsoft Visual Studio CodeDocker, Inc.

How to Use Mirroring and Caching to Optimize your Container RegistryDocker, Inc.

Monolithic to Microservices + Docker = SDLC on Steroids!Docker, Inc.

Kubernetes at Datadog ScaleDocker, Inc.

Labels, Labels, Labels Docker, Inc.

Build & Deploy Multi-Container Applications to AWSDocker, Inc.

From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...Docker, Inc.

Developing with Docker for the Arm ArchitectureDocker, Inc.

Sharing is Caring: How to Begin Speaking at ConferencesDocker, Inc.

Recently uploaded (20)

Splunk Security Update | Public Sector Summit Germany 2025Splunk

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.

Mobile App Development Company in Saudi ArabiaSteve Jonas

EmizenTech is a globally recognized software development company, proudly serving businesses since 2013. With over 11+ years of industry experience and a team of 200+ skilled professionals, we have successfully delivered 1200+ projects across various sectors. As a leading Mobile App Development Company In Saudi Arabia we offer end-to-end solutions for iOS, Android, and cross-platform applications. Our apps are known for their user-friendly interfaces, scalability, high performance, and strong security features. We tailor each mobile application to meet the unique needs of different industries, ensuring a seamless user experience. EmizenTech is committed to turning your vision into a powerful digital product that drives growth, innovation, and long-term success in the competitive mobile landscape of Saudi Arabia.

TrsLabs - Fintech Product & Business ConsultingTrs Labs

Hybrid Growth Mandate Model with TrsLabs Strategic Investments, Inorganic Growth, Business Model Pivoting are critical activities that business don't do/change everyday. In cases like this, it may benefit your business to choose a temporary external consultant. An unbiased plan driven by clearcut deliverables, market dynamics and without the influence of your internal office equations empower business leaders to make right choices. Getting things done within a budget within a timeframe is key to Growing Business - No matter whether you are a start-up or a big company Talk to us & Unlock the competitive advantage

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including: -Artificial Intelligence Market Overview -Strategies for AI Adoption in 2025 -Anticipated drivers of AI adoption and transformative technologies -Benefits of AI and Big data for your business -Tips on how to prepare your business for innovation -AI and data privacy: Strategies for securing data privacy in AI models, etc. Download your free copy nowand implement the key findings to improve your business.

How Can I use the AI Hype in my Business Context?Daniel Lehner

𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨? Everyone’s talking about AI but is anyone really using it to create real value? Most companies want to leverage AI. Few know 𝗵𝗼𝘄. ✅ What exactly should you ask to find real AI opportunities? ✅ Which AI techniques actually fit your business? ✅ Is your data even ready for AI? If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

Want to learn practical tips for designing systems that can scale efficiently without compromising speed? Join us for a workshop where we’ll address these challenges head-on and explore how to architect low-latency systems using Rust. During this free interactive workshop oriented for developers, engineers, and architects, we’ll cover how Rust’s unique language features and the Tokio async runtime enable high-performance application development. As you explore key principles of designing low-latency systems with Rust, you will learn how to: - Create and compile a real-world app with Rust - Connect the application to ScyllaDB (NoSQL data store) - Negotiate tradeoffs related to data modeling and querying - Manage and monitor the database for consistently low latencies

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55