SlideShare a Scribd company logo

Docker Registry V2

Docker, Inc., profile picture
Docker, Inc.

This document discusses Docker Registry API V2, a new model for image distribution that addresses limitations in the previous V1 API. Key changes include making layers content-addressable using cryptographic digests for identification and verification. Images are now described by manifests containing layer digests. The registry stores content in repositories and no longer exposes internal image details. Early adoption shows V2 providing significantly better performance than V1 with 80% fewer requests and 60% less bandwidth used. Future goals include improving documentation, adding features like pull-through caching, and developing the Docker distribution components to provide a foundation for more advanced distribution models.

Technology
1 of 47
Downloaded 462 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Most read
24
Most read
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
Most read
44
45
46
47
A New Model for Image Distribution
Stephen Day Distribution, Tech Lead Docker, Inc. stephen@docker.com @stevvooe github.com/stevvooe
Overview • Why does this matter? • History • Docker Registry API V2 • Implementation • The Future
What is Docker?
What is an Image?
What is an Image? • Identified by a name • ubuntu • redis • stevvooe/myapp • docker run ubuntu - Runs a container, created from image ubuntu
What is an Image? • Containers, the runtime of docker, are created from images • Filesystem made up with “layers” - Just tar files - Layers can be shared between images • Includes a description organizing layers into an image A runnable component with a filesystem
What is the Docker Registry?
What is the Docker Registry? • A central place to store and distribute docker images • Stores the layers and the description of how they make up an image • Implements a common API agreed upon by Docker clients
What is the Docker Registry? • Several Implementations • A simple web server to make images available • A complete web application • Services • Docker Hub • Docker Trusted Registry • Documentation: https://docs.docker.com/registry/ A central place to store and distribute docker images
History
Docker Registry API V1: History • Layer Oriented • Layer IDs are randomly assigned • JSON object corresponding to each layer referencing a parent • Naming accomplished through tags Layer Layer Layer Layer JSON JSON JSON JSONFetch(ID) {
Registry API V1 URL Layout Methods URL GET /v1/_ping GET, PUT /v1/images/(image_id)/layer GET, PUT /v1/images/(image_id)/json GET /v1/images/(image_id)/ancestry GET /v1/repositories/(namespace)/(repository)/tags GET, PUT, DELETE /v1/repositories/(namespace)/(repository)/tags/(tag*) DELETE /v1/repositories/(namespace)/(repository)/ GET /v1/search 13 https://docs.docker.com/reference/api/hub_registry_spec/
Docker Registry API V1: Problems • Abstraction - Exposes Internals of Image to distribution mechanism • Security - Image IDs must be kept secret - Who assigns the layer IDs? - Hard to audit, verify • Performance - Fetch a layer, fetch the parent, fetch the parent, …
Docker Registry API V1: Problems • Implementation in Python - Affected ease of deployment - Reduced sharing with main Docker Project • More information: • https://github.com/docker/docker/issues/8093
Docker Registry API V2
Docker Registry API V2: Goals • Simplicity - Easy to implement - Works with static host • Security - Verifiable Images - Straightforward access control
Docker Registry API V2: Goals • Distribution - Separate location of content from naming • Performance - Remove the single track • Implementation - Use Go to increase code sharing with Docker Engine
Docker Registry API V2: Content Addressable • Layers are treated as content-addressable blobs - Much better for security - Permits safe-distribution through untrusted channels • All data can be verified • De-duplication • Improved cache-ability • Content address is known as the “digest”
Docker Registry API V2: Digests • Uniquely identifies content • A cryptographically strong hash - Chose a name, digest, that does not conflict with other concepts (map, dict, crc, etc.) - Simply using sha256(bytes) • Independently Verifiable - By agreeing on common algorithm, IDs chosen for content without coordination • Strongly-typed with tools to parse and verify - http://godoc.org/github.com/docker/distribution/digest
Docker Registry API V2: Manifests • Describes the components of an image in a single object - Layers can be fetched immediately, in parallel LayerLayer Layer Layer JSONFetch(ID) {
Docker Registry API V2: Manifests { "name": <name>, "tag": <tag>, "fsLayers": [ { "blobSum": <digest> }, ... ] ], "history": [<v1 image json>, ... ] }
Docker Registry API V2: Manifest • Content-addressable: - docker pull ubuntu@sha256:8126991394342c2775a9ba4a843869112da815 6037451fc424454db43c25d8b0 • Leverages Merkle DAG - Because the digests of the layers are in the manifest, if any bit in the layer changes, the digest of the manifest changes - Similar to git, ipfs, camlistore and a host of other projects • Tags are in the manifest - This will going away
Docker Registry API V2: Repositories • All content is now part of a named repository - Image IDs are no longer a secret - Simplified authorization model • repository + operation (push, pull) - Clients must “prove” content is available to another repository by providing it • Opened up namespace to allow more than two components - No reason to have registry enforce “<user>/<image>” - API “reversed” to make static layout easier
Registry API V2 URL Layout Methods URL GET /v2/ GET /v2/<name>/tags/list GET, PUT, DELETE /v2/<name>/manifests/<reference> GET /v2/<name>/blobs/<digest> POST /v2/<name>/blobs/uploads/ GET, PUT, PATCH, DELETE /v2/<name>/blobs/uploads/<uuid> https://docs.docker.com/registry/spec/api/
Docker Registry API V2: Design • Shared-nothing - “Backend” ties a cluster of registries together - Allows scaling by adding instances - Performance limited by backend • Make backend faster, registry gets faster • Pull-optimized - Most important factor when distributing software - May hurt certain use cases • Resumable Pull and Push (specified but not implemented) - Resumable pull already available with http Range requests - Two-step upload start for resumable push - Built into the protocol for future support • A living specification - Meant to be used and modified - Always backwards compatible
Docker Registry API V2: Differences with V1 • Content addresses (digests) are primary identifier • Unrolled image description model • Multi-step upload - Provides flexibility in failure modes - Options for future alternative upload location (redirects) • No Search API - In V1, this API does everything - Replacing with something better • No explicit tagging API - This will change: https://github.com/docker/distribution/pull/173
Docker Registry 2.0
–Earl Milford “[A registry] should be neither seen nor heard. ”
Handlers Docker Registry 2.0: Architecture Repository Repository Storage Access Control Notifications Docker Engine Auth API
Docker Registry 2.0: An Ingredient • Move away from monolithic architecture • Narrower scope - Distribute content • Extensible - Authentication - Index - Ponies • Strong core - Docker Hub - Docker Trusted Registry
Docker Registry 2.0 • Full support released with Docker 1.6 - Minimal bugs - Most problems are common to version upgrades • Header required to declare support for 2.0 API • Validated most concepts in 1.3, 1.4 with V2 preview - Much faster pull performance - You’ve probably already used it with Docker Hub • There are some edge cases - push-heavy workflows - disk IO when verifying large images - We are mitigating these
Docker Registry 2.0: Should you use it? • Are you on Docker 1.6+? - Yes. • Evaluate it • Test it • Break it (and file bugs https://github.com/docker/distribution/issues) • Deploy it • Are you on Docker <1.6? - Are you entrenched in v1? • Perhaps, hold off - Run dual stack v1, v2 • Not recommended
Docker Registry 2.0: Deploying • Internal deployments - Use the filesystem driver — it is really fast - Backup with rsync • Scale storage - Use S3 driver • Make sure you are “close” since round trip times can have an effect • Scale Reads - Use round robin DNS • Do not use this for HA - Rsync to followers on read-only filesystem - Add machines to taste • https://docs.docker.com/registry/deploying/
Docker Registry 2.0: Docker Hub • Running the Hub - S3 backend • Having some trouble with round trips to s3 :( - Decent performance with very little caching • A lot of low hanging fruit left to tackle • No longer intertwined with Docker Hub services • Independent Authentication Service • Heightened Availability
Monitoring culture
Docker Hub Adoption 0% 50% 100% Last Three Months V1 (1.5-) V2 (1.6+)
Docker Hub Adoption • Overall usage increasing • A V2 world and growing
V1/V2 Protocol Overall Comparison 0 25 50 75 100 Requests Bandwidth V1 V2 80% Fewer Requests 60% Less Bandwidth
V1/V2 Protocol HTTP Errors Peak Average V1 V2 5
Exceptional Panicking • 1 Panic in Three Months of Production • 4000 protocol level errors per 30 minutes in V1 • 5 protocol level errors per 30 minutes in V2
Docker Registry 2.1
Docker Registry 2.1 • Key Changes - Documentation - Pull-through Caching - Soft-Deletion - Native Basic Auth Support - Stability - Catalog API - Storage Drivers • Release coming by mid-July
Docker Distribution
Docker Distribution: Goals • Goals - Improve the state of image distribution in Docker - Build a solid and secure foundation • Focus - Security - Reliability - Performance • Unlock new distribution models - Integration with trust system (notary!) - Relax reliance on registries - Peer to Peer for large deployments
Docker Distribution: Future • Ingredients - From the start, we have targeted solid packages - Provide Lego to build image distribution systems • Clean up the docker daemon code base - Defined new APIs for working with docker content - Increase feature velocity - Generalize around strong base • Current Manifest format is provisional - Still includes v1 layer JSON - Content-addressability + mediatypes make support new formats trivial - https://github.com/docker/distribution/pull/62 • Feature parity with V1 and maturity - Building collective operational knowledge • Deletes and Garbage Collection - Diverse backend support makes this hard - https://github.com/docker/distribution/issues/461 - https://github.com/docker/distribution/issues/462 • Search - See the goals of Distribution to see why this is interesting • Road Map: https://github.com/docker/distribution/wiki
Thank you Stephen Day Google Group: distribution@dockerproject.org GitHub: https://github.com/docker/distribution IRC on Freenode: #docker-distribution

More Related Content

What's hot (20)

PDF
Dockers and kubernetes
Dr Ganesh Iyer
 
PPTX
Introduction to Docker - 2017
Docker, Inc.
 
PDF
Docker Birthday #3 - Intro to Docker Slides
Docker, Inc.
 
PDF
Docker in real life
Nguyen Van Vuong
 
PPTX
Docker intro
Oleg Z
 
PPTX
Kubernetes Basics
Antonin Stoklasek
 
PDF
Kubernetes Basics
Eueung Mulyana
 
PDF
What is Docker | Docker Tutorial for Beginners | Docker Container | DevOps To...
Edureka!
 
PPTX
Kubernetes Introduction
Eric Gustafson
 
PPSX
Docker Kubernetes Istio
Araf Karsh Hamid
 
PDF
Introduction to Docker
Luong Vo
 
PDF
Introduction to Docker
Aditya Konarde
 
PDF
Docker Introduction
Peng Xiao
 
PDF
What is Docker Architecture | Edureka
Edureka!
 
PDF
Diving Through The Layers: Investigating runc, containerd, and the Docker eng...
Phil Estes
 
PDF
Docker Compose by Aanand Prasad
Docker, Inc.
 
PDF
Kubernetes 101
Crevise Technologies
 
PPT
Docker introduction
Phuc Nguyen
 
PPTX
Kubernetes Introduction
Martin Danielsson
 
PPT
presentation on Docker
Virendra Ruhela
 
Dockers and kubernetes
Dr Ganesh Iyer
 
Introduction to Docker - 2017
Docker, Inc.
 
Docker Birthday #3 - Intro to Docker Slides
Docker, Inc.
 
Docker in real life
Nguyen Van Vuong
 
Docker intro
Oleg Z
 
Kubernetes Basics
Antonin Stoklasek
 
Kubernetes Basics
Eueung Mulyana
 
What is Docker | Docker Tutorial for Beginners | Docker Container | DevOps To...
Edureka!
 
Kubernetes Introduction
Eric Gustafson
 
Docker Kubernetes Istio
Araf Karsh Hamid
 
Introduction to Docker
Luong Vo
 
Introduction to Docker
Aditya Konarde
 
Docker Introduction
Peng Xiao
 
What is Docker Architecture | Edureka
Edureka!
 
Diving Through The Layers: Investigating runc, containerd, and the Docker eng...
Phil Estes
 
Docker Compose by Aanand Prasad
Docker, Inc.
 
Kubernetes 101
Crevise Technologies
 
Docker introduction
Phuc Nguyen
 
Kubernetes Introduction
Martin Danielsson
 
presentation on Docker
Virendra Ruhela
 

Similar to Docker Registry V2 (20)

PPTX
A new model for Docker image distribution
Docker, Inc.
 
PPTX
Dockerize the World
damovsky
 
PPTX
Intro Docker october 2013
dotCloud
 
PDF
Exploring Docker in CI/CD
Henry Huang
 
PPTX
CT Software Developers Meetup: Using Docker and Vagrant Within A GitHub Pull ...
E. Camden Fisher
 
PDF
Lightweight Virtualization Docker in Practice
Docker, Inc.
 
PDF
Docker in pratice -chenyifei
dotCloud
 
PDF
Cloudsolutionday 2016: DevOps workflow with Docker on AWS
AWS Vietnam Community
 
PPTX
Devoxx 2016 - Docker Nuts and Bolts
Patrick Chanezon
 
PPTX
DockerCon EU 2015 Barcelona
Roman Dembitsky
 
PDF
Containers docker-docker hub-azureacr-azure aci
Rajesh Kolla
 
PPTX
Using Docker in production: Get started today!
Clarence Bakirtzidis
 
PPTX
Write Once and REALLY Run Anywhere | OpenStack Summit HK 2013
dotCloud
 
PPTX
Intro to Docker October 2013
Docker, Inc.
 
PDF
Docker
Codeister Technolgoies
 
PPTX
ma-formation-en-Docker-jlklk,nknkjn.pptx
imenhamada17
 
PPTX
Docker Hub: Past, Present and Future by Ken Cochrane & BC Wong
Docker, Inc.
 
PDF
DevOpsDays Houston 2019 - Shaun Ladewig, Robert Stone - From OverTheWallOps t...
DevOpsDays Houston
 
PPTX
Intro docker and demo monitor on docker
Watcharin Yang-Ngam
 
PPTX
OpenStack Summit
Docker, Inc.
 
A new model for Docker image distribution
Docker, Inc.
 
Dockerize the World
damovsky
 
Intro Docker october 2013
dotCloud
 
Exploring Docker in CI/CD
Henry Huang
 
CT Software Developers Meetup: Using Docker and Vagrant Within A GitHub Pull ...
E. Camden Fisher
 
Lightweight Virtualization Docker in Practice
Docker, Inc.
 
Docker in pratice -chenyifei
dotCloud
 
Cloudsolutionday 2016: DevOps workflow with Docker on AWS
AWS Vietnam Community
 
Devoxx 2016 - Docker Nuts and Bolts
Patrick Chanezon
 
DockerCon EU 2015 Barcelona
Roman Dembitsky
 
Containers docker-docker hub-azureacr-azure aci
Rajesh Kolla
 
Using Docker in production: Get started today!
Clarence Bakirtzidis
 
Write Once and REALLY Run Anywhere | OpenStack Summit HK 2013
dotCloud
 
Intro to Docker October 2013
Docker, Inc.
 
Docker
Codeister Technolgoies
 
ma-formation-en-Docker-jlklk,nknkjn.pptx
imenhamada17
 
Docker Hub: Past, Present and Future by Ken Cochrane & BC Wong
Docker, Inc.
 
DevOpsDays Houston 2019 - Shaun Ladewig, Robert Stone - From OverTheWallOps t...
DevOpsDays Houston
 
Intro docker and demo monitor on docker
Watcharin Yang-Ngam
 
OpenStack Summit
Docker, Inc.
 
Ad

More from Docker, Inc. (20)

PDF
Containerize Your Game Server for the Best Multiplayer Experience
Docker, Inc.
 
PDF
How to Improve Your Image Builds Using Advance Docker Build
Docker, Inc.
 
PDF
Build & Deploy Multi-Container Applications to AWS
Docker, Inc.
 
PDF
Securing Your Containerized Applications with NGINX
Docker, Inc.
 
PDF
How To Build and Run Node Apps with Docker and Compose
Docker, Inc.
 
PDF
Hands-on Helm
Docker, Inc.
 
PDF
Distributed Deep Learning with Docker at Salesforce
Docker, Inc.
 
PDF
The First 10M Pulls: Building The Official Curl Image for Docker Hub
Docker, Inc.
 
PDF
Monitoring in a Microservices World
Docker, Inc.
 
PDF
COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...
Docker, Inc.
 
PDF
Predicting Space Weather with Docker
Docker, Inc.
 
PDF
Become a Docker Power User With Microsoft Visual Studio Code
Docker, Inc.
 
PDF
How to Use Mirroring and Caching to Optimize your Container Registry
Docker, Inc.
 
PDF
Monolithic to Microservices + Docker = SDLC on Steroids!
Docker, Inc.
 
PDF
Kubernetes at Datadog Scale
Docker, Inc.
 
PDF
Labels, Labels, Labels
Docker, Inc.
 
PDF
Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment Model
Docker, Inc.
 
PDF
Build & Deploy Multi-Container Applications to AWS
Docker, Inc.
 
PDF
From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...
Docker, Inc.
 
PDF
Developing with Docker for the Arm Architecture
Docker, Inc.
 
Containerize Your Game Server for the Best Multiplayer Experience
Docker, Inc.
 
How to Improve Your Image Builds Using Advance Docker Build
Docker, Inc.
 
Build & Deploy Multi-Container Applications to AWS
Docker, Inc.
 
Securing Your Containerized Applications with NGINX
Docker, Inc.
 
How To Build and Run Node Apps with Docker and Compose
Docker, Inc.
 
Hands-on Helm
Docker, Inc.
 
Distributed Deep Learning with Docker at Salesforce
Docker, Inc.
 
The First 10M Pulls: Building The Official Curl Image for Docker Hub
Docker, Inc.
 
Monitoring in a Microservices World
Docker, Inc.
 
COVID-19 in Italy: How Docker is Helping the Biggest Italian IT Company Conti...
Docker, Inc.
 
Predicting Space Weather with Docker
Docker, Inc.
 
Become a Docker Power User With Microsoft Visual Studio Code
Docker, Inc.
 
How to Use Mirroring and Caching to Optimize your Container Registry
Docker, Inc.
 
Monolithic to Microservices + Docker = SDLC on Steroids!
Docker, Inc.
 
Kubernetes at Datadog Scale
Docker, Inc.
 
Labels, Labels, Labels
Docker, Inc.
 
Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment Model
Docker, Inc.
 
Build & Deploy Multi-Container Applications to AWS
Docker, Inc.
 
From Fortran on the Desktop to Kubernetes in the Cloud: A Windows Migration S...
Docker, Inc.
 
Developing with Docker for the Arm Architecture
Docker, Inc.
 
Ad

Recently uploaded (20)

PDF
OpenInfra ID 2025 - Are Containers Dying? Rethinking Isolation with MicroVMs.pdf
Muhammad Yuga Nugraha
 
PDF
Empowering Cloud Providers with Apache CloudStack and Stackbill
ShapeBlue
 
PPTX
The Yotta x CloudStack Advantage: Scalable, India-First Cloud
ShapeBlue
 
PDF
TrustArc Webinar - Navigating Data Privacy in LATAM: Laws, Trends, and Compli...
TrustArc
 
PDF
Building Resilience with Digital Twins : Lessons from Korea
SANGHEE SHIN
 
PDF
Bitcoin+ Escalando sin concesiones - Parte 1
Fernando Paredes García
 
PDF
Meetup Kickoff & Welcome - Rohit Yadav, CSIUG Chairman
ShapeBlue
 
PPTX
Earn Agentblazer Status with Slack Community Patna.pptx
SanjeetMishra29
 
PPTX
Machine Learning Benefits Across Industries
SynapseIndia
 
PDF
CloudStack GPU Integration - Rohit Yadav
ShapeBlue
 
PDF
NewMind AI Journal - Weekly Chronicles - July'25 Week II
NewMind AI
 
PDF
Generative AI in Healthcare: Benefits, Use Cases & Challenges
Lily Clark
 
PPTX
Building a Production-Ready Barts Health Secure Data Environment Tooling, Acc...
Barts Health
 
PDF
UiPath vs Other Automation Tools Meeting Presentation.pdf
Tracy Dixon
 
PDF
Shuen Mei Parth Sharma Boost Productivity, Innovation and Efficiency wit...
AWS Chicago
 
PPTX
AI Code Generation Risks (Ramkumar Dilli, CIO, Myridius)
Priyanka Aash
 
PDF
Upskill to Agentic Automation 2025 - Kickoff Meeting
DianaGray10
 
PDF
UiPath on Tour London Community Booth Deck
UiPathCommunity
 
PDF
Women in Automation Presents: Reinventing Yourself — Bold Career Pivots That ...
DianaGray10
 
PPTX
Building and Operating a Private Cloud with CloudStack and LINBIT CloudStack ...
ShapeBlue
 
OpenInfra ID 2025 - Are Containers Dying? Rethinking Isolation with MicroVMs.pdf
Muhammad Yuga Nugraha
 
Empowering Cloud Providers with Apache CloudStack and Stackbill
ShapeBlue
 
The Yotta x CloudStack Advantage: Scalable, India-First Cloud
ShapeBlue
 
TrustArc Webinar - Navigating Data Privacy in LATAM: Laws, Trends, and Compli...
TrustArc
 
Building Resilience with Digital Twins : Lessons from Korea
SANGHEE SHIN
 
Bitcoin+ Escalando sin concesiones - Parte 1
Fernando Paredes García
 
Meetup Kickoff & Welcome - Rohit Yadav, CSIUG Chairman
ShapeBlue
 
Earn Agentblazer Status with Slack Community Patna.pptx
SanjeetMishra29
 
Machine Learning Benefits Across Industries
SynapseIndia
 
CloudStack GPU Integration - Rohit Yadav
ShapeBlue
 
NewMind AI Journal - Weekly Chronicles - July'25 Week II
NewMind AI
 
Generative AI in Healthcare: Benefits, Use Cases & Challenges
Lily Clark
 
Building a Production-Ready Barts Health Secure Data Environment Tooling, Acc...
Barts Health
 
UiPath vs Other Automation Tools Meeting Presentation.pdf
Tracy Dixon
 
Shuen Mei Parth Sharma Boost Productivity, Innovation and Efficiency wit...
AWS Chicago
 
AI Code Generation Risks (Ramkumar Dilli, CIO, Myridius)
Priyanka Aash
 
Upskill to Agentic Automation 2025 - Kickoff Meeting
DianaGray10
 
UiPath on Tour London Community Booth Deck
UiPathCommunity
 
Women in Automation Presents: Reinventing Yourself — Bold Career Pivots That ...
DianaGray10
 
Building and Operating a Private Cloud with CloudStack and LINBIT CloudStack ...
ShapeBlue
 

Docker Registry V2

  • 1. A New Model for Image Distribution
  • 2. Stephen Day Distribution, Tech Lead Docker, Inc. [email protected] @stevvooe github.com/stevvooe
  • 3. Overview • Why does this matter? • History • Docker Registry API V2 • Implementation • The Future
  • 5. What is an Image?
  • 6. What is an Image? • Identified by a name • ubuntu • redis • stevvooe/myapp • docker run ubuntu - Runs a container, created from image ubuntu
  • 7. What is an Image? • Containers, the runtime of docker, are created from images • Filesystem made up with “layers” - Just tar files - Layers can be shared between images • Includes a description organizing layers into an image A runnable component with a filesystem
  • 8. What is the Docker Registry?
  • 9. What is the Docker Registry? • A central place to store and distribute docker images • Stores the layers and the description of how they make up an image • Implements a common API agreed upon by Docker clients
  • 10. What is the Docker Registry? • Several Implementations • A simple web server to make images available • A complete web application • Services • Docker Hub • Docker Trusted Registry • Documentation: https://docs.docker.com/registry/ A central place to store and distribute docker images
  • 12. Docker Registry API V1: History • Layer Oriented • Layer IDs are randomly assigned • JSON object corresponding to each layer referencing a parent • Naming accomplished through tags Layer Layer Layer Layer JSON JSON JSON JSONFetch(ID) {
  • 13. Registry API V1 URL Layout Methods URL GET /v1/_ping GET, PUT /v1/images/(image_id)/layer GET, PUT /v1/images/(image_id)/json GET /v1/images/(image_id)/ancestry GET /v1/repositories/(namespace)/(repository)/tags GET, PUT, DELETE /v1/repositories/(namespace)/(repository)/tags/(tag*) DELETE /v1/repositories/(namespace)/(repository)/ GET /v1/search 13 https://docs.docker.com/reference/api/hub_registry_spec/
  • 14. Docker Registry API V1: Problems • Abstraction - Exposes Internals of Image to distribution mechanism • Security - Image IDs must be kept secret - Who assigns the layer IDs? - Hard to audit, verify • Performance - Fetch a layer, fetch the parent, fetch the parent, …
  • 15. Docker Registry API V1: Problems • Implementation in Python - Affected ease of deployment - Reduced sharing with main Docker Project • More information: • https://github.com/docker/docker/issues/8093
  • 17. Docker Registry API V2: Goals • Simplicity - Easy to implement - Works with static host • Security - Verifiable Images - Straightforward access control
  • 18. Docker Registry API V2: Goals • Distribution - Separate location of content from naming • Performance - Remove the single track • Implementation - Use Go to increase code sharing with Docker Engine
  • 19. Docker Registry API V2: Content Addressable • Layers are treated as content-addressable blobs - Much better for security - Permits safe-distribution through untrusted channels • All data can be verified • De-duplication • Improved cache-ability • Content address is known as the “digest”
  • 20. Docker Registry API V2: Digests • Uniquely identifies content • A cryptographically strong hash - Chose a name, digest, that does not conflict with other concepts (map, dict, crc, etc.) - Simply using sha256(bytes) • Independently Verifiable - By agreeing on common algorithm, IDs chosen for content without coordination • Strongly-typed with tools to parse and verify - http://godoc.org/github.com/docker/distribution/digest
  • 21. Docker Registry API V2: Manifests • Describes the components of an image in a single object - Layers can be fetched immediately, in parallel LayerLayer Layer Layer JSONFetch(ID) {
  • 22. Docker Registry API V2: Manifests { "name": <name>, "tag": <tag>, "fsLayers": [ { "blobSum": <digest> }, ... ] ], "history": [<v1 image json>, ... ] }
  • 23. Docker Registry API V2: Manifest • Content-addressable: - docker pull ubuntu@sha256:8126991394342c2775a9ba4a843869112da815 6037451fc424454db43c25d8b0 • Leverages Merkle DAG - Because the digests of the layers are in the manifest, if any bit in the layer changes, the digest of the manifest changes - Similar to git, ipfs, camlistore and a host of other projects • Tags are in the manifest - This will going away
  • 24. Docker Registry API V2: Repositories • All content is now part of a named repository - Image IDs are no longer a secret - Simplified authorization model • repository + operation (push, pull) - Clients must “prove” content is available to another repository by providing it • Opened up namespace to allow more than two components - No reason to have registry enforce “<user>/<image>” - API “reversed” to make static layout easier
  • 25. Registry API V2 URL Layout Methods URL GET /v2/ GET /v2/<name>/tags/list GET, PUT, DELETE /v2/<name>/manifests/<reference> GET /v2/<name>/blobs/<digest> POST /v2/<name>/blobs/uploads/ GET, PUT, PATCH, DELETE /v2/<name>/blobs/uploads/<uuid> https://docs.docker.com/registry/spec/api/
  • 26. Docker Registry API V2: Design • Shared-nothing - “Backend” ties a cluster of registries together - Allows scaling by adding instances - Performance limited by backend • Make backend faster, registry gets faster • Pull-optimized - Most important factor when distributing software - May hurt certain use cases • Resumable Pull and Push (specified but not implemented) - Resumable pull already available with http Range requests - Two-step upload start for resumable push - Built into the protocol for future support • A living specification - Meant to be used and modified - Always backwards compatible
  • 27. Docker Registry API V2: Differences with V1 • Content addresses (digests) are primary identifier • Unrolled image description model • Multi-step upload - Provides flexibility in failure modes - Options for future alternative upload location (redirects) • No Search API - In V1, this API does everything - Replacing with something better • No explicit tagging API - This will change: https://github.com/docker/distribution/pull/173
  • 29. –Earl Milford “[A registry] should be neither seen nor heard. ”
  • 30. Handlers Docker Registry 2.0: Architecture Repository Repository Storage Access Control Notifications Docker Engine Auth API
  • 31. Docker Registry 2.0: An Ingredient • Move away from monolithic architecture • Narrower scope - Distribute content • Extensible - Authentication - Index - Ponies • Strong core - Docker Hub - Docker Trusted Registry
  • 32. Docker Registry 2.0 • Full support released with Docker 1.6 - Minimal bugs - Most problems are common to version upgrades • Header required to declare support for 2.0 API • Validated most concepts in 1.3, 1.4 with V2 preview - Much faster pull performance - You’ve probably already used it with Docker Hub • There are some edge cases - push-heavy workflows - disk IO when verifying large images - We are mitigating these
  • 33. Docker Registry 2.0: Should you use it? • Are you on Docker 1.6+? - Yes. • Evaluate it • Test it • Break it (and file bugs https://github.com/docker/distribution/issues) • Deploy it • Are you on Docker <1.6? - Are you entrenched in v1? • Perhaps, hold off - Run dual stack v1, v2 • Not recommended
  • 34. Docker Registry 2.0: Deploying • Internal deployments - Use the filesystem driver — it is really fast - Backup with rsync • Scale storage - Use S3 driver • Make sure you are “close” since round trip times can have an effect • Scale Reads - Use round robin DNS • Do not use this for HA - Rsync to followers on read-only filesystem - Add machines to taste • https://docs.docker.com/registry/deploying/
  • 35. Docker Registry 2.0: Docker Hub • Running the Hub - S3 backend • Having some trouble with round trips to s3 :( - Decent performance with very little caching • A lot of low hanging fruit left to tackle • No longer intertwined with Docker Hub services • Independent Authentication Service • Heightened Availability
  • 37. Docker Hub Adoption 0% 50% 100% Last Three Months V1 (1.5-) V2 (1.6+)
  • 38. Docker Hub Adoption • Overall usage increasing • A V2 world and growing
  • 39. V1/V2 Protocol Overall Comparison 0 25 50 75 100 Requests Bandwidth V1 V2 80% Fewer Requests 60% Less Bandwidth
  • 40. V1/V2 Protocol HTTP Errors Peak Average V1 V2 5
  • 41. Exceptional Panicking • 1 Panic in Three Months of Production • 4000 protocol level errors per 30 minutes in V1 • 5 protocol level errors per 30 minutes in V2
  • 43. Docker Registry 2.1 • Key Changes - Documentation - Pull-through Caching - Soft-Deletion - Native Basic Auth Support - Stability - Catalog API - Storage Drivers • Release coming by mid-July
  • 45. Docker Distribution: Goals • Goals - Improve the state of image distribution in Docker - Build a solid and secure foundation • Focus - Security - Reliability - Performance • Unlock new distribution models - Integration with trust system (notary!) - Relax reliance on registries - Peer to Peer for large deployments
  • 46. Docker Distribution: Future • Ingredients - From the start, we have targeted solid packages - Provide Lego to build image distribution systems • Clean up the docker daemon code base - Defined new APIs for working with docker content - Increase feature velocity - Generalize around strong base • Current Manifest format is provisional - Still includes v1 layer JSON - Content-addressability + mediatypes make support new formats trivial - https://github.com/docker/distribution/pull/62 • Feature parity with V1 and maturity - Building collective operational knowledge • Deletes and Garbage Collection - Diverse backend support makes this hard - https://github.com/docker/distribution/issues/461 - https://github.com/docker/distribution/issues/462 • Search - See the goals of Distribution to see why this is interesting • Road Map: https://github.com/docker/distribution/wiki
  • 47. Thank you Stephen Day Google Group: [email protected] GitHub: https://github.com/docker/distribution IRC on Freenode: #docker-distribution