SlideShare a Scribd company logo
Successful k8s
Platforms in
Airgapped
Environments
Drew Malone
Staff Solutions Engineer
Tanzu
Federal
VMware
drmalone@vmware.com
2
Overview
- Problem Statement
- ClusterAPI - a Primer
- Challenges and Solutions:
- Cloud Permissions
- Node Image Creation
- Cloud API Access
- Bootstrapping and Troubleshooting
- Now What? Creating a defensible Day 2 story
3
What do US Government Users Say About Kubernetes?
● Complex to manage Day Two Operations
● Disconnected Environments from Day One
● Need capability both in the cloud and at the edge (various meanings)
● Developer Experience is Lacking because of the complexity
● Ton of Hype about Kubernetes but Commanders and CIO’s want outcomes
● Everyone talks about installing and securing Kubernetes, but rarely do we see Developers pushing
apps to production on kubernetes
4
What do US Government Users Say About Kubernetes?
Standing up Airgapped Kubernetes is Hard
Securing, Operating and Pushing Code to k8s is even
Harder
5
Where Do I Begin?
6
Bring the declarative nature of
app lifecycles in k8s…. to
lifecycling k8s clusters
kind: Cluster
metadata:
name: a-team-k8s-cluster
namespace: a-team
spec:
distribution:
version: v1.20
topology:
controlPlane:
count: 3
class: m5.xlarge
Start at the Beginning - ClusterAPI
7
ClusterAPI - Bootstrapping a new Env
8
1. Configure IaaS Permissions
2. Configure IaaS API Access
3. Create node image (i.e. base OS image)
4. Bootstrap and Profit!
But a lot has to happen first…
9
ClusterAPI creates roles in your cloud. This
can be done a couple of ways:
- directly added to your cloud
- create a file to be handed to your PMO
Do you have those rights?
Is it a request to your PMO?
Step 1 - IaaS Permissions
ref: https://cluster-api.sigs.k8s.io/user/quick-start.html
10
API requests go over the Internet.
But if there’s no Internet, how do
you reach the API?
Step 2 -IaaS API Access - Challenge
11
Clouds will provide a “endpoint service” to allow
you to reach API sets without needing to
create/maintain Internet connectivity.
Step 2 -IaaS API Access - Solution
12
Image builder needs to download the entire Internet.
But you need to bake in <special things> like private
cloud CA’s, private registry CA’s, etc.
Step 3 -Build the Node Image - Challenge
13
Build the image “outside”. Then use
pre-kubeadm hooks in ClusterAPI to install
certs and run any “last mile” configurations.
This has the added benefit of being able to
update things like CA’s without needing to
cut another image.
NB - are you using an Isolated cloud region?
You need to ask about their diff doc.
Step 3 -Build the Node Image - Solution
ref: https://doc.crds.dev/github.com/kubernetes-sigs/cluster-api/bootstrap.cluster.x-k8s.io/KubeadmConfigTemplate/v1alph
14
If you absolutely must build the image inside the
firewall, you can do that. But you need:
- an OS repo (apt, yum)
- k8s bits (either packaged, or zip)
- a place to host said bits
- to configure Packer
- to trust your private cloud
- [possibly] to disable the use of GP3 disk
Step 3 -Build the Node Image - Addendum
15
Fire away and see
what fails!
Step 4 -Bootstrap and Profit!
ref: https://cluster-api.sigs.k8s.io/user/quick-start.html
16
Hi-level (CLI) failures are exactly that - high level.
To get to the source of truth, look in the pod logs in your KIND cluster:
- capi-kubeadm-bootstrap-system
- capi-kubeadm-control-plane-system
- capa-kubeadm…..
Step 5 -Troubleshooting
17
If there’s no Internet, where does your Load Balancer point?
Remember to annotate your k8s services accordingly
Step N -Load Balancers
ref: https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer
18
Getting a k8s is only the start.
How do you make it useful?
You Have a Kubernetes! Now What?
19
Committed to chopping wood and carrying water
VMware is a Leader in the Open Source
Community
Kubernetes
contributions by
release number
according to
k8s.devstats.cncf.io
Cluster API
● 2 of 7 steering committee members
● 10 of 23 special interest groups led by VMware
● 4 working groups led by VMware
● 2nd leading contributor to kubernetes
20
Immutable Infrastructure
● K8s that is scaled up
● Destroyed
● Recreated
● Not persisted
Cluster API
● Repave
● Rotate
● Repair
21
Immutable
Infrastructure
Continuous
Reconciliation
Multi Cluster
k8’s
Provenance In
Secure Supply
Chain
Production Ready k8s
ref: https://dodcio.defense.gov/Library/
(search for ‘multi-cluster kubernetes’)
22
Production Ready Kubernetes
Cluster API
Pinniped
Management Cluster
Cluster Lifecycle Management
Container Networking Observability
Shared and In-Cluster Services
Container registry Ingress Controller
API-Driven IaaS
Declarative API
Log Forwarding
Load Balancing
Identity & Authentication Backup and Restore Ingress & Services DNS
Local Control
Plane
Platform Operator
uses
GitOps
Workload
Clusters
Workload
Clusters
Workload
Clusters
External-dns
23
Characteristics of a “great” k8s platform
● Rapid patching of k8s and applications/services
● Prevention of config drift
● Align with approved baselines (i.e. gov’t regs)
● Build once, Run anywhere
24
How to get a “great” k8s platform
● Immutable Infrastructure (aka image-builder)
● Continuous Reconciliation
● Multi-cluster Kubernetes
● Provenance in Secure Supply Chain
● Prod-ready k8s, even in Dev
25
Continuous Reconciliation
Carvel Cluster API
26
Immutable Infrastructure and Continuous Reconciliation
Control Plane
Manifest Manifest
Manifest
Reconciliation Loop
Kapp-Controller
Workload
Clusters
Git Repo
Cluster Creation
and Enforcement
Day 2 Reconciliation
Cluster API
Operator Desired State
Kapp-Controller
Pinniped
27
Edge Deployment - Build Once, Run Anywhere
Public Cloud
Development and Test Environments Disconnected Production(Edge)
Edge
Local Control Plane
Management Cluster
Cluster Lifecycle Management
Workload
Clusters
Workload
Clusters
Workload
Clusters
Local Control Plane
Management Cluster
Cluster Lifecycle Management
Workload
Clusters
Workload
Clusters
Workload
Clusters
imgpkg
28
Summary
- k8s is (still) hard airgapped but you can make it easy
- ClusterAPI has everything you need
- Know who owns Cloud Permissions
- Create your Node images “outside” if possible
- Use Endpoints for Cloud API Access
- CAPI/CAP{x} pod logs for Troubleshooting
- Create a defensible Day 2 story with immutable infra + gitops
Thank You!!!
Drew Malone
Staff Solutions Engineer
Tanzu
Federal
VMware
drmalone@vmware.com

Recommended

Kubernetes: Learning from Zero to Production
Kubernetes: Learning from Zero to Production
Rosemary Wang
 
Xpdays: Kubernetes CI-CD Frameworks Case Study
Xpdays: Kubernetes CI-CD Frameworks Case Study
Denys Vasyliev
 
Production grade edge computing on Kubernetes OSS EU 2018
Production grade edge computing on Kubernetes OSS EU 2018
Steve Wong
 
Kubernetes Presentation
Kubernetes Presentation
Crevise Technologies
 
Why kubernetes matters
Why kubernetes matters
Platform9
 
Kubernetes extensibility: crd & operators
Kubernetes extensibility: crd & operators
Giacomo Tirabassi
 
Kubernetes extensibility: CRDs & Operators
Kubernetes extensibility: CRDs & Operators
SIGHUP
 
A GitOps model for High Availability and Disaster Recovery on EKS
A GitOps model for High Availability and Disaster Recovery on EKS
Weaveworks
 
Simplify Your Way To Expert Kubernetes Management
Simplify Your Way To Expert Kubernetes Management
DevOps.com
 
LISA2017 Kubernetes: Hit the Ground Running
LISA2017 Kubernetes: Hit the Ground Running
Chris McEniry
 
DNUG46 - Build your own private Cloud environment
DNUG46 - Build your own private Cloud environment
panagenda
 
Build your own private Cloud environment
Build your own private Cloud environment
Nico Meisenzahl
 
KubeCon 2019 Recap (Parts 1-3)
KubeCon 2019 Recap (Parts 1-3)
Ford Prior
 
Meetup 2023 - Gateway API.pdf
Meetup 2023 - Gateway API.pdf
Red Hat
 
Running and Managing Kubernetes on OpenStack
Running and Managing Kubernetes on OpenStack
Victor Palma
 
Presentation ING for ISC2 Secure Summits EMEA
Presentation ING for ISC2 Secure Summits EMEA
Thijs Ebbers
 
Immediate download Kubernetes Best Practices 1st Edition Brendan Burns ebooks...
Immediate download Kubernetes Best Practices 1st Edition Brendan Burns ebooks...
seinersofhia
 
Kubernetes security with AWS
Kubernetes security with AWS
Kasun Madura Rathnayaka
 
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware
VMUG IT
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Sonja Schweigert
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Weaveworks
 
K8Guard - An Auditing System For Kubernetes
K8Guard - An Auditing System For Kubernetes
Medya Ghazizadeh
 
Cloud-Native Operations with Kubernetes and CI/CD
Cloud-Native Operations with Kubernetes and CI/CD
VMware Tanzu
 
Kubernetes and Terraform in the Cloud: How RightScale Does DevOps
Kubernetes and Terraform in the Cloud: How RightScale Does DevOps
RightScale
 
Free GitOps Workshop + Intro to Kubernetes & GitOps
Free GitOps Workshop + Intro to Kubernetes & GitOps
Weaveworks
 
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Krishna-Kumar
 
stackconf 2020 | The blinking cursor or kubernetes for people who aren´t supp...
stackconf 2020 | The blinking cursor or kubernetes for people who aren´t supp...
NETWAYS
 
Future of Kubernetes and its Impact on Technology Industry.pdf
Future of Kubernetes and its Impact on Technology Industry.pdf
Urolime Technologies
 
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC
 
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptx
FIDO Alliance
 

More Related Content

Similar to Successful K8S Platforms in Airgapped Environments (20)

Simplify Your Way To Expert Kubernetes Management
Simplify Your Way To Expert Kubernetes Management
DevOps.com
 
LISA2017 Kubernetes: Hit the Ground Running
LISA2017 Kubernetes: Hit the Ground Running
Chris McEniry
 
DNUG46 - Build your own private Cloud environment
DNUG46 - Build your own private Cloud environment
panagenda
 
Build your own private Cloud environment
Build your own private Cloud environment
Nico Meisenzahl
 
KubeCon 2019 Recap (Parts 1-3)
KubeCon 2019 Recap (Parts 1-3)
Ford Prior
 
Meetup 2023 - Gateway API.pdf
Meetup 2023 - Gateway API.pdf
Red Hat
 
Running and Managing Kubernetes on OpenStack
Running and Managing Kubernetes on OpenStack
Victor Palma
 
Presentation ING for ISC2 Secure Summits EMEA
Presentation ING for ISC2 Secure Summits EMEA
Thijs Ebbers
 
Immediate download Kubernetes Best Practices 1st Edition Brendan Burns ebooks...
Immediate download Kubernetes Best Practices 1st Edition Brendan Burns ebooks...
seinersofhia
 
Kubernetes security with AWS
Kubernetes security with AWS
Kasun Madura Rathnayaka
 
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware
VMUG IT
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Sonja Schweigert
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Weaveworks
 
K8Guard - An Auditing System For Kubernetes
K8Guard - An Auditing System For Kubernetes
Medya Ghazizadeh
 
Cloud-Native Operations with Kubernetes and CI/CD
Cloud-Native Operations with Kubernetes and CI/CD
VMware Tanzu
 
Kubernetes and Terraform in the Cloud: How RightScale Does DevOps
Kubernetes and Terraform in the Cloud: How RightScale Does DevOps
RightScale
 
Free GitOps Workshop + Intro to Kubernetes & GitOps
Free GitOps Workshop + Intro to Kubernetes & GitOps
Weaveworks
 
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Krishna-Kumar
 
stackconf 2020 | The blinking cursor or kubernetes for people who aren´t supp...
stackconf 2020 | The blinking cursor or kubernetes for people who aren´t supp...
NETWAYS
 
Future of Kubernetes and its Impact on Technology Industry.pdf
Future of Kubernetes and its Impact on Technology Industry.pdf
Urolime Technologies
 
Simplify Your Way To Expert Kubernetes Management
Simplify Your Way To Expert Kubernetes Management
DevOps.com
 
LISA2017 Kubernetes: Hit the Ground Running
LISA2017 Kubernetes: Hit the Ground Running
Chris McEniry
 
DNUG46 - Build your own private Cloud environment
DNUG46 - Build your own private Cloud environment
panagenda
 
Build your own private Cloud environment
Build your own private Cloud environment
Nico Meisenzahl
 
KubeCon 2019 Recap (Parts 1-3)
KubeCon 2019 Recap (Parts 1-3)
Ford Prior
 
Meetup 2023 - Gateway API.pdf
Meetup 2023 - Gateway API.pdf
Red Hat
 
Running and Managing Kubernetes on OpenStack
Running and Managing Kubernetes on OpenStack
Victor Palma
 
Presentation ING for ISC2 Secure Summits EMEA
Presentation ING for ISC2 Secure Summits EMEA
Thijs Ebbers
 
Immediate download Kubernetes Best Practices 1st Edition Brendan Burns ebooks...
Immediate download Kubernetes Best Practices 1st Edition Brendan Burns ebooks...
seinersofhia
 
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware
VMUG IT
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Sonja Schweigert
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Weaveworks
 
K8Guard - An Auditing System For Kubernetes
K8Guard - An Auditing System For Kubernetes
Medya Ghazizadeh
 
Cloud-Native Operations with Kubernetes and CI/CD
Cloud-Native Operations with Kubernetes and CI/CD
VMware Tanzu
 
Kubernetes and Terraform in the Cloud: How RightScale Does DevOps
Kubernetes and Terraform in the Cloud: How RightScale Does DevOps
RightScale
 
Free GitOps Workshop + Intro to Kubernetes & GitOps
Free GitOps Workshop + Intro to Kubernetes & GitOps
Weaveworks
 
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Krishna-Kumar
 
stackconf 2020 | The blinking cursor or kubernetes for people who aren´t supp...
stackconf 2020 | The blinking cursor or kubernetes for people who aren´t supp...
NETWAYS
 
Future of Kubernetes and its Impact on Technology Industry.pdf
Future of Kubernetes and its Impact on Technology Industry.pdf
Urolime Technologies
 

Recently uploaded (20)

OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC
 
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptx
FIDO Alliance
 
Daily Lesson Log MATATAG ICT TEchnology 8
Daily Lesson Log MATATAG ICT TEchnology 8
LOIDAALMAZAN3
 
"Database isolation: how we deal with hundreds of direct connections to the d...
"Database isolation: how we deal with hundreds of direct connections to the d...
Fwdays
 
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
yosra Saidani
 
A Constitutional Quagmire - Ethical Minefields of AI, Cyber, and Privacy.pdf
A Constitutional Quagmire - Ethical Minefields of AI, Cyber, and Privacy.pdf
Priyanka Aash
 
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
Priyanka Aash
 
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Josef Weingand
 
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
Edge AI and Vision Alliance
 
UserCon Belgium: Honey, VMware increased my bill
UserCon Belgium: Honey, VMware increased my bill
stijn40
 
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
Fwdays
 
"Scaling in space and time with Temporal", Andriy Lupa.pdf
"Scaling in space and time with Temporal", Andriy Lupa.pdf
Fwdays
 
Cyber Defense Matrix Workshop - RSA Conference
Cyber Defense Matrix Workshop - RSA Conference
Priyanka Aash
 
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Priyanka Aash
 
PyCon SG 25 - Firecracker Made Easy with Python.pdf
PyCon SG 25 - Firecracker Made Easy with Python.pdf
Muhammad Yuga Nugraha
 
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?
Shashi Sathyanarayana, Ph.D
 
From Manual to Auto Searching- FME in the Driver's Seat
From Manual to Auto Searching- FME in the Driver's Seat
Safe Software
 
Quantum AI Discoveries: Fractal Patterns Consciousness and Cyclical Universes
Quantum AI Discoveries: Fractal Patterns Consciousness and Cyclical Universes
Saikat Basu
 
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
caoyixuan2019
 
You are not excused! How to avoid security blind spots on the way to production
You are not excused! How to avoid security blind spots on the way to production
Michele Leroux Bustamante
 
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC
 
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptx
FIDO Alliance
 
Daily Lesson Log MATATAG ICT TEchnology 8
Daily Lesson Log MATATAG ICT TEchnology 8
LOIDAALMAZAN3
 
"Database isolation: how we deal with hundreds of direct connections to the d...
"Database isolation: how we deal with hundreds of direct connections to the d...
Fwdays
 
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
yosra Saidani
 
A Constitutional Quagmire - Ethical Minefields of AI, Cyber, and Privacy.pdf
A Constitutional Quagmire - Ethical Minefields of AI, Cyber, and Privacy.pdf
Priyanka Aash
 
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
Priyanka Aash
 
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Josef Weingand
 
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
Edge AI and Vision Alliance
 
UserCon Belgium: Honey, VMware increased my bill
UserCon Belgium: Honey, VMware increased my bill
stijn40
 
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
Fwdays
 
"Scaling in space and time with Temporal", Andriy Lupa.pdf
"Scaling in space and time with Temporal", Andriy Lupa.pdf
Fwdays
 
Cyber Defense Matrix Workshop - RSA Conference
Cyber Defense Matrix Workshop - RSA Conference
Priyanka Aash
 
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Priyanka Aash
 
PyCon SG 25 - Firecracker Made Easy with Python.pdf
PyCon SG 25 - Firecracker Made Easy with Python.pdf
Muhammad Yuga Nugraha
 
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?
Shashi Sathyanarayana, Ph.D
 
From Manual to Auto Searching- FME in the Driver's Seat
From Manual to Auto Searching- FME in the Driver's Seat
Safe Software
 
Quantum AI Discoveries: Fractal Patterns Consciousness and Cyclical Universes
Quantum AI Discoveries: Fractal Patterns Consciousness and Cyclical Universes
Saikat Basu
 
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
caoyixuan2019
 
You are not excused! How to avoid security blind spots on the way to production
You are not excused! How to avoid security blind spots on the way to production
Michele Leroux Bustamante
 

Successful K8S Platforms in Airgapped Environments

  • 1. Successful k8s Platforms in Airgapped Environments Drew Malone Staff Solutions Engineer Tanzu Federal VMware [email protected]
  • 2. 2 Overview - Problem Statement - ClusterAPI - a Primer - Challenges and Solutions: - Cloud Permissions - Node Image Creation - Cloud API Access - Bootstrapping and Troubleshooting - Now What? Creating a defensible Day 2 story
  • 3. 3 What do US Government Users Say About Kubernetes? ● Complex to manage Day Two Operations ● Disconnected Environments from Day One ● Need capability both in the cloud and at the edge (various meanings) ● Developer Experience is Lacking because of the complexity ● Ton of Hype about Kubernetes but Commanders and CIO’s want outcomes ● Everyone talks about installing and securing Kubernetes, but rarely do we see Developers pushing apps to production on kubernetes
  • 4. 4 What do US Government Users Say About Kubernetes? Standing up Airgapped Kubernetes is Hard Securing, Operating and Pushing Code to k8s is even Harder
  • 5. 5 Where Do I Begin?
  • 6. 6 Bring the declarative nature of app lifecycles in k8s…. to lifecycling k8s clusters kind: Cluster metadata: name: a-team-k8s-cluster namespace: a-team spec: distribution: version: v1.20 topology: controlPlane: count: 3 class: m5.xlarge Start at the Beginning - ClusterAPI
  • 8. 8 1. Configure IaaS Permissions 2. Configure IaaS API Access 3. Create node image (i.e. base OS image) 4. Bootstrap and Profit! But a lot has to happen first…
  • 9. 9 ClusterAPI creates roles in your cloud. This can be done a couple of ways: - directly added to your cloud - create a file to be handed to your PMO Do you have those rights? Is it a request to your PMO? Step 1 - IaaS Permissions ref: https://cluster-api.sigs.k8s.io/user/quick-start.html
  • 10. 10 API requests go over the Internet. But if there’s no Internet, how do you reach the API? Step 2 -IaaS API Access - Challenge
  • 11. 11 Clouds will provide a “endpoint service” to allow you to reach API sets without needing to create/maintain Internet connectivity. Step 2 -IaaS API Access - Solution
  • 12. 12 Image builder needs to download the entire Internet. But you need to bake in <special things> like private cloud CA’s, private registry CA’s, etc. Step 3 -Build the Node Image - Challenge
  • 13. 13 Build the image “outside”. Then use pre-kubeadm hooks in ClusterAPI to install certs and run any “last mile” configurations. This has the added benefit of being able to update things like CA’s without needing to cut another image. NB - are you using an Isolated cloud region? You need to ask about their diff doc. Step 3 -Build the Node Image - Solution ref: https://doc.crds.dev/github.com/kubernetes-sigs/cluster-api/bootstrap.cluster.x-k8s.io/KubeadmConfigTemplate/v1alph
  • 14. 14 If you absolutely must build the image inside the firewall, you can do that. But you need: - an OS repo (apt, yum) - k8s bits (either packaged, or zip) - a place to host said bits - to configure Packer - to trust your private cloud - [possibly] to disable the use of GP3 disk Step 3 -Build the Node Image - Addendum
  • 15. 15 Fire away and see what fails! Step 4 -Bootstrap and Profit! ref: https://cluster-api.sigs.k8s.io/user/quick-start.html
  • 16. 16 Hi-level (CLI) failures are exactly that - high level. To get to the source of truth, look in the pod logs in your KIND cluster: - capi-kubeadm-bootstrap-system - capi-kubeadm-control-plane-system - capa-kubeadm….. Step 5 -Troubleshooting
  • 17. 17 If there’s no Internet, where does your Load Balancer point? Remember to annotate your k8s services accordingly Step N -Load Balancers ref: https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer
  • 18. 18 Getting a k8s is only the start. How do you make it useful? You Have a Kubernetes! Now What?
  • 19. 19 Committed to chopping wood and carrying water VMware is a Leader in the Open Source Community Kubernetes contributions by release number according to k8s.devstats.cncf.io Cluster API ● 2 of 7 steering committee members ● 10 of 23 special interest groups led by VMware ● 4 working groups led by VMware ● 2nd leading contributor to kubernetes
  • 20. 20 Immutable Infrastructure ● K8s that is scaled up ● Destroyed ● Recreated ● Not persisted Cluster API ● Repave ● Rotate ● Repair
  • 21. 21 Immutable Infrastructure Continuous Reconciliation Multi Cluster k8’s Provenance In Secure Supply Chain Production Ready k8s ref: https://dodcio.defense.gov/Library/ (search for ‘multi-cluster kubernetes’)
  • 22. 22 Production Ready Kubernetes Cluster API Pinniped Management Cluster Cluster Lifecycle Management Container Networking Observability Shared and In-Cluster Services Container registry Ingress Controller API-Driven IaaS Declarative API Log Forwarding Load Balancing Identity & Authentication Backup and Restore Ingress & Services DNS Local Control Plane Platform Operator uses GitOps Workload Clusters Workload Clusters Workload Clusters External-dns
  • 23. 23 Characteristics of a “great” k8s platform ● Rapid patching of k8s and applications/services ● Prevention of config drift ● Align with approved baselines (i.e. gov’t regs) ● Build once, Run anywhere
  • 24. 24 How to get a “great” k8s platform ● Immutable Infrastructure (aka image-builder) ● Continuous Reconciliation ● Multi-cluster Kubernetes ● Provenance in Secure Supply Chain ● Prod-ready k8s, even in Dev
  • 26. 26 Immutable Infrastructure and Continuous Reconciliation Control Plane Manifest Manifest Manifest Reconciliation Loop Kapp-Controller Workload Clusters Git Repo Cluster Creation and Enforcement Day 2 Reconciliation Cluster API Operator Desired State Kapp-Controller Pinniped
  • 27. 27 Edge Deployment - Build Once, Run Anywhere Public Cloud Development and Test Environments Disconnected Production(Edge) Edge Local Control Plane Management Cluster Cluster Lifecycle Management Workload Clusters Workload Clusters Workload Clusters Local Control Plane Management Cluster Cluster Lifecycle Management Workload Clusters Workload Clusters Workload Clusters imgpkg
  • 28. 28 Summary - k8s is (still) hard airgapped but you can make it easy - ClusterAPI has everything you need - Know who owns Cloud Permissions - Create your Node images “outside” if possible - Use Endpoints for Cloud API Access - CAPI/CAP{x} pod logs for Troubleshooting - Create a defensible Day 2 story with immutable infra + gitops
  • 29. Thank You!!! Drew Malone Staff Solutions Engineer Tanzu Federal VMware [email protected]