SlideShare a Scribd company logo

Kubernetes deployment on bare metal with container linux

M
macchiang

This document discusses deploying Kubernetes on bare metal servers using Container Linux (CoreOS). It describes why bare metal and Container Linux are used, how to deploy the Kubernetes control plane and worker nodes, and how to configure Kubernetes components. The deployment uses CoreOS, etcd, flannel, and TLS assets to set up a highly available Kubernetes cluster on bare metal servers without virtualization. Matchbox can also be used for provisioning nodes by generating Ignition configs from profiles, groups and templates.

Technology
1 of 39
Downloaded 38 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
Kubernetes Deployment on Bare Metal with Container Linux 資訊與通訊研究所 Mac Chiang (蔣是文)
Copyright 2017 ITRI 工業技術研究院 Agenda • Why bare metal? • Why Container Linux? • How to deployment? • Conclusion 2
Copyright 2017 ITRI 工業技術研究院 Why bare metal? • Hardware can’t support virtualization ▪ CPU Model: Dual Core AMD Opteron(tm) Processor 270 • Better performance ▪ Bare metal vs. VM 3
Copyright 2017 ITRI 工業技術研究院 Why Container Linux (CoreOS)? • Lightweight Linux • Container optimized OS • Security focused • Auto update • Integrated well with Kubernetes 4
Copyright 2017 ITRI 工業技術研究院 Agenda • Why bare metal? • Why Container Linux? • How to deployment? • Conclusion 5
Copyright 2017 ITRI 工業技術研究院 Deployment Approach • Manual Installation (Step by Step) ▪ https://coreos.com/kubernetes/docs/latest/getting-started.html • Matchbox + ignition (Recommended) ▪ https://github.com/coreos/matchbox/tree/master/examples/groups/k8s- install 6
Copyright 2017 ITRI 工業技術研究院 Environment Deployment Server Node1:Controller,Etcd Node2: Worker Node3: Worker 7
Copyright 2017 ITRI 工業技術研究院 CoreOS + Kubernetes Steps • Install CoreOS • Setup an etcd cluster • Generate the certificates for Kubernetes components • Deploy a controller (master) node • Deploy worker nodes • Configure kubectl to work with our cluster • Deploy the add-ons ▪ DNS ▪ Dashboard 8
Copyright 2017 ITRI 工業技術研究院 Install CoreOS • PXE and iPXE ▪ Booting with iPXE ▪ Booting with PXE ▪ Required RAM :1024M+ • Disk ▪ Installing to Disk Boot from PXE or iPXE Boot from ISO Install to Disk coreos-install -d /dev/sda -c cloud-config.yaml 9
Copyright 2017 ITRI 工業技術研究院 What’s etcd? • Distributed key, value store • Used for configuration and monitoring store • Used for Service discovery • JSON/REST API 10
Copyright 2017 ITRI 工業技術研究院 Deploy etcd Cluster Single-Node/Development Multi-Node/Production https://coreos.com/os/docs/latest/cluster-architectures.html 11
Copyright 2017 ITRI 工業技術研究院 What’s flannel? • A virtual network that gives a subnet to each host for use with container runtimes 12
Copyright 2017 ITRI 工業技術研究院 Deployment Options • MASTER_HOST ▪ Publicly routable IP of master node. a. Worker nodes must be able to reach the master node(s) via this address on port 443 ▪ Multiple master nodes a. Network load balancer b. DNS configure • ETCD_ENDPOINTS ▪ List of etcd machines (http://ip1:port,http://ip2:port,http://ip3:p ort) • POD_NETWORK=10.2.0.0/16 ▪ The flannel overlay network will provide routing to this network. • SERVICE_IP_RANGE=10.3.0.0/24 ▪ The CIDR network to use for service cluster VIPs (Virtual IPs) ▪ Handled by a local kube-proxy service to each host • K8S_SERVICE_IP=10.3.0.1 ▪ The VIP (Virtual IP) address of the Kubernetes API Service. • DNS_SERVICE_IP=10.3.0.10 ▪ The VIP (Virtual IP) address of the cluster DNS service. 13
Copyright 2017 ITRI 工業技術研究院 Generate Kubernetes TLS Assets • Root CA Public Key ▪ ca.pem • API Server Public & Private Keys ▪ apiserver.pem ▪ apiserver-key.pem • Worker Node Public & Private Keys ▪ ${WORKER_FQDN}-worker.pem ▪ ${WORKER_FQDN}-worker-key.pem • Cluster Admin Public & Private Keys ▪ admin.pem ▪ admin-key.pem https://coreos.com/kubernetes/docs/latest/openssl.html 14
Copyright 2017 ITRI 工業技術研究院 Deploy Kubernetes Master Node • Configure Service Components ▪ TLS Assets ▪ Network Configuration ▪ Docker Configuration ▪ Create the kubelet Unit ▪ Set Up the kube-* Pod a. kube-apiserver b. kube-proxy c. kube-controller-manager d. kube-scheduler • Start Services ▪ Load Changed Units ▪ Configure flannel Network ▪ Start kubelet ▪ Basic Health Checks 15
Copyright 2017 ITRI 工業技術研究院 Master TLS Assets • /etc/kubernetes/ssl/ca.pem • /etc/kubernetes/ssl/apiserver.pem • /etc/kubernetes/ssl/apiserver-key.pem 16
Copyright 2017 ITRI 工業技術研究院 Network & Docker Configuration /etc/flannel/options.env FLANNELD_ETCD_ENDPOINTS=${ETCD_ENDPOINTS} 17
Copyright 2017 ITRI 工業技術研究院 Kubelet Unit and Kube-* PODs /etc/systemd/system/ kubelet.service /usr/lib/coreos/kubelet-wrapper --pod-manifest-path=/etc/kubernetes/manifests Hyperkube /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-proxy.yaml /etc/kubernetes/manifests/kube-controller-manager.yaml /etc/kubernetes/manifests/kube-scheduler.yaml An all-in-one binary for the Kubernetes server components 18
Copyright 2017 ITRI 工業技術研究院 Start Services • Load Changed Units • Configure flannel Network • Start kubelet • Basic Health Checks curl http://127.0.0.1:8080/version 19
Copyright 2017 ITRI 工業技術研究院 Deploy Kubernetes Worker Node • Configure Service Components ▪ TLS Assets ▪ Networking Configuration ▪ Docker Configuration ▪ Create the kubelet Unit ▪ Set Up the kube-proxy Pod ▪ Set Up kubeconfig • Start Services ▪ Load Changed Units ▪ Start kubelet, and flannel 20
Copyright 2017 ITRI 工業技術研究院 Worker TLS Assets • /etc/kubernetes/ssl/ca.pem • /etc/kubernetes/ssl/${WORKER_FQDN}-worker.pem • /etc/kubernetes/ssl/${WORKER_FQDN}-worker- key.pem 21
Copyright 2017 ITRI 工業技術研究院 Kubelet Unit and kube-proxy/kubeconfig /etc/systemd/system/ kubelet.service /usr/lib/coreos/kubelet-wrapper --api-servers=https://${MASTER_HOST} --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml --pod-manifest-path=/etc/kubernetes/manifests Hyperkube /etc/kubernetes/manifests/kube-proxy.yaml An all-in-one binary for the Kubernetes server components 22
Copyright 2017 ITRI 工業技術研究院 Start Services • Load Changed Units • Start kubelet, and flannel 23
Copyright 2017 ITRI 工業技術研究院 Configure kubectl • Download the kubectl Executable • Configure kubectl ▪ Master server host ▪ Root CA public key ▪ Cluster admin public & private Keys • Verify kubectl Configuration and Connection kubectl get nodes NAME LABELS STATUS X.X.X.X kubernetes.io/hostname=X.X.X.X Ready • Enabling shell autocompletion echo "source <(kubectl completion bash)" >> ~/.bashrc 24
Copyright 2017 ITRI 工業技術研究院 Deploy the Add-ons • DNS • Dashboard kubectl port-forward kubernetes-dashboard-xxxx 9090 --namespace=kube- system Then visit http://127.0.0.1:9090 in your browser. 25
Copyright 2017 ITRI 工業技術研究院 Kube Dashboard namespace=kube-system 26
Copyright 2017 ITRI 工業技術研究院 What’s MatchBox? • HTTP and gRPC service that renders signed Ignition configs, cloud-configs, network boot configs, and metadata to machines to create CoreOS clusters 27
Copyright 2017 ITRI 工業技術研究院 Machbox workflow https://github.com/coreos/matchbox/blob/master/Documentation/matchbox.md 28
Copyright 2017 ITRI 工業技術研究院 Matchbox Steps • Get CoreOS • Generate TLS assets • Prepare groups, profiles and ignition files • Setup dnsmasq and matchbox container • Start deployment • Configure kubectl to work with our cluster • Check all PODs and Services 29
Copyright 2017 ITRI 工業技術研究院 Get CoreOS ./scripts/get-coreos channel version examples/assets/ └── coreos └── 1298.6.0 ├── CoreOS_Image_Signing_Key.asc ├── coreos_production_image.bin.bz2 ├── coreos_production_image.bin.bz2.sig ├── coreos_production_pxe_image.cpio.gz ├── coreos_production_pxe_image.cpio.gz.sig ├── coreos_production_pxe.vmlinuz └── coreos_production_pxe.vmlinuz.sig https://github.com/coreos/matchbox/tree/master/scripts 30
Copyright 2017 ITRI 工業技術研究院 Generate TLS Assets ./scripts/tls/k8s-certgen -h Usage: k8s-certgen Options: -d DEST Destination for generated files (default: .examples/assets/tls) -s SERVER Reachable Server IP for kubeconfig (e.g. node1.example.com) -m MASTERS Controller Node Names/Addresses in SAN format (e.g. IP.1=10.3.0.1,DNS.1=node1.example.com) -w WORKERS Worker Node Names/Addresses in SAN format (e.g. DNS.1=node2.example.com,DNS.2=node3.example.com) -h Show help 31
Copyright 2017 ITRI 工業技術研究院 Prepare groups, profiles and ignition examples/ ├── assets │ ├── coreos │ │ ├── 1298.6.0 │ │ └── tls ├── groups │ ├── install.json │ ├── node1.json │ ├── node2.json │ └── node3.json ├── profiles │ ├── install-reboot.json │ ├── k8s-controller.json │ └── k8s-worker.json └──ignition ├── install-reboot.yaml ├── k8s-controller.yaml └── k8s-worker.yaml https://github.com/coreos/matchbox/tree/master/examples/groups/k8s-install 32
Copyright 2017 ITRI 工業技術研究院 Installation Flow install.json install- reboot.json install- reboot.yaml curl "{{.ignition_endpoint}}?{{.request.r aw_query}}&os=installed" -o ignition.json node1.json k8s- controller.json "selector": { "os": "installed", "mac": "00:26:2d:06:ff:bc" }, k8s- controller.yaml "coreos_channel": "stable", "coreos_version": “1298.6.0", 33
Copyright 2017 ITRI 工業技術研究院 Setup dnsmasq and matchbox • Dnsmasq docker run --name dnsmasq --cap-add=NET_ADMIN --network="host" -v $PWD/dnsmasq.conf:/etc/dnsmasq.conf:z quay.io/coreos/dnsmasq -d • Matchbox docker run -p 8080:8080 --rm -v $PWD/example:/var/lib/matchbox:Z quay.io/coreos/matchbox:latest -address=0.0.0.0:8080 -log-level=debug Notice: Don’t forget to open firewall port for matchbox(8080), dns, tftp and dhcp 34
Copyright 2017 ITRI 工業技術研究院 PXE boot time="2017-04-05T07:31:13Z" level=info msg="Starting matchbox HTTP server on 0.0.0.0:8080" time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /boot.ipxe" time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /ipxe?uuid=03000200-0400-0500-0006-000700080009&mac=00- 26-2d-07-00-78&domain=k8s.itri&hostname=WR1-43&serial=To%20Be%20Filled%20By%20O.E.M." time="2017-04-05T07:34:03Z" level=debug msg="Matched an iPXE config" labels=map[uuid:03000200-0400-0500-0006- 000700080009 mac:00:26:2d:07:00:78 domain:k8s.itri hostname:WR1-43 serial:To Be Filled By O.E.M.] profile=install-reboot time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /assets/coreos/current/coreos_production_pxe.vmlinuz" time="2017-04-05T07:34:04Z" level=info msg="HTTP GET /assets/coreos/current/coreos_production_pxe_image.cpio.gz" time="2017-04-05T07:36:29Z" level=info msg="HTTP GET /ignition?uuid=03000200-0400-0500-0006- 000700080009&mac=00-26-2d-07-00-78&os=installed" time="2017-04-05T07:36:29Z" level=debug msg="Matched an Ignition or Fuze template" group=node3 labels=map[uuid:03000200-0400-0500-0006-000700080009 mac:00:26:2d:07:00:78 os:installed] profile=k8s-controller matchbox logs Demo: https://youtu.be/z9eYOuWLc8k 35
Copyright 2017 ITRI 工業技術研究院 Configure kubectl • Use the generated kubeconfig directly KUBECONFIG=examples/assets/tls/kubeconfig • Overwrite kubeconfig cp examples/assets/tls/kubeconfig ~/.kube/config 36
Copyright 2017 ITRI 工業技術研究院 Check all PODs and Services [root@centos7 matchbox]# kubectl get po --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system heapster-v1.2.0-4088228293-7vwxd 2/2 Running 0 15h kube-system kube-apiserver-10.201.3.44 1/1 Running 0 15h kube-system kube-controller-manager-10.201.3.44 1/1 Running 0 15h kube-system kube-dns-782804071-j52dv 4/4 Running 0 15h kube-system kube-dns-autoscaler-2715466192-krz0p 1/1 Running 0 15h kube-system kube-proxy-10.201.3.42 1/1 Running 0 15h kube-system kube-proxy-10.201.3.43 1/1 Running 0 15h kube-system kube-proxy-10.201.3.44 1/1 Running 0 15h kube-system kube-scheduler-10.201.3.44 1/1 Running 0 15h kube-system kubernetes-dashboard-3543765157-xj185 1/1 Running 0 15h [root@centos7 matchbox]# kubectl get svc --all-namespaces NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes 10.3.0.1 <none> 443/TCP 15h kube-system heapster 10.3.0.95 <none> 80/TCP 15h kube-system kube-dns 10.3.0.10 <none> 53/UDP,53/TCP 15h kube-system kubernetes-dashboard 10.3.0.66 <none> 80/TCP 15h 37
Copyright 2017 ITRI 工業技術研究院 Conclusion • Container Linux (CoreOS) is a good choice for bare metal & production • Manual installation vs. Matchbox+ignition • What’s next? ▪ Try it ▪ Join Kubernetes Taiwan User Group ▪ Kubernetes Training Courses and Playground a. https://www.katacoda.com/courses/kubernetes b. https://www.katacoda.com/courses/kubernetes/playground 38
Thank you! macchiang@itri.org.tw Kubernetes Taiwan User Group

More Related Content

What's hot (20)

PDF
Container Security
Salman Baset
 
PDF
Cloud-native applications with Java and Kubernetes - Yehor Volkov
Kuberton
 
PPTX
Docker Container Security - A Network View
NeuVector
 
PDF
Veer's Container Security
Jim Barlow
 
PPTX
Understanding container security
John Kinsella
 
PDF
Container Security Essentials
DNIF
 
PDF
CI / CD / CS - Continuous Security in Kubernetes
Sysdig
 
PPTX
Docker Container Security
Suraj Khetani
 
PPTX
Csa container-security-in-aws-dw
Cloud Security Alliance, UK chapter
 
PDF
Deploying Kubernetes without scaring off your security team - KubeCon 2017
Major Hayden
 
PDF
Cloud networking deep dive
amylynn11
 
PPT
Container security
Anthony Chow
 
PDF
OSS Japan 2019 service mesh bridging Kubernetes and legacy
Steve Wong
 
PDF
Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13
Zach Hill
 
PDF
Kubernetes Security
inovex GmbH
 
PDF
Docker security introduction-task-2016
Ricardo Gerardi
 
PDF
Effective Data Pipelines with Docker & Jenkins - Brian Donaldson
Docker, Inc.
 
PDF
Why you need a private container image registry SCALE 2019
Steve Wong
 
PDF
Production grade edge computing on Kubernetes OSS EU 2018
Steve Wong
 
PDF
DockerCon SF 2015: Docker Security
Docker, Inc.
 
Container Security
Salman Baset
 
Cloud-native applications with Java and Kubernetes - Yehor Volkov
Kuberton
 
Docker Container Security - A Network View
NeuVector
 
Veer's Container Security
Jim Barlow
 
Understanding container security
John Kinsella
 
Container Security Essentials
DNIF
 
CI / CD / CS - Continuous Security in Kubernetes
Sysdig
 
Docker Container Security
Suraj Khetani
 
Csa container-security-in-aws-dw
Cloud Security Alliance, UK chapter
 
Deploying Kubernetes without scaring off your security team - KubeCon 2017
Major Hayden
 
Cloud networking deep dive
amylynn11
 
Container security
Anthony Chow
 
OSS Japan 2019 service mesh bridging Kubernetes and legacy
Steve Wong
 
Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13
Zach Hill
 
Kubernetes Security
inovex GmbH
 
Docker security introduction-task-2016
Ricardo Gerardi
 
Effective Data Pipelines with Docker & Jenkins - Brian Donaldson
Docker, Inc.
 
Why you need a private container image registry SCALE 2019
Steve Wong
 
Production grade edge computing on Kubernetes OSS EU 2018
Steve Wong
 
DockerCon SF 2015: Docker Security
Docker, Inc.
 

Similar to Kubernetes deployment on bare metal with container linux (20)

PDF
Kubernetes in kubernetes 搭建高可用環境
inwin stack
 
PDF
Kubernetes and devops
macchiang
 
PPTX
CKA_1st.pptx
YIJHEHUANG
 
PDF
Introduction of k8s rancher
cyberblack28 Ichikawa
 
PPTX
kubernetes baremetal installation and practice
wonyong hwang
 
PDF
Introduction of kubernetes rancher
cyberblack28 Ichikawa
 
PPTX
Kubernetes Introduction
Eric Gustafson
 
PDF
Build Your Own CaaS (Container as a Service)
HungWei Chiu
 
PDF
Kubernetes Basis: Pods, Deployments, and Services
Jian-Kai Wang
 
PDF
Kubernetes from scratch at veepee sysadmins days 2019
🔧 Loïc BLOT
 
PPTX
Tectonic Summit 2016: Kubernetes 1.5 and Beyond
CoreOS
 
PDF
Hands-On Introduction to Kubernetes at LISA17
Ryan Jarvinen
 
PDF
Demystifying kubernetes
Works Applications
 
PDF
Kubernetes111111111111111111122233334334
adnansalam11
 
PDF
Kubernetes - Starting with 1.2
William Stewart
 
PPTX
It's not too late to learn about k8s
Cesar Tron-Lozai
 
PPTX
使用 Prometheus 監控 Kubernetes Cluster
inwin stack
 
PDF
Kubernetes
Meng-Ze Lee
 
PPTX
K8s in 3h - Kubernetes Fundamentals Training
Piotr Perzyna
 
PDF
Kubernetes Boston — Custom High Availability of Kubernetes
Mike Splain
 
Kubernetes in kubernetes 搭建高可用環境
inwin stack
 
Kubernetes and devops
macchiang
 
CKA_1st.pptx
YIJHEHUANG
 
Introduction of k8s rancher
cyberblack28 Ichikawa
 
kubernetes baremetal installation and practice
wonyong hwang
 
Introduction of kubernetes rancher
cyberblack28 Ichikawa
 
Kubernetes Introduction
Eric Gustafson
 
Build Your Own CaaS (Container as a Service)
HungWei Chiu
 
Kubernetes Basis: Pods, Deployments, and Services
Jian-Kai Wang
 
Kubernetes from scratch at veepee sysadmins days 2019
🔧 Loïc BLOT
 
Tectonic Summit 2016: Kubernetes 1.5 and Beyond
CoreOS
 
Hands-On Introduction to Kubernetes at LISA17
Ryan Jarvinen
 
Demystifying kubernetes
Works Applications
 
Kubernetes111111111111111111122233334334
adnansalam11
 
Kubernetes - Starting with 1.2
William Stewart
 
It's not too late to learn about k8s
Cesar Tron-Lozai
 
使用 Prometheus 監控 Kubernetes Cluster
inwin stack
 
Kubernetes
Meng-Ze Lee
 
K8s in 3h - Kubernetes Fundamentals Training
Piotr Perzyna
 
Kubernetes Boston — Custom High Availability of Kubernetes
Mike Splain
 
Ad

Recently uploaded (20)

PDF
NLJUG Speaker academy 2025 - first session
Bert Jan Schrijver
 
PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
PDF
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
PPTX
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PDF
Automating Feature Enrichment and Station Creation in Natural Gas Utility Net...
Safe Software
 
PDF
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
PPT
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PDF
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
PDF
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
PDF
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
 
PDF
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
PDF
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
PDF
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
NLJUG Speaker academy 2025 - first session
Bert Jan Schrijver
 
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
Automating Feature Enrichment and Station Creation in Natural Gas Utility Net...
Safe Software
 
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
Staying Human in a Machine- Accelerated World
Catalin Jora
 
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
Ad

Kubernetes deployment on bare metal with container linux

  • 1. Kubernetes Deployment on Bare Metal with Container Linux 資訊與通訊研究所 Mac Chiang (蔣是文)
  • 2. Copyright 2017 ITRI 工業技術研究院 Agenda • Why bare metal? • Why Container Linux? • How to deployment? • Conclusion 2
  • 3. Copyright 2017 ITRI 工業技術研究院 Why bare metal? • Hardware can’t support virtualization ▪ CPU Model: Dual Core AMD Opteron(tm) Processor 270 • Better performance ▪ Bare metal vs. VM 3
  • 4. Copyright 2017 ITRI 工業技術研究院 Why Container Linux (CoreOS)? • Lightweight Linux • Container optimized OS • Security focused • Auto update • Integrated well with Kubernetes 4
  • 5. Copyright 2017 ITRI 工業技術研究院 Agenda • Why bare metal? • Why Container Linux? • How to deployment? • Conclusion 5
  • 6. Copyright 2017 ITRI 工業技術研究院 Deployment Approach • Manual Installation (Step by Step) ▪ https://coreos.com/kubernetes/docs/latest/getting-started.html • Matchbox + ignition (Recommended) ▪ https://github.com/coreos/matchbox/tree/master/examples/groups/k8s- install 6
  • 7. Copyright 2017 ITRI 工業技術研究院 Environment Deployment Server Node1:Controller,Etcd Node2: Worker Node3: Worker 7
  • 8. Copyright 2017 ITRI 工業技術研究院 CoreOS + Kubernetes Steps • Install CoreOS • Setup an etcd cluster • Generate the certificates for Kubernetes components • Deploy a controller (master) node • Deploy worker nodes • Configure kubectl to work with our cluster • Deploy the add-ons ▪ DNS ▪ Dashboard 8
  • 9. Copyright 2017 ITRI 工業技術研究院 Install CoreOS • PXE and iPXE ▪ Booting with iPXE ▪ Booting with PXE ▪ Required RAM :1024M+ • Disk ▪ Installing to Disk Boot from PXE or iPXE Boot from ISO Install to Disk coreos-install -d /dev/sda -c cloud-config.yaml 9
  • 10. Copyright 2017 ITRI 工業技術研究院 What’s etcd? • Distributed key, value store • Used for configuration and monitoring store • Used for Service discovery • JSON/REST API 10
  • 11. Copyright 2017 ITRI 工業技術研究院 Deploy etcd Cluster Single-Node/Development Multi-Node/Production https://coreos.com/os/docs/latest/cluster-architectures.html 11
  • 12. Copyright 2017 ITRI 工業技術研究院 What’s flannel? • A virtual network that gives a subnet to each host for use with container runtimes 12
  • 13. Copyright 2017 ITRI 工業技術研究院 Deployment Options • MASTER_HOST ▪ Publicly routable IP of master node. a. Worker nodes must be able to reach the master node(s) via this address on port 443 ▪ Multiple master nodes a. Network load balancer b. DNS configure • ETCD_ENDPOINTS ▪ List of etcd machines (http://ip1:port,http://ip2:port,http://ip3:p ort) • POD_NETWORK=10.2.0.0/16 ▪ The flannel overlay network will provide routing to this network. • SERVICE_IP_RANGE=10.3.0.0/24 ▪ The CIDR network to use for service cluster VIPs (Virtual IPs) ▪ Handled by a local kube-proxy service to each host • K8S_SERVICE_IP=10.3.0.1 ▪ The VIP (Virtual IP) address of the Kubernetes API Service. • DNS_SERVICE_IP=10.3.0.10 ▪ The VIP (Virtual IP) address of the cluster DNS service. 13
  • 14. Copyright 2017 ITRI 工業技術研究院 Generate Kubernetes TLS Assets • Root CA Public Key ▪ ca.pem • API Server Public & Private Keys ▪ apiserver.pem ▪ apiserver-key.pem • Worker Node Public & Private Keys ▪ ${WORKER_FQDN}-worker.pem ▪ ${WORKER_FQDN}-worker-key.pem • Cluster Admin Public & Private Keys ▪ admin.pem ▪ admin-key.pem https://coreos.com/kubernetes/docs/latest/openssl.html 14
  • 15. Copyright 2017 ITRI 工業技術研究院 Deploy Kubernetes Master Node • Configure Service Components ▪ TLS Assets ▪ Network Configuration ▪ Docker Configuration ▪ Create the kubelet Unit ▪ Set Up the kube-* Pod a. kube-apiserver b. kube-proxy c. kube-controller-manager d. kube-scheduler • Start Services ▪ Load Changed Units ▪ Configure flannel Network ▪ Start kubelet ▪ Basic Health Checks 15
  • 16. Copyright 2017 ITRI 工業技術研究院 Master TLS Assets • /etc/kubernetes/ssl/ca.pem • /etc/kubernetes/ssl/apiserver.pem • /etc/kubernetes/ssl/apiserver-key.pem 16
  • 17. Copyright 2017 ITRI 工業技術研究院 Network & Docker Configuration /etc/flannel/options.env FLANNELD_ETCD_ENDPOINTS=${ETCD_ENDPOINTS} 17
  • 18. Copyright 2017 ITRI 工業技術研究院 Kubelet Unit and Kube-* PODs /etc/systemd/system/ kubelet.service /usr/lib/coreos/kubelet-wrapper --pod-manifest-path=/etc/kubernetes/manifests Hyperkube /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-proxy.yaml /etc/kubernetes/manifests/kube-controller-manager.yaml /etc/kubernetes/manifests/kube-scheduler.yaml An all-in-one binary for the Kubernetes server components 18
  • 19. Copyright 2017 ITRI 工業技術研究院 Start Services • Load Changed Units • Configure flannel Network • Start kubelet • Basic Health Checks curl http://127.0.0.1:8080/version 19
  • 20. Copyright 2017 ITRI 工業技術研究院 Deploy Kubernetes Worker Node • Configure Service Components ▪ TLS Assets ▪ Networking Configuration ▪ Docker Configuration ▪ Create the kubelet Unit ▪ Set Up the kube-proxy Pod ▪ Set Up kubeconfig • Start Services ▪ Load Changed Units ▪ Start kubelet, and flannel 20
  • 21. Copyright 2017 ITRI 工業技術研究院 Worker TLS Assets • /etc/kubernetes/ssl/ca.pem • /etc/kubernetes/ssl/${WORKER_FQDN}-worker.pem • /etc/kubernetes/ssl/${WORKER_FQDN}-worker- key.pem 21
  • 22. Copyright 2017 ITRI 工業技術研究院 Kubelet Unit and kube-proxy/kubeconfig /etc/systemd/system/ kubelet.service /usr/lib/coreos/kubelet-wrapper --api-servers=https://${MASTER_HOST} --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml --pod-manifest-path=/etc/kubernetes/manifests Hyperkube /etc/kubernetes/manifests/kube-proxy.yaml An all-in-one binary for the Kubernetes server components 22
  • 23. Copyright 2017 ITRI 工業技術研究院 Start Services • Load Changed Units • Start kubelet, and flannel 23
  • 24. Copyright 2017 ITRI 工業技術研究院 Configure kubectl • Download the kubectl Executable • Configure kubectl ▪ Master server host ▪ Root CA public key ▪ Cluster admin public & private Keys • Verify kubectl Configuration and Connection kubectl get nodes NAME LABELS STATUS X.X.X.X kubernetes.io/hostname=X.X.X.X Ready • Enabling shell autocompletion echo "source <(kubectl completion bash)" >> ~/.bashrc 24
  • 25. Copyright 2017 ITRI 工業技術研究院 Deploy the Add-ons • DNS • Dashboard kubectl port-forward kubernetes-dashboard-xxxx 9090 --namespace=kube- system Then visit http://127.0.0.1:9090 in your browser. 25
  • 26. Copyright 2017 ITRI 工業技術研究院 Kube Dashboard namespace=kube-system 26
  • 27. Copyright 2017 ITRI 工業技術研究院 What’s MatchBox? • HTTP and gRPC service that renders signed Ignition configs, cloud-configs, network boot configs, and metadata to machines to create CoreOS clusters 27
  • 28. Copyright 2017 ITRI 工業技術研究院 Machbox workflow https://github.com/coreos/matchbox/blob/master/Documentation/matchbox.md 28
  • 29. Copyright 2017 ITRI 工業技術研究院 Matchbox Steps • Get CoreOS • Generate TLS assets • Prepare groups, profiles and ignition files • Setup dnsmasq and matchbox container • Start deployment • Configure kubectl to work with our cluster • Check all PODs and Services 29
  • 30. Copyright 2017 ITRI 工業技術研究院 Get CoreOS ./scripts/get-coreos channel version examples/assets/ └── coreos └── 1298.6.0 ├── CoreOS_Image_Signing_Key.asc ├── coreos_production_image.bin.bz2 ├── coreos_production_image.bin.bz2.sig ├── coreos_production_pxe_image.cpio.gz ├── coreos_production_pxe_image.cpio.gz.sig ├── coreos_production_pxe.vmlinuz └── coreos_production_pxe.vmlinuz.sig https://github.com/coreos/matchbox/tree/master/scripts 30
  • 31. Copyright 2017 ITRI 工業技術研究院 Generate TLS Assets ./scripts/tls/k8s-certgen -h Usage: k8s-certgen Options: -d DEST Destination for generated files (default: .examples/assets/tls) -s SERVER Reachable Server IP for kubeconfig (e.g. node1.example.com) -m MASTERS Controller Node Names/Addresses in SAN format (e.g. IP.1=10.3.0.1,DNS.1=node1.example.com) -w WORKERS Worker Node Names/Addresses in SAN format (e.g. DNS.1=node2.example.com,DNS.2=node3.example.com) -h Show help 31
  • 32. Copyright 2017 ITRI 工業技術研究院 Prepare groups, profiles and ignition examples/ ├── assets │ ├── coreos │ │ ├── 1298.6.0 │ │ └── tls ├── groups │ ├── install.json │ ├── node1.json │ ├── node2.json │ └── node3.json ├── profiles │ ├── install-reboot.json │ ├── k8s-controller.json │ └── k8s-worker.json └──ignition ├── install-reboot.yaml ├── k8s-controller.yaml └── k8s-worker.yaml https://github.com/coreos/matchbox/tree/master/examples/groups/k8s-install 32
  • 33. Copyright 2017 ITRI 工業技術研究院 Installation Flow install.json install- reboot.json install- reboot.yaml curl "{{.ignition_endpoint}}?{{.request.r aw_query}}&os=installed" -o ignition.json node1.json k8s- controller.json "selector": { "os": "installed", "mac": "00:26:2d:06:ff:bc" }, k8s- controller.yaml "coreos_channel": "stable", "coreos_version": “1298.6.0", 33
  • 34. Copyright 2017 ITRI 工業技術研究院 Setup dnsmasq and matchbox • Dnsmasq docker run --name dnsmasq --cap-add=NET_ADMIN --network="host" -v $PWD/dnsmasq.conf:/etc/dnsmasq.conf:z quay.io/coreos/dnsmasq -d • Matchbox docker run -p 8080:8080 --rm -v $PWD/example:/var/lib/matchbox:Z quay.io/coreos/matchbox:latest -address=0.0.0.0:8080 -log-level=debug Notice: Don’t forget to open firewall port for matchbox(8080), dns, tftp and dhcp 34
  • 35. Copyright 2017 ITRI 工業技術研究院 PXE boot time="2017-04-05T07:31:13Z" level=info msg="Starting matchbox HTTP server on 0.0.0.0:8080" time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /boot.ipxe" time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /ipxe?uuid=03000200-0400-0500-0006-000700080009&mac=00- 26-2d-07-00-78&domain=k8s.itri&hostname=WR1-43&serial=To%20Be%20Filled%20By%20O.E.M." time="2017-04-05T07:34:03Z" level=debug msg="Matched an iPXE config" labels=map[uuid:03000200-0400-0500-0006- 000700080009 mac:00:26:2d:07:00:78 domain:k8s.itri hostname:WR1-43 serial:To Be Filled By O.E.M.] profile=install-reboot time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /assets/coreos/current/coreos_production_pxe.vmlinuz" time="2017-04-05T07:34:04Z" level=info msg="HTTP GET /assets/coreos/current/coreos_production_pxe_image.cpio.gz" time="2017-04-05T07:36:29Z" level=info msg="HTTP GET /ignition?uuid=03000200-0400-0500-0006- 000700080009&mac=00-26-2d-07-00-78&os=installed" time="2017-04-05T07:36:29Z" level=debug msg="Matched an Ignition or Fuze template" group=node3 labels=map[uuid:03000200-0400-0500-0006-000700080009 mac:00:26:2d:07:00:78 os:installed] profile=k8s-controller matchbox logs Demo: https://youtu.be/z9eYOuWLc8k 35
  • 36. Copyright 2017 ITRI 工業技術研究院 Configure kubectl • Use the generated kubeconfig directly KUBECONFIG=examples/assets/tls/kubeconfig • Overwrite kubeconfig cp examples/assets/tls/kubeconfig ~/.kube/config 36
  • 37. Copyright 2017 ITRI 工業技術研究院 Check all PODs and Services [root@centos7 matchbox]# kubectl get po --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system heapster-v1.2.0-4088228293-7vwxd 2/2 Running 0 15h kube-system kube-apiserver-10.201.3.44 1/1 Running 0 15h kube-system kube-controller-manager-10.201.3.44 1/1 Running 0 15h kube-system kube-dns-782804071-j52dv 4/4 Running 0 15h kube-system kube-dns-autoscaler-2715466192-krz0p 1/1 Running 0 15h kube-system kube-proxy-10.201.3.42 1/1 Running 0 15h kube-system kube-proxy-10.201.3.43 1/1 Running 0 15h kube-system kube-proxy-10.201.3.44 1/1 Running 0 15h kube-system kube-scheduler-10.201.3.44 1/1 Running 0 15h kube-system kubernetes-dashboard-3543765157-xj185 1/1 Running 0 15h [root@centos7 matchbox]# kubectl get svc --all-namespaces NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes 10.3.0.1 <none> 443/TCP 15h kube-system heapster 10.3.0.95 <none> 80/TCP 15h kube-system kube-dns 10.3.0.10 <none> 53/UDP,53/TCP 15h kube-system kubernetes-dashboard 10.3.0.66 <none> 80/TCP 15h 37
  • 38. Copyright 2017 ITRI 工業技術研究院 Conclusion • Container Linux (CoreOS) is a good choice for bare metal & production • Manual installation vs. Matchbox+ignition • What’s next? ▪ Try it ▪ Join Kubernetes Taiwan User Group ▪ Kubernetes Training Courses and Playground a. https://www.katacoda.com/courses/kubernetes b. https://www.katacoda.com/courses/kubernetes/playground 38