Kubernetes Operators And The Redis Enterprise Journey: Michal Rabinowitch
0 likes471 views
This document provides an overview of Redis Enterprise and its journey towards being deployed and managed as a Kubernetes operator. Some key points: - Redis Enterprise allows for multi-tenancy by supporting over 200 applications on a single cloud instance through its shared-nothing cluster architecture. - Running stateful applications on Kubernetes presents challenges around ordering operations, persistent storage, and supporting multiple Kubernetes platforms/distributions. - The operator approach was taken to gain full control over the application lifecycle through Go code running as a Kubernetes deployment, including upgrades, validation of configuration changes, and tying the deployment to custom resources. - This provides a simpler way to package, deploy and manage Redis Enterprise on Kubernetes compared to static YAML files or
![PRESENTED BY
22
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
labelSelector:
matchExpressions:
key: app
operator: In
values:
{{ template "redisenterprise.name" . }}
key: release
operator: In
values:
{{ .Release.Name }}
key: redis.io/role
operator: In
values:
node
topologyKey: kubernetes.io/hostname
terminationGracePeriodSeconds: 31536000
serviceAccountName: {{ template "redisenterprise.serviceAccountName" . }}
{{ with .Values.imagePullSecrets }}
imagePullSecrets:
{{ toYaml . | indent 8 }}
{{ end }}
containers:
name: redis
image: {{ .Values.redisImage.repository }}:{{ .Values.redisImage.tag }}
imagePullPolicy: {{ .Values.redisImage.pullPolicy }}
readinessProbe:
exec:
command:
# check that the node is bootstrapped and that its connected and synced.
bash
c
curl silent localhost:8080/v1/bootstrap && /opt/redislabs/bin/rladmin status
nodes | grep node:$(cat /etc/opt/redislabs/node.id) | grep OK
initialDelaySeconds: 20
timeoutSeconds: 5
lifecycle:
preStop:
exec:
command:
# enslave the node, if this current node is master, change the master to
the first slave node.
bash
c
/opt/redislabs/bin/rladmin node $(cat /etc/opt/redislabs/node.id) enslave
&& ((/opt/redislabs/bin/rladmin status nodes | grep node:$(cat
/etc/opt/redislabs/node.id) | grep q master) && /opt/redislabs/bin/rlutil
change_master master=$(/opt/redislabs/bin/rladmin status nodes | grep slave |
head 1 | cut d " " f 1| cut d ":" f2) && sleep 10) || /bin/true
resources:
{{ toYaml .Values.redisResources | indent 10 }}
ports:
containerPort: 8001
containerPort: 8443
containerPort: 9443
securityContext:
capabilities:
add:
SYS_RESOURCE
{{ if .Values.persistentVolume.enabled }}
volumeMounts:
mountPath: "/opt/persistent"
name: redisstorage
{{ end }}
env:
name: K8S_ORCHASTRATED_DEPLOYMENT
value: "yes"
name: JOIN_HOSTNAME
value: {{ template "redisenterprise.fullname" . }}
{{ if .Values.persistentVolume.enabled }}
name: PERSISTANCE_PATH
value: /opt/persistent
{{ end }}
name: K8S_SERVICE_NAME
value: {{ template "redisenterprise.fullname" . }}
name: BOOTSTRAP_HANDLE_REDIRECTS
value: "enabled"
name: BOOTSTRAP_CLUSTER_FQDN
value: {{ template "redisenterprise.clusterDNS" . }}
name: BOOTSTRAP_DMC_THREADS
value: "10"
name: BOOTSTRAP_USERNAME
valueFrom:
secretKeyRef:
name: {{ template "redisenterprise.fullname" . }}
key: username
name: BOOTSTRAP_PASSWORD
valueFrom:
secretKeyRef:
name: {{ template "redisenterprise.fullname" . }}
key: password
name: BOOTSTRAP_LICENSE
valueFrom:
secretKeyRef:
name: {{ template "redisenterprise.fullname" . }}
key: license
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: {{ template "redisenterprise.statefulsetName" . }}
labels:
app: {{ template "redisenterprise.name" . }}
chart: {{ template "redisenterprise.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
{{ if .Values.persistentVolume.enabled }}
volumeClaimTemplates:
metadata:
name: redisstorage
labels:
app: {{ template "redisenterprise.name" . }}
chart: {{ template "redisenterprise.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: {{ .Values.persistentVolume.size | quote }}
{{ if .Values.persistentVolume.storageClass }}
{{ if (eq "" .Values.persistentVolume.storageClass) }}
storageClassName: ""
{{ else }}
storageClassName: "{{ .Values.persistentVolume.storageClass }}"
{{ end }}
{{ end }}
{{ end }}
serviceName: {{ template "redisenterprise.fullname" . }}
replicas: {{ .Values.replicas }}
updateStrategy:
type: "RollingUpdate"
template:
metadata:
labels:
redis.io/role: node
app: {{ template "redisenterprise.name" . }}
chart: {{ template "redisenterprise.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
{{ with .Values.nodeSelector }}
nodeSelector:
{{ toYaml . | indent 8 }}
{{ end }}](https://image.slidesharecdn.com/01michalrabinowitch-190508035527/85/Kubernetes-Operators-And-The-Redis-Enterprise-Journey-Michal-Rabinowitch-22-320.jpg)
![[OpenInfra Days Korea 2018] (Track 2) Neutron LBaaS 어디까지 왔니? - Octavia 소개](https://cdn.slidesharecdn.com/ss_thumbnails/26octavia-180704054917-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Kubernetes Operators And The Redis Enterprise Journey: Michal Rabinowitch (20)
Ad
More from Redis Labs (20)
Ad
Recently uploaded (20)
Kubernetes Operators And The Redis Enterprise Journey: Michal Rabinowitch
- 1. PRESENTED BY Kubernetes Operators and The Redis Enterprise Journey Michal Rabinowitch Redis Labs, Software Engineer
- 2. PRESENTED BY
- 3. PRESENTED BY Redis Labs Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source home and commercial provider of Redis Enterprise (Redise) technology, platform, products & services.
- 4. PRESENTED BY Redis Enterprise Technology Redis Enterprise Node Redis Enterprise Cluster • Shared nothing cluster architecture • Fully compatible with open source commands & data structures
- 5. PRESENTED BY Redis Enterprise : Multi-Tenancy Maximizes Resource Utilization 200+ applications or shards on a single 4vcore cloud instance Application A Application B Application N
- 6. PRESENTED BY Multiple Delivery Models Cloud Essentials Cloud Pro Downloadable Native Kubernetes Essentials PCF over BOSH or PKS Essentials OpenShift over Kubernetes DATABASE-AS-A-SERVICE SOFTWARE Fully-managed, server-less and hosted Fully-managed, hosted or in customer’s VPC
- 7. PRESENTED BY Redis Enterprise Software and K8s
- 8. PRESENTED BY Who Does What • Node auto-healing • Node scaling • Failover & scaling • Configuration & monitoring • Service discovery • Upgrade +
- 9. PRESENTED BY Challenge 1: Running a Stateful App • Ordered Operations • Persistent Storage
- 10. PRESENTED BY Challenge 1: Running a Stateful App • Ordered Operations • Persistent Storage Stateful Set PV-0 PV-1 PV-2 pod-1pod-0 pod-2pod-1
- 11. PRESENTED BY • Supporting multiple K8s platforms and distributions • storage • security • ingress • Packaging Challenge 2: Vendors Operator
- 12. PRESENTED BY Building Our Deployment Puzzle Persistence: Stateful Set with PVs PV-0 PV-1 PV-2 pod-0 pod-1 pod-2 StatefulSet
- 13. PRESENTED BY Building Our Deployment Puzzle Persistence: Stateful Set with PVs DB Connectivity: Designated Deployment Deployment Services Rigger db-1.ns.svc db-2.ns.svc PV-0 PV-1 PV-2 pod-0 pod-1 pod-2 StatefulSet
- 14. PRESENTED BY Building Our Deployment Puzzle Persistence: Stateful Set with PVs DB Connectivity: Designated Deployment Deployment Services Rigger db-1.ns.svc db-2.ns.svc UI service API service UI, API: Services PV-0 PV-1 PV-2 pod-0 pod-1 pod-2 StatefulSet
- 15. PRESENTED BY Building Our Deployment Puzzle Persistence: Stateful Set with PVs DB Connectivity: Designated Deployment High Availability: Anti Affinity Deployment Services Rigger db-1.ns.svc db-2.ns.svc UI service API service UI, API: Services K8s Node1 K8s Node2 K8s Node3 PV-0 PV-1 PV-2 pod-0 pod-1 pod-2 StatefulSet more resources ...
- 16. PRESENTED BY Our Journey Towards an Operator Started out with 9 static yaml files – Hard to maintain, no parameterization – Hard to distribute - poor UX – No control over the deployment life-cycle .yaml
- 17. PRESENTED BY Our Journey Towards an Operator Helm - A package manager – Customized deployment - templating – Requires installation inside the cluster – No control over the deployment life-cycle
- 18. PRESENTED BY Our Journey Towards an Operator Operator – K8s native - no plugin needed – Simple deployment – Simple packaging – Operator SDK to minimize boilerplate – Go code that runs as a K8s deployment
- 19. PRESENTED BY Our Journey Towards an Operator Operator – Add domain specific knowledge and full control over life-cycle • Upgrade • Validate configuration changes e.g. downgrade • Reconfiguration of deployments • The spec is saved to k8s and not some config file
- 20. PRESENTED BY Enter Operator Operator manage Custom Resource Redis Enterprise Cluster
- 21. PRESENTED BY Enter Operator Deployment Services Rigger db-1.ns.svc db-2.ns.svc UI service API service K8s Node1 K8s Node2 K8s Node3 PV-0 PV-1 PV-2 pod-0 pod-1 pod-2 StatefulSet more resources ... Operator Custom Resource: Redis Enterprise Cluster manage
- 22. PRESENTED BY 22 affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: labelSelector: matchExpressions: key: app operator: In values: {{ template "redisenterprise.name" . }} key: release operator: In values: {{ .Release.Name }} key: redis.io/role operator: In values: node topologyKey: kubernetes.io/hostname terminationGracePeriodSeconds: 31536000 serviceAccountName: {{ template "redisenterprise.serviceAccountName" . }} {{ with .Values.imagePullSecrets }} imagePullSecrets: {{ toYaml . | indent 8 }} {{ end }} containers: name: redis image: {{ .Values.redisImage.repository }}:{{ .Values.redisImage.tag }} imagePullPolicy: {{ .Values.redisImage.pullPolicy }} readinessProbe: exec: command: # check that the node is bootstrapped and that its connected and synced. bash c curl silent localhost:8080/v1/bootstrap && /opt/redislabs/bin/rladmin status nodes | grep node:$(cat /etc/opt/redislabs/node.id) | grep OK initialDelaySeconds: 20 timeoutSeconds: 5 lifecycle: preStop: exec: command: # enslave the node, if this current node is master, change the master to the first slave node. bash c /opt/redislabs/bin/rladmin node $(cat /etc/opt/redislabs/node.id) enslave && ((/opt/redislabs/bin/rladmin status nodes | grep node:$(cat /etc/opt/redislabs/node.id) | grep q master) && /opt/redislabs/bin/rlutil change_master master=$(/opt/redislabs/bin/rladmin status nodes | grep slave | head 1 | cut d " " f 1| cut d ":" f2) && sleep 10) || /bin/true resources: {{ toYaml .Values.redisResources | indent 10 }} ports: containerPort: 8001 containerPort: 8443 containerPort: 9443 securityContext: capabilities: add: SYS_RESOURCE {{ if .Values.persistentVolume.enabled }} volumeMounts: mountPath: "/opt/persistent" name: redisstorage {{ end }} env: name: K8S_ORCHASTRATED_DEPLOYMENT value: "yes" name: JOIN_HOSTNAME value: {{ template "redisenterprise.fullname" . }} {{ if .Values.persistentVolume.enabled }} name: PERSISTANCE_PATH value: /opt/persistent {{ end }} name: K8S_SERVICE_NAME value: {{ template "redisenterprise.fullname" . }} name: BOOTSTRAP_HANDLE_REDIRECTS value: "enabled" name: BOOTSTRAP_CLUSTER_FQDN value: {{ template "redisenterprise.clusterDNS" . }} name: BOOTSTRAP_DMC_THREADS value: "10" name: BOOTSTRAP_USERNAME valueFrom: secretKeyRef: name: {{ template "redisenterprise.fullname" . }} key: username name: BOOTSTRAP_PASSWORD valueFrom: secretKeyRef: name: {{ template "redisenterprise.fullname" . }} key: password name: BOOTSTRAP_LICENSE valueFrom: secretKeyRef: name: {{ template "redisenterprise.fullname" . }} key: license apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: {{ template "redisenterprise.statefulsetName" . }} labels: app: {{ template "redisenterprise.name" . }} chart: {{ template "redisenterprise.chart" . }} release: {{ .Release.Name }} heritage: {{ .Release.Service }} spec: {{ if .Values.persistentVolume.enabled }} volumeClaimTemplates: metadata: name: redisstorage labels: app: {{ template "redisenterprise.name" . }} chart: {{ template "redisenterprise.chart" . }} release: {{ .Release.Name }} heritage: {{ .Release.Service }} spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: {{ .Values.persistentVolume.size | quote }} {{ if .Values.persistentVolume.storageClass }} {{ if (eq "" .Values.persistentVolume.storageClass) }} storageClassName: "" {{ else }} storageClassName: "{{ .Values.persistentVolume.storageClass }}" {{ end }} {{ end }} {{ end }} serviceName: {{ template "redisenterprise.fullname" . }} replicas: {{ .Values.replicas }} updateStrategy: type: "RollingUpdate" template: metadata: labels: redis.io/role: node app: {{ template "redisenterprise.name" . }} chart: {{ template "redisenterprise.chart" . }} release: {{ .Release.Name }} heritage: {{ .Release.Service }} spec: {{ with .Values.nodeSelector }} nodeSelector: {{ toYaml . | indent 8 }} {{ end }}
- 24. PRESENTED BY 24 Example: Upgrade Flow Update Spec with new version1 Running
- 25. PRESENTED BY 25 Example: Upgrade Flow Update Spec with new version Operator Validates - An upgrade is not taking place at the moment - This is not a downgrade - Redis Enterprise Cluster is ready 1 2 Running Invalid Upgrade
- 26. PRESENTED BY 26 Example: Upgrade Flow User edits RedisEnterpriseCluster Spec with new image Operator Validates 1 2 3 Upgrade - Edit Stateful Set spec - Switch Operator state to “Upgrade” Running Upgrade
- 28. PRESENTED BY Geo distributed Active Active database running on multiple clouds environments deployed using an Operator
- 29. PRESENTED BY Active-Active Geo Distribution Database (CRDT-Based) App App App
- 30. PRESENTED BY Multi Cloud, Different Deployment Methods Google Kubernetes Engine US EASTUS WEST EU WEST Google Kubernetes Engine Demo
- 31. PRESENTED BY Getting started with Redis Enterprise Operator https://docs.redislabs.com/latest/rs/getting-started/k8s-openshift/
- 32. Thank you!