Create a Varnish cluster in Kubernetes for Drupal caching - DrupalCon North America 2021

Create a Varnish cluster in Kubernetes
for Drupal caching
Vadym Myrgorod
Slides: http://bit.ly/k8s-varnish-drupal

Vadym Myrgorod
Using Drupal since 2008
Web App Developer
Howard Hughes Medical Institute
@dealancer1
https://www.myrgorod.net

I will make a quick intro into Varnish and Kubernetes.
We will learn how to conﬁgure Varnish cluster for Drupal 8
and run it on Kubernetes platform.
In this session

More users means slower website

Slower website means bigger bounce rate

1. Cutting visitors off
2. Arranging visitors in an online queue
3. Buying more hardware
Approaches we won’t cover in this session

In computing, a cache is a hardware or software
component that stores data so that future requests for
that data can be served faster; the data stored in a cache
might be the result of an earlier computation or a copy of
data stored elsewhere.
Caching

Levels of caching
1
Opcode Caching
Compile PHP script into opcode and cache it.
Examples: OPcache extension.
3
Drupal Caching
Store blocks, views and other Drupal
components in memory.
Examples: Memcache, Redis.
5
Content Delivery Network
Store static assets in geographically
distributed caching system.
Examples: Cloudﬂare, Fastly, Akamai.
2
Database Caching
Store results of SQL queries in memory.
Examples: memcached.
4
Static Content Caching
Store static assets in memory.
Examples: Varnish, Nginx.

● Aggregated CSS ﬁles
● Aggregated JS ﬁles
● Images of various image styles
● HTML pages (for anonymous users)
Static content generated by Drupal

● Caching HTTP reverse proxy
● Speeds up delivery with a factor of 300 - 1000x
● Varnish Conﬁguration Language (VCL)
● Can be scaled into Varnish Cluster

Backend - server which is providing
the content Varnish will accelerate.
Frontend - client facing Varnish
server.
Terms
Backend
Frontend

vcl 4.0;
backend default {
.host = "127.0.0.1";
.port = "8080";
}
sub vcl_recv {
if (req.http.host == "www.example.com") {
set req.backend_hint = default;
}
}
See following VCL example for Drupal 8 and 7:
https://www.varnish-software.com/wiki/content/tutorials/drupal/drupal_vcl.html
Minimal Varnish VCL
/etc/varnish/default.vcl

now let’s talk about
Kubernetes

1. You want to manage services you are running e.g. MySQL,
Varnish, Redis, Apache Solr, Elasticsearch...
2. You want to have an ability to migrate from one cloud
provider to another quickly.
3. You have on-premises infrastructure you need to utilize.
4. You want to use multicloud or hybrid-cloud approaches.
5. You want to have some of the cool features that
Kubernetes provides.
When using Kubernetes is a good idea

1. Scaling: runs more containers if needed.
2. Healing: restarts containers when they are down.
3. Deployment strategies: rolling updates, blue/green, canary...
4. Service discovery and load balancing: distributes access to
containers by service name.
5. Security: secret management, security policies.
6. Storage orchestration: cloud, NFS, or storage providers.
7. Flexibility and extensibility: Go is widely used in k8s world.
Things Kubernetes is good at

● Resource - endpoint in k8s API that stores a
collection of certain kind. Declared using YAML
syntax. Example: config map, secret, volume, volume
claim, pod, deployment, stateful set, service, ingress,
etc...
● Config map - stores software configuration that can
be mounted as files or passed as env variables.
● Volume - used to preserve file system after container
is restarted.
● Pod - k8s resource that represents running
container (or set set of containers).
Kubernetes resources
Config map
Pod
Deployment
Service
Ingress
Stateful Set
Volume

● Deployment - ensures certain amount of pods are
up and running.
● Stateful Set - acts as deployment for stateful
services that have speciﬁc requirements to storage
and pod identity.
● Service - provides a way to access deployments or
stateful sets using different behaviours. Creates a
common DNS record that can be used to access
pods and does load balancing.
● Ingress - manages external access to services using
various routing rules.
Kubernetes resources
Conﬁg map
Pod
Deployment
Service
Ingress
Stateful Set
Volume

nginx-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
Minimal Kubernetes deployment and service
nginx-service.yaml:
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
type: NodePort
ports:
- protocol: TCP
port: 8080
targetPort: 80
nodePort: 30080

1. Apply deployment
$ kubectl apply -f nginx-deployment.yaml
2. Check deployments
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 0/2 0 0 1s
3. Check pods
$ kubectl get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-deployment-75675f5897-7ci7o 1/1 Running 0 18s app=nginx,pod-template-hash=3123191453
nginx-deployment-75675f5897-kzszj 1/1 Running 0 18s app=nginx,pod-template-hash=3123191453

4. Apply service
$ kubectl apply -f nginx-service.yaml
5. Check services
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 41d
nginx-service NodePort 10.98.115.133 <none> 8080:30080/TCP 5m11s
6. Access nginx using node port
$ curl localhost:30080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

To run Varnish in k8s we need to declare app deployment, app
service, Varnish deployment, Varnish service, conﬁg maps and
ingress.
Example: https://github.com/dealancer/k8s-varnish-test.
The simplest approach

varnish-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: varnish-deployment
labels:
app: varnish
spec:
replicas: 1
selector:
matchLabels:
app: varnish
template:
metadata:
labels:
app: varnish
spec:
containers:
- name: varnish
image: varnish:6.6.0
env:
- name: CACHE_SIZE
value: 128m
- name: VCL_CONFIG
value: /etc/varnish/configmap/default.vcl
volumeMounts:
- name: varnish-config
mountPath: /etc/varnish/configmap
ports:
- containerPort: 80
volumes:
- name: varnish-config
configMap:
name: varnish-configmap

varnish-configmap.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: varnish-configmap
labels:
app: varnish
data:
default.vcl: |
vcl 4.0;
backend default {
.host = "nginx-service";
.port = "8080";
}
sub vcl_recv {
set req.backend_hint = default;
}
varnish-service.yaml:
apiVersion: v1
kind: Service
metadata:
name: varnish-service
spec:
selector:
app: nginx
type: NodePort
ports:
- protocol: TCP
port: 80
targetPort: 80
nodePort: 30081

Questions to this approach:
1. Can we eliminate App Service and let Varnish talk to
application pods (backends) directly?
2. How do we scale Varnish pods (frontends)?
3. How do we shard cache across multiple Varnish pods?
4. How do we invalidate cache in multiple Varnish pods?

kube-httpcache is an open Source Kubernetes controller written in Go to run Varnish
cluster. It is a free solution comparing to Varnish Cache Plus, but it requires to configure
Varnish in certain way using VCL.
GitHub: https://github.com/mittwald/kube-httpcache.
Features:
● Monitors backend (app) and frontend (Varnish) pods.
● Dynamically update Varnish VCL and reload it on the the fly in all Varnish pods.
● Supports Go-template syntax in VCL file.
● Sends cache invalidation requests to all Varnish pods using Signaller component.
kube-httpcache

spec:
containers:
- name: cache
image: quay.io/mittwald/kube-httpcache:stable
imagePullPolicy: Always
args:
- -admin-addr=0.0.0.0
- -admin-port=6083
- -signaller-enable
- -signaller-port=8090
- -frontend-watch
- -frontend-namespace=$(NAMESPACE)
- -frontend-service=frontend-service
- -backend-watch
- -backend-namespace=$(NAMESPACE)
- -backend-service=backend-service
- -varnish-secret-file=/etc/varnish/k8s-secret/secret
- -varnish-vcl-template=/etc/varnish/tmpl/default.vcl.tmpl
- -varnish-storage=malloc,128M
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: template
mountPath: /etc/varnish/tmpl
- name: secret
mountPath: /etc/varnish/k8s-secret
varnish-service.yaml (part)

{{ range .Backends }}
backend be-{{ .Name }} {
.host = "{{ .Host }}";
.port = "{{ .Port }}";
}
{{- end }}
{{ range .Frontends }}
backend {{ .Name }} {
.host = "{{ .Host }}";
.port = "{{ .Port }}";
}
{{- end }}
sub vcl_init {
new lb = directors.round_robin();
{{ range .Backends -}}
lb.add_backend(be-{{ .Name }});
{{ end }}
}
sub vcl_recv {
set req.backend_hint = lb.backend();
}
Eliminate the need in backend service

In this approach cache is sharded across multiple Varnish frontends.
1. Varnish Service routes requests to a random Varnish frontend.
2. A corresponding Varnish frontend (x-shard) determined based on the
URL using hash director:
a. http://example.com/foo -> frontend 1
b. http://example.com/bar -> frontend 2
c. http://example.com/bar2 -> frontend 1
3. If x-shard is not the current Varnish frontend, request is forwarded to
x-shard.
4. Otherwise, request is handled by the current Varnish node:
a. If request is cacheable and cache exist, cached content is returned.
b. If cache does not exists, Varnish requests a backend to save cache
and serve cached content.
c. If request is cacheable request is routed to a backend.
See https://info.varnish-software.com/blog/creating-self-routing-varnish-cluster
Running Varnish cluster (sharded cache)

sub vcl_init {
# ...
new cluster = directors.hash();
{{ range .Frontends -}}
cluster.add_backend({{ .Name }}, 1);
{{ end }}
}
sub vcl_recv {
# ...
unset req.http.x-cache;
set req.backend_hint = cluster.backend(req.url);
set req.http.x-shard = req.backend_hint;
if (req.http.x-shard != server.identity) {
return(pass);
}
# ...
return(hash);
}
Running Varnish cluster (sharded cache)

Problem
In case if you horizontally scale your Varnish cluster, a
hash director will recalculate hashes for existing URLs
and will route requests to different Varnish frontends
which do not store cached content for these URLs.
Thus users will experience slowdown.
Solution
Consistent caching mechanism is provided by Varnish
through shard director. Consistent caching represents
URLs and frontends as points on the ring. Adding new
frontend, simply adds a new point on the ring.
Scaling Varnish cluster

sub vcl_init {
# ...
new cluster = directors.shard();
{{ range .Frontends -}}
cluster.add_backend({{ .Name }});
{{ end }}
cluster.set_warmup(180);
}
sub vcl_recv {
# ...
unset req.http.x-cache;
set req.backend_hint = cluster.backend(by=URL);
set req.http.x-shard = req.backend_hint;
if (req.http.x-shard != server.identity) {
return(pass);
}
# ...
return(hash);
}
Scaling Varnish cluster

Varnish Signaller:
● Is aware of all Varnish frontends running
● acts similarly to Varnish Broadcaster (component of Varnish Plus)
● broadcasts ﬂush cache request to all Varnish frontends
● runs on port 8090
BAN requests
$ curl -H "X-Url: /path" -X BAN http://cache-service:8090
$ curl -H "Cache-Tags: node-1" -X BAN http://cache-service:8090
PURGE requests
$ curl -H "X-Host: www.example.com" -X PURGE http://cache-service:8090/path
Invalidating cache

sub vcl_recv {
# ...
if (req.method == "PURGE") {
if (client.ip !~ privileged) {
return (synth(403, "Not allowed."));
}
if (req.http.X-Host) {
set req.http.host = req.http.X-Host;
}
return (purge);
}
if (req.method == "BAN") {
if (client.ip !~ privileged) {
return (synth(403, "Not allowed."));
}
if (req.http.Cache-Tags) {
ban("obj.http.Cache-Tags ~ " + req.http.Cache-Tags);
return (synth(200, "Ban added " + req.http.host));
}
if (req.http.X-Url) {
ban("obj.http.X-Url == " + req.http.X-Url);
return (synth(200, "Ban added " + req.http.host));
}
return (synth(403, "Cache-Tags or X-Url header missing."));
}
# ...
}
Invalidating cache

1. Kubernetes Playground
https://www.katacoda.com/courses/kubernetes/playground
2. Kubernetes Patterns Book
https://www.redhat.com/cms/managed-ﬁles/cm-oreilly-kubernetes-patterns-ebook-f19824-2
01910-en.pdf
3. Article this presentation is based on
https://dealancer.medium.com/creating-a-scalable-and-resilient-varnish-cluster-using-kuber
netes-853f03ec9731
4. kube-httpcache project
https://github.com/mittwald/kube-httpcache
5. Complete VCL ﬁle for Drupal 8 and 9 (IMPORTANT)
https://gist.github.com/dealancer/968297d6ddd93df80d012af7f4093294
Resources

1. Purge module - https://www.drupal.org/project/purge
Clreans external caching systems, reverse proxies and CDNs
2. Varnish purger - https://www.drupal.org/project/varnish_purge
Extension of Purge module to clear Varnish cache
3. Conﬁguration URL: /admin/conﬁg/development/performance/purge
Configuring Drupal

Q&A?
Please, take a survey!

Thanks!
Please, take a survey!

Create a Varnish cluster in Kubernetes for Drupal caching - DrupalCon North America 2021

Recommended

More Related Content

What's hot (20)

Similar to Create a Varnish cluster in Kubernetes for Drupal caching - DrupalCon North America 2021 (20)

More from Ovadiah Myrgorod (8)

Recently uploaded (20)

Create a Varnish cluster in Kubernetes for Drupal caching - DrupalCon North America 2021