Building a scalable learning platform - Erik Veld - Codemotion Amsterdam 2018

Building a scalable learning platform
“how to make scaling the cloud’s problem”
instruqt

I like building and learning new things,
and sharing knowledge with others.
ERIK VELD
2instruqt

Founder of instruqt, focussing mostly
on the technical side.
ERIK VELD
3instruqt

Hands-on challenges that build skills on
DevOps and cloud.
WHAT IS INSTRUQT
5instruqt

Each user gets real infrastructure to
complete the challenges in.
WHAT IS INSTRUQT
6instruqt

Solutions are validated against the user
infrastructure.
WHAT IS INSTRUQT
7instruqt

Enough talking, demo time.
8instruqt

HOW IT STARTED
10
2016 2017 2018
HashiConf EU
instruqt

HOW IT STARTED
13
2016 2017 2018
HashiConf EU
HashiConf US
instruqt

HOW IT STARTED
15
2016 2017 2018
HashiConf EU
HashiConf US
Google Cloud Next
instruqt

HOW IT STARTED
17
2016 2017 2018
HashiConf EU
HashiConf US
Google Cloud Next
Current iteration
instruqt

How did we build this platform?
18instruqt

HOW IT IS BUILT
19
Google Cloud Platform Terraform Golang
instruqt

Entirely on Google Kubernetes Engine.
GOOGLE CLOUD PLATFORM
20instruqt

GKE gives us resilience, isolation,
service discovery and scaling for free.
21instruqt

Everything as a service from Google.
22instruqt

23
more
Container Registry
Datastore Kubernetes Engine Cloud SQL Cloud PubSub Cloud Storage
Compute Engine Cloud Functions Firebase
instruqt

24
more
Container Registry
instruqt

25
more
Container Registry
instruqt

Terraform manages our platform and
all of the services we use.
TERRAFORM
26instruqt

TERRAFORM
27
Describe what the desired state is,
don’t worry about how it’s achieved.
instruqt

GOLANG
28
With the small footprint and fast start
time we scale in seconds not minutes.
instruqt

GOLANG
29
With many interdependencies, a
monorepo solves a lot of problems.
instruqt

Let’s take a look behind the scenes.
31instruqt

Terraform creates environments on
the fly, inserting variables where
needed.
BEHIND THE SCENES
32instruqt

BEHIND THE SCENES
34
> PODSSERVICES
CONFIG
SECRETS
PROXY TRACK
NAMESPACE
OTHER INFRA
instruqt

Any container will work, platform
dependencies are injected at runtime.
BEHIND THE SCENES
35instruqt

# The volume is mounted in the pod from a git repository.
volume {
name = "bootstrap"
git_repo {
repository = "https://url-to-git-repo/bootstrap.git"
}
}
volume_mount {
name = "bootstrap"
mount_path = "/opt/bootstrap"
}
# Wait until the volume is mounted, then execute the setup script.
args = ["until [ -d /opt/bootstrap/ ]; do sleep 1; done;
/opt/bootstrap/setup.sh"]
36instruqt

Because terraform stores the state of
each user environment, cleanup is easy.
BEHIND THE SCENES
37instruqt

How does terraform know
what to build?
39instruqt

The SDK creates a skeleton, which is
then filled out and sent to the pipeline.
TRACK PIPELINE
40instruqt

Golang templates generate terraform
code that creates user environments.
TRACK PIPELINE
42instruqt

# The variables that need to go into the template.
data := struct {
Containers Container
}{ … }
# Create the destination file.
f, err := os.Create(outputPath)
# Parse the templates.
t := template.New(templateName)
_, err = t.ParseFiles(inputPath)
# Execute the template with the data.
for _, tmpl := range t.Templates() {
tmpl.ExecuteTemplate(f, tmpl.Name(), data)
}
43instruqt

# Loop over the containers in the config.yml file.
{{range .Containers}}
resource "kubernetes_replication_controller" "{{.Name}}" {
spec {
template {
container {
name = "{{.Name}}"
image = "{{.Image}}"
}
# Etc.
}
}
}
{{- end}}
44instruqt

Because of Container Builder we do
not have to run a build server anymore.
TRACK PIPELINE
46instruqt

The platform and build pipeline are
scalable.
The challenge is scaling ourselves.
LESSONS LEARNED
48instruqt

Stand on the shoulders of giants and
use this to focus on your product.
49
LESSONS LEARNED
instruqt

Migrating to a different cloud provider
is not as much work as you might think.
LESSONS LEARNED
50instruqt

51
Thank you!
Erik Veld
@erikveld
instruqt.com

Building a scalable learning platform - Erik Veld - Codemotion Amsterdam 2018

Recommended

More Related Content

What's hot (20)

Similar to Building a scalable learning platform - Erik Veld - Codemotion Amsterdam 2018 (20)

More from Codemotion (20)

Recently uploaded (20)

Building a scalable learning platform - Erik Veld - Codemotion Amsterdam 2018