Continuous Integration & Continuous Delivery with GCP

Continuous
Integration &
Continuous
Delivery with
GCP

https://www.facebook.com/groups/GCPUG.TW/
https://plus.google.com/u/0/communities/116100913832589966421
Google Cloud Platform User Group Taiwan
我們是Google Cloud Platform Taiwan User Group。在Google雲端服務在台灣地區展露頭角之後，
有許多新的服務、新的知識、新的創意，歡迎大家一起分享，一起了解 Google雲端服務...
GCPUG透過網際網路串聯喜好 Google Cloud的使用者，分享與交流使用 GCP的點滴鑑驗。如果您
是Google Cloud Platform的初學者，您應該來聽聽前輩們的使用經驗；如果您是 Google Cloud
Platform的Expert，您應該來分享一下寶貴的經驗，並與更多高手互相交流；如果您還沒開始用
Google Cloud Platform，那麼您應該馬上來聽聽我們是怎麼使用 Google Cloud的！

Hello!
I am Cage Chung
I am here because I like
to share my
experiences.
You can find me at:
http://kaichu.io

Outline
◎ App Engine Development
◎ Devops: CI & CD (Docker, gitlab, GCP)
◎ Tips & Study information

1.
App Engine
Development
Let’s start with the first set of slides

Who is using GAE?
Angry Birds
Khan Academy
Royal Wedding
Snapchat
EA Games
Forbes
Bestbuy
https://cloud.google.com/customers/ Appengine

Google Cloud Platform | Compute
PaasIaaS
Application
Data
Runtime
Middleware
OS
Virtualization
Servers
Storage
Networking
Application
Data
Runtime
Middleware
OS
Virtualization
Servers
Storage
Networking
Saas
Application
Data
Runtime
Middleware
OS
Virtualization
Servers
Storage
Networking
Infrastructure as a Service Platform as a Service Service as a Service
You manage Google manage

GAE Architecture
image: https://cloud.google.com/solutions/architecture/webapp

App Engine APIs
◎ Storage
◎ Cloud Storage
◎ Cloud SQL
◎ Cloud Datastore
◎ Memcache
◎ Task Queue
◎ Cloud Endpoints
Find more https://cloud.google.com/appengine/

App Engine runtime
Python2.7 Java7 Php Go
RubyNode.js
standard
Flexible
Beta
Python2.7/3.4 Java8 Go custom
[Google App Engine Documentation - App Engine — Google Cloud Platform](https://cloud.google.com/appengine/docs)

App Engine | waldo
queue
MetadataWaldo Server
App Engine
Task Queues
Cloud Storage
Optimal-
waypoints
managed VMs
(python27)
...
endpoints API
(OAuth 2)
Object Notification
Cloude
Datastore
snapshot
managed VMs
(nodejs)
Android App
CSV
uploadwaypoints
Store/retrieve
metadata
waypoint map
snapshot
optim
al
road
trip
RESTful APIs for waypoints list, upload
waypoints
standard
Flexible
Beta
[Waldo-gcp](http://www.slideshare.net/cagechung/waldogcp)

2.
Devops: CI & CD
(Docker, gitlab,
GCP)
Let’s start with the second set of
slides

Devops
DevOps (a clipped compound of
development and operations) is
a culture, movement or practice
that emphasizes the
collaboration and
communication of both software
developers and other
information-technology (IT)
professionals while automating
the process of software delivery
and infrastructure changes.[1][2]
It aims at establishing a culture
and environment where
building, testing, and releasing
software, can happen rapidly,
frequently, and more reliably -
DevOps - Wikipedia
[image](https://goo.gl/pMC6UZ)

Devops: CI & CD (Docker, gitlab, GCP)
[image](https://goo.gl/pMC6UZ)

Origin imgae from and modified [DevOps：持續整合＆持續交付（Docker、CircleCI、AWS）](http://blog.amowu.com/2015/04/devops-
continuous-integration-delivery-docker-circleci-aws-beanstalk.html)

Simple flow
◎ Python (gae-flask)
○ Local Development
○ Local Test
◎ Gitlab
◎ Code Review with GitHub(Gitlab) Flow
◎ gitlab-ci-multi-runner
○ Local runner exec test docker
○ Gitlab runner test docker
◎ Google App Engine
○ Google Cloud Source Repositories
○ Stackdriver Debugger

Local Development / Test | gae-flask
// clone repo
$ git clone git@gitlab.com:cage1016/gae-flask.git
// make a new virtualenv
$ mkvirtualenv gae-flask
// pip install requirements.test.txt or link env as lib folder
(gae-flask-gitlab-ci) $ pip install -r requirements.testing.txt -t lib
// or
(gae-flask-gitlab-ci) $ pip install git+https://github.com/ze-phyr-us/linkenv.git
(gae-flask-gitlab-ci) $ linkenv env/lib/python2.7/site-packages lib
// local run app engine project
(gae-flask-gitlab-ci) $ dev_appserver.py app.yaml
// local run test
(gae-flask-gitlab-ci) $ python ./runtests.py

Local Development / mian.py
import flask
import requests
app = flask.Flask(__name__)
@app.route("/")
def hello():
return "hello world"
if __name__ == "__main__":
app.run()

Local Development / test_main.py
import pytest
import main
@pytest.fixture
def app():
return main.app
@pytest.fixture
def test_client(app):
return app.test_client()
def test_hello(test_client):
response = test_client.get("/")
assert response.data.decode("utf-8") == "hello world"

Gitlab Project Runner
https://gitlab.com/cage1016/gae-flask/runners

gitlab-ci.yml
image: cage1016/gitlab-ci-gcp:v1.5
before_script:
- export CLOUDSDK_CORE_DISABLE_PROMPTS=1
- export CLOUDSDK_PYTHON_SITEPACKAGES=1
- export GCP_PROJECT=gae-lab-001
- export GCP_PROJECT_VERSION=uat
types:
- test
- deploy
test:
stage: test
script:
- sh ./scripts/tests.sh
deploy:
stage: deploy
script:
- sh ./scripts/deploy.sh
only:
master
[GitLab Documentation](http://doc.gitlab.com/ee/ci/yaml/README.html#only-and-except)

gitlab-ci.yml (tests.sh)
#!/usr/bin/env bash
virtualenv env
source env/bin/activate
# install test env packages from requirements.testing.txt
pip install -r requirements.testing.txt -t lib
# link env as lib folder if you just execute `pip install -r requirements.testing.txt`
# pip install git+https://github.com/ze-phyr-us/linkenv.git
# linkenv env/lib/python2.7/site-packages lib
# run tests
python runtests.py

gitlab-ci.yml (deploy.sh)
#!/usr/bin/env bash
pip install -r requirements.txt -t lib/
echo $GCLOUD_KEY > key.json
gcloud auth activate-service-account $GCLOUD_ACCOUNT --key-file key.json
gcloud --quiet config set project $GCP_PROJECT
gcloud --quiet preview app deploy app.yaml --no-promote --version $GCP_PROJECT_VERSION

Local runner exec test docker
// [install gitlab-ci-multi-runner](https://goo.gl/a99Aa4)
// list and register gitlab-ci-runner
$ gitlab-ci-multi-runner list
Listing configured runners ConfigFile=/Users/cage/.gitlab-runner/config.toml
gae-flask-gitlab-ci Executor=docker Token=95d56b26104856b9f3852f2d82b1e5 URL=https://gitlab.com/ci
// prepare a docker-machine for gitlab-ci-multi-runner
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
runner - virtualbox Running tcp://192.168.99.100:2376 v1.11.0
// run a gitlab-ci-multi-runner test via docker
$ gitlab-ci-multi-runner --debug exec docker test --env "GCLOUD_KEY=$GCLOUD_KEY
GCLOUD_ACCOUNT=$GCLOUD_ACCOUNT GCP_PROJECT_VERSION=$GCP_PROJECT_VERSION"
// push git
(gae-flask-gitlab-ci) $ git push master

Docker cage1016/gitlab-ci-gcp:v1.5
FROM buildpack-deps:jessie
# remove several traces of debian python
RUN apt-get purge -y python.*
RUN apt-get update && apt-get install -y -qq --no-install-recommends unzip openssh-client python-openssl
# install "virtualenv", since the vast majority of users of this image will want it
RUN pip install --no-cache-dir virtualenv
# Install the Google Cloud SDK.
# archive gcloud sdk https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-105.0.0-
linux-x86_64.tar.gz
ENV HOME /
ENV CLOUDSDK_PYTHON_SITEPACKAGES 1
RUN wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-102.0.0-linux-x86_64.
tar.gz && tar -zxvf google-cloud-sdk-102.0.0-linux-x86_64.tar.gz && rm google-cloud-sdk-102.0.0-linux-
x86_64.tar.gz
RUN google-cloud-sdk/install.sh --usage-reporting=true --path-update=true --bash-completion=true --rc-
path=/.bashrc --additional-components app-engine-java app-engine-python app kubectl alpha beta
...
https://github.com/cage1016/gitlab-ci-gcp/blob/master/Dockerfile

Gitlab Build Status
https://gitlab.com/cage1016/gae-flask/builds

Gitlab Build detail
https://gitlab.com/cage1016/gae-flask/builds/1321684

Push a new branch to Gitlab
https://gitlab.com/cage1016/gae-flask/builds

Google Cloud Source Repositories
[/ - GAE Labs](https://goo.gl/c99Dz8)

Google Cloud Source Repositories repo
$ git remote add google https://source.developers.google.com/p/[PROJECT_ID]/r/default
// generates a file named source-context.json, which contains information about the version of the
source code used to build the application
$ gcloud preview app gen-repo-info-file
// push code to google cloud source repositories
$ git push google master
[/ - GAE Labs](https://goo.gl/c99Dz8)

Stackdriver Debugger | Source Code
[Debug - GAE Labs](https://goo.gl/I3K6J2)

Stackdriver Debugger | Debug
[Debug - GAE Labs](https://goo.gl/I3K6J2)

3.
Tips & Study
information
Let’s start with the third set of
slides

“
5G Google cloud storage for App
engine for FREE
<project-id>.appspot.com
staging.<project-id>.appspot.com

“
App Engine Managed VMs is
renamed to App Engine flexible
environment. March 24, 2016
[Python Release Notes - Python — Google Cloud Platform](https://cloud.google.com/appengine/docs/python/release-notes#1.9.35)

“
App Engine Admin API
Cloud Source Repositories API
Google Cloud Storage

“
At the moment, the Go app engine SDK is still based
on Go 1.4, even if you use the go1.6beta version for
your app. This means that the SDK doesn't
understand 1.6-style vendoring, and so doesn't
correctly upload the vendored dependencies.
This will be fixed when 1.6 becomes the stable (go1)
version and is used as the basis of the SDK. Stay
tuned!
[gcpug.tw.go/.travis.yml at master · gcpug-tw/gcpug.tw.go](https://github.com/gcpug-tw/gcpug.tw.go/blob/master/.travis.yml)
[Vendorization for deployment - Google Groups](https://groups.google.com/forum/#!topic/google-appengine-go/umcPV_4tAm8)

Study information
◎ Gitlab
○ [Quick Start](http://doc.gitlab.com/ce/ci/quick_start/README.
html)
○ [Configuration of your builds with .gitlab-ci.yml](http://doc.gitlab.
com/ee/ci/yaml/README.html)
○ [GitLab.org / gitlab-ci-multi-runner · GitLab](https://gitlab.
com/gitlab-org/gitlab-ci-multi-runner)
◎ Deploying to App Engine or Managed VMs from Travis CI
○ [Deploying to App Engine or Managed VMs from Travis CI - YouTube]
(https://www.youtube.com/watch?v=7U4jjRw_AJk&feature=youtu.
be)
○ [GoogleCloudPlatform/continuous-deployment-demo](https:
//github.com/GoogleCloudPlatform/continuous-deployment-
demo)

Study information cont.
◎ GCP
○ [Cloud Source Repositories - Private Git Repositories — Google
Cloud Platform](https://cloud.google.com/source-repositories/)
○ [Stackdriver Debugger - Production Debugging - Google Cloud
Platform — Google Cloud Platform](https://cloud.google.
com/debugger/)
○ [Managed VMs lab · kaichu.io](http://kaichu.io/2015/09/managed-
vms-lab/)

Thanks!
Any questions?
You can find me at:
http://kaichu.io
cage.chung@gmail.com

Continuous Integration & Continuous Delivery with GCP

Recommended

More Related Content

What's hot (20)

Viewers also liked (11)

Similar to Continuous Integration & Continuous Delivery with GCP (20)

More from KAI CHU CHUNG (13)

Recently uploaded (20)

Continuous Integration & Continuous Delivery with GCP