Google Cloud Platform

Chapter 2

Prelude
If you have heard the term Cloud Computing, then you would be aware of Virtual
Machine (VM). Google Compute Engine (GCE) in GCP will let you run the VM in
Google Cloud Platform Infrastructure. You will configure your VM in such a way that
you will build a physical server by specifying its CPU power, memory, storage types
and OS.

In this topic, you are going to acquire knowledge on how GCE works with Google

Virtual Networking.

Virtual Private Cloud Network

 The first thing to do in GCP is to define a Virtual Private Cloud (VPC)
network inside the GCP Project. You can either create the new VPC network or use
the default VPC network.
 The job of VPC networks is to connect your GCP resources with each other or with
the internet.
 You can divide your VPC network into segments, apply firewall rules to them (allow
or restrict access to VMs), and create static routes to forward traffic to specific
destinations.
 The VPC network that you create in your project will have a Global scope.
 VPC network can consist of Subnets, which can be present in the zones that make
up a region.

3)

More About Google Compute Engine

 The basic functionality of the Compute Engine is to create and run the Virtual
Machines on the Google Infrastructure.
 You can create your own Virtual Machine in GCP by using GCP
Console or gcloud command-line tool.
 Your VM is capable of running images of either Linux or Windows provided
by Google or you can create your own Custom Images or you can import
images from your Physical Servers.
  You can choose how much Memory and how many Virtual CPUs your VM
should have.
 Regarding Processing Power, if you have workloads like Machine
Learning and Data Processing, then you can consider GPUs for your VM.

More About Google Compute Engine

 Persistent Storage, that is made available to your VM is of two types,
namely, Standard and SSD. You can also add Local SSD for high
performance.
 The data storage in Local SSD is not safe because when the VM terminates,
data will get erased. Therefore, it is better to consider Persistent Disks for
the permanent storage of your data. The default disk is Standard Persistent
Disk
 If you want your VM to spin up with configurations like Installing software
packages on first boot, you can do it with Start-up scripts. There are also
other fields like Metadata, which gives unique information about the VM.
 You can take snapshots of your disk for the backups. Suppose if you want
to do a batch job like analyzing large data sets, then consider options
like Preemptible VM, which saves a lot of money.
 The maximum number of virtual CPUs in a VM is 96, and the maximum
memory size is 624 Gigabytes, which will be useful for heavy workloads
like In-memory databases and CPU intensive analytics.
 GCP also supports a feature called AutoScaling, which allows you to add or
remove VMs. You can also balance the incoming traffic among different VMs
through Load Balancing.

Google cloud storage and bit table

Introduction
Developers develop many applications that require the storage of large amounts of
data. Data can be in many forms like Media, Confidential data from devices,
Customer account balances, and so on.
Earlier, we read that data can be stored in Persistent disks. Also, GCP provides
other storage options for Structured or Unstructured data, Transactional, and
Relational data.
In this topic, you will be learning the various storage options like Cloud Storage,
Cloud Bigtable, Cloud SQL, Cloud Spanner, and Cloud Data storage .
More About Cloud Storage
 Google Cloud Storage is an Object Storage.
 Google Cloud Storage will store the data in objects, which will be in the form
of an arbitrary bunch of bytes, that can be addressed using a unique key.
 These unique keys are in the form of URL, which makes it easier to interact
with Web Technologies.
 Cloud storage is comprised of Buckets, which is used to hold the Storage
Objects.
 Storage Objects are immutable, that is, for every new change, a new version
will be created.
 Object Versioning is another important concept in cloud storage. If you
turn ON this feature, it will store every version of the object. Otherwise, the
newer version will override the old one.
 When you create a bucket, you will provide a Unique name to it, specify
a Geographic location to store the data and also choose a Storage
Class by default.
 You can control the access to the bucket using Cloud IAM and Access
Control List (ACL).

About Cloud Bigtable

The following are a few reasons to select Cloud Bigtable as your storage
option:

 Scalability: With the help of Bigtable, you can increase your machine count,
which does not require any downtime. Also, it handles all Administration
tasks like Upgrades and Restarts.
 The data present in the Cloud Bigtable is encrypted. You can use IAM
roles to specify who can access the data.
 From an application API perspective, data is written to and from Bigtable
through data service layers like Managed VMs, HBase Rest Server, or Java
Server using HBase Client.
 Cloud Bigtable will serve the data to Applications, Dashboards and Data
services.
  Data can be read from and written to Bigtable through batch processes
like Hadoop MapReduce, DataFlow or Spark.

CLOUD SQL SPANNER AND DATASTORE

More About Cloud SQL

The following example will give you an idea on the benefits of using Cloud SQL.
Consider you have two accounts in a bank, and you want to transfer Rs.10,000 from
one account to the other.
Due to a glitch, the amount gets deducted from one account but is not credited to the
other. Relational databases require a lot of work to set up, maintain, manage, and
administer, but still if you want to protect your relational databases like in the above
case, then Cloud SQL is your choice.

 Cloud SQL offers MySQL and PostgreSQL databases as a managed

service, that are capable of handling terabytes of storage.
 It also provides several replica services like Read, failover and external.
 Cloud SQL instances include firewalls, customer data encrypted on
Google's internal networks, and schedule backups, and it can also be
accessed by other GCP services.
 It can be scaled vertically by changing the machine type and horizontally
through Read replicas.
 If you need horizontal scalability more than what is provided by Cloud SQL,
then you can go for Cloud Spanner.

CONTAINERS
Google Compute Engine and Google App
Engine - Drawbacks
 In Google Compute Engine (GCE), you are allowed to share resources by
virtualizing the hardware using VMs. The developer can deploy the OS,
access the hardware, and build their applications in a self-
contained environment along with access to Networking Interfaces, File
Systems, RAM, and so on.
 When the demand for your application increases, then you need to copy the
entire VM and boot the guest OS for each instance of the application which
can be slow and expensive.
 By using the Google App Engine (GAE), you will gain access to the
programming services. The only job that you need to do is to write the code,
and the self-contained workloads will use these services and include any
dependent libraries.
 As the demand for your application increases, the platform scales your
applications seamlessly and independently, and you need to give up the
underlying server architecture control.

Containers
 Containers are preferred due to the drawbacks in the Compute
Engine and App Engine.
 Containers are responsible for the independent scalability of workloads. They
also act as an abstraction layer between the OS and Hardware.
 All you need when you get a host with an OS is it should
support Containers and Container Runtime.
 Your code will be portable and you can treat the OS and hardware as Black
Box. Using this functionality, you can move between stages
like Development, Staging, and Production from on-premises to the cloud.
 For example, if you want to scale a web server a hundred times, then you can
do it in seconds on a single host depending on the size of the workload.

Kubernetes
 If you want to build your application using containers acting like microservices,
which are connected through network connections, then you can make
 them modular, scale independently, and easily deployable across a group
of hosts.
 Containers can be scaled up or down, started, or stopped by the hosts on-
demand as your application changes or else, the host fails.
 The above process can be done using a tool called Kubernetes.
 The function of Kubernetes is to orchestrate several containers on hosts,
scale them as microservices, and perform rollouts, rollbacks.
 An open-source tool called Docker, helps you to define a format for bundling
your application with machine-specific settings and dependencies into a
container.
 GCP also has a separate tool called Google Container Builder, a managed
service that is used for building containers.

Playing with Docker

 Consider a simple Python app whose output is Hello World. When you hit its
endpoint, it will give you the version.
 Here, the question is, how will you get the app into Kubernetes.
 The versions of Python and Flask are managed
through Requirements.txt file along with the other dependencies.
 You can use Docker Build command to build the container and store it on the
local system as a runnable image.
 Docker run command is used to run the image.
 This image can also be saved and uploaded to Container Registry
Service (a place to manage all docker images), from where it can be
downloaded.

Google Kubernetes Engine

 Kubernetes is a set of APIs which can be used to deploy containers on a set
of nodes called Cluster.
 If you want to start Kubernetes on a cluster in Google Kubernetes Engine
(GKE), run the following command in Cloud Shell.

–– container cluster create k1

 You can deploy Containers on the nodes using a wrapper called Pod.

  Pod is the smallest unit in Kubernetes that you can create or deploy. It can be
a component of an application, or it can be an entire application and can
contain single or multiple containers.
 You can package multiple containers into a single pod, and these containers
will automatically share networking and can have disk storage volumes in
common.
 Each pod will get a separate and unique IP, and a set of ports for your
containers.
 Containers in the pod will communicate with each other using localhost and
fixed ports.
 Kubectl command is used to run the container in the pod, resulting in an
immediate deployment of a container in the running pod.

Google Kubernetes Engine

 Deployment is represented as a group of replicas of the same pod that helps
you to run the pods continuously, even though the nodes fail.
 kubectl get pods command is used to get the pods.
 Pods in the Deployment are only reachable inside the GKE Cluster. You
can make them available to the public by connecting them to an external
load balancer using the kubectl expose command.
 In Google Kubernetes Engine, a load balancer is created as a Network
Load Balancer.
 A Service groups a set of pods and provide Fixed IP for them.

Kubectl Commands in Pods

 kubectl scale command can be used to scale the deployment.
 You can check the deployment to make sure that replicas are running by
using the command kubectl get
 deployments or kubectl describe deployments.
 You can get the External IP of the service by running the command kubectl
get services.
 A Configuration file can also be provided instead of using commands to
specify what your desired state looks like.
 For example, there are 3 replicas of an Nginx pod, and you want to update it
to 5 replicas. Then, you can edit the deployment configuration file from 3 to 5.
 Run kubectl apply command to update the changes to the configuration file
and run kubectl get replica sets command to see the replicas and their
updated states.
App ENGINE STANDARD

Introduction
By now you are familiar with two crucial GCP products,  Compute
Engine and  Kubernetes Engine.

 One common feature in these two products is you will choose the
infrastructure in which your application will run, that is, Virtual
Machines for Compute Engine and Containers for Kubernetes Engine.
 If you want to focus on the application code, then your choice should be App
Engine.
 App Engine comes under Platform as a Service (Paas). App Engine
manages both Hardware and Network Infrastructure.
 App Engine has many built-in services like No SQL databases, in-memory
caching, load balancing, health checks, logging, and authenticating
users, which can be used by your application.
 App Engine will automatically scale applications like web
applications and mobile backend.

App Engine Standard

App Engine offers two main environments namely  App Engine
Standard and  App Engine Flexible.

 App Engine Standard environment offers a more straightforward deployment

than App Engine Flexible.
 It offers free usage quota for certain services, which means low utilization
applications will be able to run at a small charge.
 Google App Engine Software Development Kits (SDK) are available in
several languages. You can test your application using these kits before
deploying your application in the real App Engine Service.
 Google provides runtime for App Engine Standard Environment in Java,
Python, Go, Ruby, Node.js and PHP.
App Engine Standard
 App Engine can apply restrictions to your code by running it in a software
construct known as Sandbox.
 Sandbox is independent of the hardware, Operating System, and Physical
location of the server. Due to this, the App Engine can manage and scale your
applications.
 App Engine Standard composes certain constraints like one cannot write to
the local file system.
 If you are unable to install third-party software which is not suitable for you,
then you should choose App Engine Flexible.

App Engine Flexible

 In case the App Engine Standard sandbox model constraints are not suitable
for you, but you want to use App Engine, then you can choose App Engine
Flexible environment.
 App Engine Flexible will let you decide the container in which your application
should run, unlike Sandbox.
 It enables you to run Docker Containers with Compute Engine Virtual
Machines. These VMs are health checked and healed, as necessary.
 You can also choose the geographical location in which they have to run.
 App Engine Flexible uses standard runtimes and also has access to services
such as Data Store, Memcached, Task Queues, and so on.

App Engine Standard vs. App Engine

Flexible
Kubernetes App Engine
App Engine Flexible
Engine Standard

Language Java, Python, Go,

Any Any
Support PHP
 Kubernetes App Engine
App Engine Flexible
Engine Standard

Service
Hybrid PaaS PaaS
Model

Primary Use- Container-based Web and Mobile Both web and mobile applications
Case workloads applications and container-based workloads

Application Programming Interface

Let us discuss theApplication Programming Interface (API) in brief.

 A Software Service implementation is complex and versatile.

 The developers will structure the software to present a clean and well-
defined interface, which will be documented after abstracting the needless
information. This interface is known as API.
 You can make changes to an API by adding or deleting a feature. Each
change will be considered as a different version of the API.
 The program which wants to use an API can mention the version in the API
call.

API Management Tools

 Google Cloud Platform provides two types of API Management Tools,
namely, Google Cloud Endpoints and Apigee Edge.
 Consider you are developing a Software Service and GCP Backend, and you
want to expose the API.
 If you want the API to be consumed by trusted developers, you should
monitor and log its use. You will also require to know which end-user is
making a call. This is where Cloud Endpoints come into play.
 Cloud Endpoints will deploy proxy in front of the software service and provide
API Console to combine all the capabilities mentioned.
 Apigee Edge is also helpful in developing and managing API proxies.
 Apigee Edge focuses on Rate limiting, Quotas, and Analytics.
 By using Apigee Edge, instead of removing the entire application at once, you
can remove the services one by one until the whole application is retired.
DEVELOPEMNET< DEPLOYMENT MONITORING

Development in the Cloud

People develop significant applications in Google Cloud. GCP provides
extraordinary tools for different phases like:

1. Development
2. Deployment
3. Monitoring

 Most of the developers will store and maintain their code in the Git
Repository.
 You can also create Git Instance (which gives you great control) or Git
Provider (which lessens your work).
 In addition to the above options, you can keep your code private and add IAM
permissions to protect it by using the Cloud Source Repository (CSR).

Cloud Source Repository

 Cloud Source Repository provides the Git version control for the applications
that also run on Compute Engine, App Engine and Kubernetes Engine.
 You can have any number of Private Git Repositories with CSR.
 Source Viewer in CSR helps you to browse and view the repository files
through the GCP Console.
 Event-driven actions can also be present in the applications. Let us consider
an example. Suppose you have an app in which the users need to upload
images.
 For uploading the images, you need to process the image in various ways
such as converting it to the standard image format, resizing it, and storing it in
the repository.
 You can integrate the above function into the application, but the major
drawback is providing resources to it.
Cloud Functions
 Cloud Functions is a single purpose function that will perform image
manipulations and run automatically when a new image is uploaded.
 It can trigger events in Cloud Pub/Sub, Cloud Storage and HTTP call.
 For each event type, you will specify which event you are interested in, and
this type of declaration is called Triggers.
 You will attach JavaScript functions to the triggers, and the functions respond
whenever the event occurs.
 Cloud Functions can be used to enhance the application without worrying
about scaling.

Deployment
Setting up an environment in GCP will take a lot of steps like

1. You can set up compute network, storage network, and their configurations.
2. If you want to change the environment, you can do it with some commands.
3. You can also clone the environment by executing some commands.

 The above steps will take a lot of time and you can reduce it by using
a Template. ( Requires specification for the environment that you want it to
be).
 GCP provides , Deployment Manager to automate the creation and
management of the above steps by using the template.
 You can create the Template file either by using YAML Markup
language or Python. Then the template will be consumed by the Deployment
Manager and performs the actions that need to be done.
 You can edit the template and tell the Deployment Manager to make changes
according to that.

Deployment Manager Templates can be stored and version controlled in Cloud Sources
Repository.

Monitoring
Without  Monitoring your application you cannot run it stably.
  It will analyze and let you know whether the changes are functional or not. It
will respond with some information whenever your application is down.
 Stackdriver is the monitoring tool that you can use in GCP for Monitoring,
Logging and Diagnosing.
 Stackdriver will give you an entry to receive signals from Infrastructure
Platform, Virtual Machines, Middleware, Application tier, Logs, Metrics,
and Traces.
 It also helps you to check the Application health, Performance and
Availability.
 With Stackdriver, you can perform Monitoring, Logging, Tracing, Error
Reporting and Debugging.
 You can configure uptime checks which are associated with URLs and
Resources such as Instances and Load Balancers.
 You can also set up alerts on Health check results or Uptime falls.

Stackdriver
 You can also use a Monitoring tool with Notification tools. You can also view
your Application state by creating Dashboards in Stackdriver.
 Stackdriver Logging will allow you to view logs of your application.
 Logging also allows you to define metrics depending on the Log content that
is included in the Dashboards and Alerts.
 You can export logs to Cloud Pub/Sub, Cloud Storage and BigQuery.
 Stackdriver Error Reporting will track and group all the errors in the cloud
application.
 Stackdriver Trace is used to sample the latency of App Engine applications.
 Stackdriver Debugger connects your production data to the source code. It
works efficiently when your source code is available in Cloud Source
Repository.

GOOGLE CLOUD BIG DATA

Google Big Data Platform

Google Big Data solutions will help to transform the users and their business
experiences through data insights which are known as  Integrated Serverless
Platform.
  Google Big Data solutions are a part of GCP Services that are fully
maintained and managed. You only pay for the resources that you consume.
 The following services are offered by Google Big Data Platform and are
integrated to help create custom solutions:
o Apache Hadoop is an open-source framework for Hadoop, which is
based on the MapReduce programming model.
o MapReduce model consists of Map Function, which runs with a large
dataset to generate intermediate results. By taking these results as
input, Reduce Function will produce the final output.
o Along with Apache Hadoop, there are other related projects
like Apache Pig, Hive and Spark.
o On Google Cloud Platform, Cloud Dataproc can be used to
run Hadoop, Spark, Hive and Pig.

Cloud Dataproc
 When you request a Hadoop Cluster, it will be built in less than 90 seconds
on top of the VM. Scaling can be done up and down based on the processing
power.
 You can monitor the cluster using Processing Power.
 Running the clusters in On-premises will require hardware investment. But,
running them in Dataproc will allow you to pay only for the hardware
resources that you use while creating the cluster.
 Cloud Dataproc is billed per second, and GCP stops the billing once the
cluster is deleted.
 You can also use Preemptible instances for the batch processing to save
costs.
 Once the cluster consumes the data, Spark and SparkSQL can be used for
data mining.
 You can also use Apache Spark Machine Learning Libraries to discover
patterns through Machine Learning.

Cloud Dataflow
Cloud Dataproc is suitable when you know your cluster size. But if your
cluster size is unpredictable or your data shows up in real-time, then your
choice should be  Cloud Dataflow.
  Cloud Dataflow is a managed service that allows you to develop and execute
a large range of processing patterns by extracting, transforming, and
loading batch computation or continuous computation.
 Cloud Dataflow is used to build Data pipelines for
both Batch and Streaming data.
 It is used to automate processing resources, free you from operational tasks
like Performance optimization and Resource Management.
 Cloud Dataflow can read data from BigQuery, process it, apply transforms
like Map operations and Reduce Operations and write it to the Cloud
Storage.
 Use cases include Fraud Detection and Financial Services, IoT Analytics,
Manufacturing, Logistics, HealthCare and so on.

BigQuery
Say you possess a large dataset, and you need to perform ad-hoc SQL
queries, then you need to go for  BigQuery.

 BigQuery is Google's fully managed, low-cost analytical data warehouse that

has Petabyte storage.
 You can get data into BigQuery either from Cloud Storage or Cloud
Datastore and can stream that to BigQuery up to 100,000 rows per second.*
You can perform super-fast SQL queries and read, write data to BigQuery
through Cloud Dataflow, Spark and Hadoop.
 You only pay for the queries that you are running.
 When the data in BigQuery reaches 90 days, Google will automatically
decrease the storage price.
 BigQuery has an availability of 99.99%

Cloud Pub/Sub and Cloud Datalab

If you are working with events in real-time and you need a  messaging service,
then  Cloud Pub/Sub will help you in the following ways:

 Cloud Pub/Sub is a simple, reliable, and scalable foundation for stream

analytics. By using it, you can build independent applications to send and
receive messages.
 Pub/Sub stands for Publishers and Subscribers.
  The application will publish their messages to pub/sub, and the subscribers
who are subscribed to them will receive the messages.
 Cloud Pub/Sub can also be integrated with Cloud Dataflow.
 Cloud Datalab helps to explore the data, and it can also be integrated with
multiple GCP Services like BigQuery, Cloud Storage and Compute Engine.
 Cloud Datalab runs on a compute engine, and you can mention the specific
region to run it.

Google Cloud Machine Learning Platform

 Machine Learning (ML) is one of the branches of Artificial Intelligence.
 In ML, coders will build systems that will improve the performance by
repeatedly exposing to sample data, known as the training data.
 Youtube, Photos, Google Mobile app, and Google Translate are some of
the Google applications that use ML.
 Google Machine Learning Platform is available as a cloud service. It provides
machine learning services, and pre-trained models to generate custom
models.
 Tensor Flow is an open-source software library for developing machine
learning applications like Neural Networks.
 Tensor Flow will use Tensor Processing Units (hardware devices designed
to accelerate ML workloads with TensorFlow).
 You can build ML models of any size by using the Google Cloud Machine
Learning Platform.

More on ML Platform
 If you want to add different machine learning capabilities to your applications,
then you can add it through Machine Learning APIs, which are provided
through Google Cloud.
 Cloud Machine Learning Platform can be used in different applications based
on the type of data, namely, Structured or Unstructured.
 For structured data, ML is used for Classification and Regression
tasks like Customer churn analysis, product diagnostics and
Forecasting.
 For unstructured data, you can use ML for Image Analytics such as Damage
Shipment, Identifying Styles and Flagging Content.
 You can also perform Text analytics like Blog analysis, Language
Identification, and Topic Classification.
Machine Learning APIs
There are several Machine Learning APIs that are provided to the users by
Google Cloud. They are as follows:

 Cloud Vision API helps its users to identify the content of an image and to
classify the image into a predictable number of categories.
 Cloud Speech API helps to convert audio to text. It can recognize around 80
languages.
 Cloud Natural API provides different natural language technologies to
developers around the world by doing Syntax analysis, identifying verbs,
nouns, adverbs, adjectives, and it can also find the relationship between
words.
 Cloud Translation API can convert a simple arbitrary string to a repositoryed
language through a simple interface.
 Cloud Video Intelligence API helps to annotate videos in different formats.
You can use it to make your video content searchable.

Conclusion
Learn as if you were to live forever -Mahatma Gandhi

In this course, we have discussed the following topics that laid the foundation
for Google Cloud Platform:

 Google Cloud VPC and Compute Engine

 Google Cloud Storage and Bigtable
 Cloud SQL, Spanner, and Datastore
 Containers, Kubernetes and Kubernetes Engine
 App Engine Standard and Flexible
 Cloud Development, Deployment and Monitoring
 Google Cloud Big data and Machine Learning