Module 5

Dr. C.R.Dhivyaa
Assistant Professor
School of Computer Science and Engineering
Vellore Institute of Technology, Vellore
Cloud application development
using third party APIs
 What is the 3rd party API?
• A third-party API, or Application Programming Interface,
is a tool that allows you to access the data or functionality of a
third party on your application or site.
• Third-party APIs are often provided by companies like
Facebook, Twitter, or Google.
• Third-party APIs can be used for a variety of applications,
including: Social login, Online payment, Customer data
management, and Webhook implementation
 What is the 3rd party API?
• When you come to a restaurant and want to order, you need a
waiter who will deliver your order to the kitchen and bring it to
your table.
• In this case, API performs as a waiter.
• It sends the request from your mobile app to a third-party API
system and delivers the result back to your app.
• In a nutshell, API describes how one computer program interacts
with another.
• Classes, procedures, functions, structures, and constants are all part
of it.
What is the 3rd party API?
What is the 3rd party API?
 What is the 3rd party API?
• A third-party API is located on a server that belongs to a third-party service.
• Developers can access its features by linking to a JavaScript library or by making an
HTTP request to a specified URL pattern.
• When the connection is made, the user’s request will be sent from the app to a server
through the API and back. – it is a two-way path.
• A 3rd party API consists of the following components:
❑ Client, or in other words system that sends the request to the server
❑ The server might be the computer or database that responds to the request
 Different types of APIs
• There are more types of APIs, based on how they function.
❖ Public or Free APIs
❖ Partner APIs
❖ Internal APIs
o Public or Free APIs
• A public API or open API is free to access by any outside business or developer.
• Public APIs usually have an easier authentication and authorization process.
• The most common open APIs are those provided by Google to access its many
products like AdSense, Telegram, WordPress, Google Maps, etc
 Different types of APIs
o Partner APIs
• These APIs facilitate business-to-business processes and are only available to
authorized subscribers.
• For example, if your app wants to connect to an external Customer relationship
management (CRM) provided by a vendor, then an API will be provided by the
vendor to connect internal data systems with external parties.
• Partner APIs usually have a license agreement and have higher authentication,
authorization, and security mechanisms in place.
• Similar is the case with payment gateway APIs.
 Different types of APIs

o Internal APIs
• These are private APIs that are intended for use only within
the business enterprise.
• For example, data sharing between HR and payroll systems.
 Third-party API examples
❑ Google Maps API is a must for any navigation or geolocation mobile app since it shows
maps and driving directions so your app users don’t get lost.
❑ Facebook API is the social network API integration we use for our projects most often since
it allows authorizing new app users via their existing Facebook profile; thus, users avoid
filling in long registry forms.
❑ Skyscanner API might be helpful if you want to create a travel mobile app and provide
users with flight searches, ticket prices, and live quotes from ticketing agencies.
❑ Booking API will add such information to your mobile app as accommodation queries,
facilities, prices, and other info from hotels around the world.
 Third-party API examples
❑ Spoonacular API with recipes, food, and nutrition is a must for diet and
fitness apps since it includes over 365,000 recipes and 86,000 food products.
Also, your app users can calculate the number of calories in each meal, convert
ingredients, find recipes to meet special diets, and so on.
❑ Movie Database API will work great if you consider developing an on-
demand video streaming platform like Netflix. Such an API provides access to
movie titles, IMDB rating, runtime, release date, awards, plot, and lots of other
data for each title.
❑ Uber API will allow your app users to hail an Uber taxi via your app, pay for a
drive in-app, and leave reviews.
How to do API integration to your mobile
app
 How to do API integration to your mobile app
❑ Step 1 - Hire an API integration developer
• If you are not a tech person, to integrate an API into your app you will
need to hire a mobile app developer.
• For this task, a mobile development specialist will use an SDK
(software development kit) that includes a bunch of software
development tools in one installable package.
• In this way, the developer will correctly integrate the selected API.
 How to do API integration to your mobile app

❑ Step 2 - Create the project within the API provider system

• A mobile app developer should register in the API provider system because
every call to the third-party API endpoint requires authentication.

❑ Step 3 - Receive API key and authorization token

• After registration, the developer gets an API key so the API provider
system can identify the calling project and authentication token to verify
that the calling user has access to make this request.
 How to do API integration to your mobile app
❑ Step 4 - Integrate the API framework for the app
• Now the developer installs the selected SDK with APIs to an app build environment
using dependency managers such as CocoaPods for iOS and Maven for Android apps.

❑ Step 5 - Use API request instances and methods

• To make the app use new features integrated via API, the mobile developer will
implement new functionality using dependencies added during the previous step.
 Working with EC2 API
https://aws.amazon.com/blogs/compute/introducing-amazon-api-gateway-
private-endpoints/
 Amazon API Gateway
• Amazon API Gateway is a fully managed service that makes it easy for
developers to create, publish, maintain, monitor, and secure APIs at any scale.
• APIs act as the "front door" for applications to access data, business logic, or
functionality from your backend services.
• Using API Gateway, you can create RESTful APIs (GET, POST, PUT,
DELETE-REST(Representational State Transfer) and WebSocket APIs that
enable real-time two-way communication applications.
• API Gateway supports containerized and server less workloads, as well as web
applications.
• API Gateway handles all the tasks involved in accepting and processing up to hundreds
of thousands of concurrent API calls, including traffic management, Cross-origin
resource sharing (CORS) [CORS mechanism supports secure cross-origin requests
and data transfers between browsers and servers] support, authorization and access
control, monitoring, and API version management.
• API Gateway has no minimum fees or startup costs.
• You pay for the API calls you receive and the amount of data transferred out and, with
the API Gateway tiered pricing model, you can reduce your cost as your API usage
scales.
How API Gateway Works?
 Google
App Engine API
https://cloud.google.com/appengine/docs/admin-api
 An Overview of the Admin API
• The Google App Engine Admin API is a RESTful API for managing your App Engine
applications regardless of programming language.
• The Admin API provides programmatic access to several of the App Engine
administrative operations that are found in the Google Cloud console.
• After configuring authentication, you can begin managing your apps, including
deploying versions of applications and managing the amount of traffic to those
versions.
 An Overview of the Admin API
• With the Admin API, you can manage your App Engine applications in a manner that
best suits your environment or process.
• For example, the Admin API provides you with:
❑ An integration point for your development and build tools.
❑ Tighter control around deploying new versions, including the ability to automate
traffic migration between two versions or traffic splitting across one or more versions.
❑ The ability to programmatically manage applications across multiple Google Cloud
projects.
 Facebook API

https://www.geeksforgeeks.org/facebook-api-set-1/
 Graph API
• Graph API is the primary way to access means read and write data on the
social media Facebook social graph.
• Graph API is basically composed of 3 components:
✓ Nodes
✓ Edges
✓ Fields

Nodes
• These are individual objects such as a User, Page, Photo, or Comment.
• We use Nodes to gather data about individual Objects.
Fields
• These represent the data or Information about an object. User Age and Date
of Birth are examples of fields that are associated with a node User.
• fields are used to specify the data that we want to include in a response.
Edges
• These represent the connections between collections of objects and single objects.
• Edges are used to publish single objects to the collection of objects or to get collections
of objects that are connected to a single object.
• Suppose we consider a Photo posted on the Facebook Profile of an Individual.
• Now the comments associated with this Photo refer to the collection of Objects that
are associated with a single Object which is a Photo in our case.
• Graph API is HTTP-based, so it works with any Language that has an HTTP, HTTP-
based library like curl, urllib, etc.
• All requests are passed to the graph.facebook.com host URL.
• Access Token here majorly serves 2 purposes:
❑ It allows the app to access a User’s Data without requiring the User’s credentials.
❑ It allows the server to identify who is using the app and to determine accordingly
what functions that user is permitted to do.
 4 types of Access Tokens
User Access Token
▪ This is the most commonly used access token, needed when an app calls an API to read,
modify, or write data on a specific person’s Facebook Page.
▪ This is achieved through a login dialog where a person is required to permit the app to
obtain an access token

App Access Token

▪ This token is obtained using a server-to-server call.
▪ This token is used to read and modify the app settings.
Page Access Token
• Once you have the User Access Token you obtain the Page Access Token via the Graph
API.
• The procedure is very simple, first, obtain the User Access Token and then ask for the
manage_posts permission.
• You will get your Page Access Token. This token is very similar to the User Access
Token, except for the case where it provides permission to the API that reads, modifies, or
writes data to a Facebook Page.

Client Token
▪ A client token is an identifier that can be embedded in native mobile binaries or desktop
apps.
▪ It is used to access app-level APIs.
 Twitter API

https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api
• The Twitter API can be used to programmatically retrieve and analyze Twitter
data, as well as build for the conversation on Twitter.

• Over the years, the Twitter API has grown by adding additional levels of access
for developers and academic researchers to be able to scale their access to enhance
and research the public conversation.

• Recently, we released the Twitter API v2. The Twitter API v2 includes a modern
foundation, new and advanced features, and quick onboarding to Basic access.
 How to get access to the Twitter API (1/3)

❑ Step one: Sign up for a developer account

• Signing up for a developer account is quick and easy!
• Just click on the signup button, answer a few questions, and you can start exploring and
building on the Twitter API v2 using Basic access.
• Next, you will create a Project and an associated developer App during the onboarding
process, which will provide you with a set of credentials that you will use to
authenticate all requests to the API.
 How to get access to the Twitter API (2/3)
❑ Step two: Save your App's key and tokens and keep them secure
• Once you have access and have created a Project and App, you will be able to
find or generate the following credentials within your developer App:
➢ API Key and Secret:
✓ Essentially the username and password for your App.
➢ Access Token and Secret:
✓ In general, Access Tokens represent the user that you are making the request on
behalf of.
✓ The ones that you can generate via the developer portal represent the user that
owns the App.
 How to get access to the Twitter API (3/3)
➢ Client ID and Client Secret:
✓ These credentials are used to obtain a user Access Token with OAuth 2.0
authentication.
✓ The user Access Tokens are used to authenticate requests that provide private user
account information or perform actions on behalf of another account
➢ App only Access Token:
✓ You will use this token when making requests to endpoints that respond with
information publicly available on Twitter.
❑ Step three: Make your first request
 Hadoop
• Hadoop is an open-source software
framework that is used for storing and
processing large amounts of data in a
distributed computing environment.

• Hadoop at its core is essentially two

elements:
• Map Reduce

• Distributed File System HDFS

 Hadoop Distributed File System (HDFS)
• The Hadoop Distributed File System (HDFS) is a data storage system that's part
of the Hadoop ecosystem, an open-source framework for managing big data.

• HDFS is used to store and manage large data sets, and is a key component of
many Hadoop systems.

• HDFS is a distributed file system that is fault-tolerant, scalable, and extremely

easy to expand.

• HDFS is the primary distributed storage for Hadoop applications.

• HDFS provides interfaces for applications to move themselves closer to data.

• HDFS is designed to ‘just work’, however, working knowledge helps in

diagnostics and improvements.
 HDFS Architecture
• Hadoop Distributed File System follows the master-slave architecture. Each
cluster comprises a single master node and multiple slave nodes.
• Internally the files get divided into one or more blocks, and each block is stored on
different slave machines depending on the replication factor (which you will see
later in this article).
• The master node stores and manages the file system namespace, that is information
about blocks of files like block locations, permissions, etc.
• The slave nodes store data blocks of files.
• The Master node is the NameNode and DataNodes are the slave nodes.
Components of HDFS
 Components of HDFS
Name Node
• It is the master daemon that maintains and manages the
Data Nodes.
• It records the metadata of all the files stored in the cluster.
• For example, The location of blocks stored, the size of the
files, permissions, hierarchy, etc.,
• Maps a file name to a set of blocks
• Maps a block to the Data Nodes where it resides
 Components of HDFS
Functions of NameNode
✓It is the master daemon that maintains and manages the DataNodes (slave nodes)
✓It records the metadata of all the files stored in the cluster, e.g. The location
of blocks stored, the size of the files, permissions, hierarchy, etc. There are two
files associated with the metadata:
FsImage: It contains the complete state of the file system namespace
since the start of the NameNode.
EditLogs: It contains all the recent modifications made to the file system with
respect to the most recent FsImage.
✓It keeps a record of all the blocks in HDFS and in which nodes these blocks are
located.
 Components of HDFS
Data Node
✓DataNodes are the slave nodes in HDFS.
✓DataNode is a commodity hardware, that is, a non-expensive system which is
not of high quality or high-availability.
Functions of DataNode:

✓These are slave daemons or process which runs on each slave machine.
✓The actual data is stored on DataNodes.
✓The DataNodes perform the low-level read and write requests from the file
system’s clients.
✓ They send heartbeats to the NameNode periodically to report the overall health of
HDFS, by default, this frequency is set to 3 seconds.
 Components of HDFS
Block
✓Blocks are the nothing but the smallest continuous location on your
hard drive where data is stored.
✓A Block is the minimum amount of data that it can read or write.
HDFS blocks are 128 MB by default and this is configurable.
 HDFS – Data Organization
• Each file written into HDFS is split into data blocks.
• Each block is stored on one or more nodes.
 HDFS – Data Organization
• Each copy of the block is called a replica.

• Block placement policy

• The first replica is placed on the local node.

• The second replica is placed in a different rack.

• The third replica is placed in the same rack as the second replica.

• Additional replicas are randomly placed

• Clients read from the nearest replicas.

 HDFS – Data Organization

• HDFS provides a reliable way to store

huge data in a distributed environment
as data blocks.
• The blocks are also replicated to
provide fault tolerance.
• The default replication factor is 3 which
is again configurable.
• Each block is replicated three times and
stored on different DataNodes
HDFS – Data Organization
HDFS Architecture
HDFS Architecture
 An example of HDFS
• Consider a file that includes the phone numbers for everyone in the United States;
the numbers for people with a last name starting with A might be stored on server 1,
B on server 2, and so on.
• With Hadoop, pieces of this phone book would be stored across the cluster, and to
reconstruct the entire phone book, your program would need the blocks from every
server in the cluster.
• To ensure availability if and when a server fails, HDFS replicates these smaller
pieces onto two additional servers by default.
• The redundancy also allows the Hadoop cluster to break up work into smaller
chunks and run those jobs on all the servers in the cluster for better scalability.
HDFS Read Operation
HDFS Read Operation
HDFS Read Operation
HDFS Write Operation
HDFS Write Operation
HDFS Write Operation
 Features of HDFS

• Failure tolerant - data is duplicated across multiple Data Nodes

to protect against machine failures. The default is a replication
factor of 3 (every block is stored on three machines).
• Scalability - data transfers happen directly with the Data Nodes
so your read/write capacity scales fairly well with the number of
Data Nodes
• Availability
 Suitability

Where to use HDFS?

• Very Large Files:
• Files should be of hundreds of megabytes, gigabytes, or more.
• Streaming Data Access:
• The time to read the whole dataset is more important than latency in
reading the first.
• HDFS is built on a write-once and read-many-times pattern.
• Commodity Hardware:
• It works on low-cost hardware
 Suitability

Where not to use HDFS?

Low Latency data access:
• Applications that require very little time to access the first data should not use
HDFS as it gives importance to the whole data rather than time to fetch the first
record.
Lots Of Small Files:
• The name node contains the metadata of files in memory
Multiple Writes:
• It should not be used when we have to write multiple times.
 MapReduce Programming Model
• MapReduce is a Hadoop framework and programming model for
processing big data using automatic parallelization and distribution in the
Hadoop ecosystem.
• Apache Hadoop is a software framework that processes and stores big data
across the cluster of commodity hardware.
• Hadoop is based on the MapReduce model for processing huge amounts of data
in a distributed manner.
• MapReduce is a software framework for writing applications that can process
huge amounts of data across the clusters of in-expensive nodes. Hadoop
MapReduce is the processing part of Apache Hadoop.
 MapReduce Programming Model
Features
 MapReduce Programming Model
Different Phases of MapReduce

MapReduce model has three major and one optional phase.​

•Mapping

•Shuffling and Sorting

•Reducing

•Combining
 MapReduce Programming Model
Different Phases of MapReduce
 MapReduce Programming Model
Different Phases of MapReduce
 MapReduce Programming Model
Different Phases of MapReduce
 MapReduce Programming Model
Different Phases of MapReduce

Mapping - The input data is processed and divided into smaller segments in the
mapper phase, where the number of mappers is equal to the number of input splits.
Then, map tasks create a <key, value> pair for every word.
Shuffling - the output of the mapper phase is passed to the reducer phase by
removing duplicate values and grouping the values.
Sorting - Sorting is performed simultaneously with shuffling. The Sorting phase
involves merging and sorting the output generated by the mapper.
Reducing - reduced to produce a single output value that summarizes the entire
dataset
MapReduce Programming Model
 MapReduce Programming Model
1. Application starts a Master instance and M worker for the Map
phase and R worker for the Reduce phase

2. The master partitions the data into M segments

3. Each map instance reads its data segment

4. The results are stored on the local disk and R instance reads the
results

5. The final results are written by Reduce instances

6. The Master instance monitors the Reduce instances

 MapReduce Programming Model
Example 1
Example 2
Solution
 Mapper and Reducer

CSI3001 - Cloud Computing Methodologies SCOPE