Introduction to big data and apache spark

© COPYRIGHT 2016 GLASSBEAM INC. CONFIDENTIAL. DO NOT DISTRIBUTE 1
Mohammed Guller
May 17, 2016
Introduction to
Big Data and Apache Spark

Big Data
Big Data Technologies
Hadoop
Spark Core
Spark Libraries
Agenda

About Me
• Principal Architect at Glassbeam
• Founded two startups
• Passionate about building products, big data analytics, and
machine learning
• www.linkedin.com/in/mohammedguller
• @MohammedGuller
4

• Hands-on guide with lots of examples
• Covers both fundamental and advanced
topics such as machine learning
• Includes a primer on functional
programming and Scala
• Introduces other important Big Data
technologies such as HDFS, Parquet,
Kafka, HBase, Cassandra, Mesos, and
YARN
Big Data Analytics with Spark
Available on Amazon

About Glassbeam
Glassbeam brings structure and meaning to data from any connected machine or device while providing
actionable intelligence
Cloud based analytics platform that helps
organizations turn raw machine data to insights
Making sense of multi
structured machine data
 Data center devices
 Medical devices
 Sensors
 ATMs
 Automobiles
 Data from any machine
Providing comprehensive set of apps
& tools for machine data analysis
 50,000+ systems being tracked today
 1,500+ different software rev codes
 1.2 Billion sensor readings per day
 1+ Trillion sensor readings tracked

Big Data
Open-source Big Data Technologies
Hadoop
Spark Core
Spark SQL
Spark Streaming
Machine Learning
GraphX

What is Big Data
Volume
Scale of Data
Variety
Diversity of Data
Velocity
Speed of Data
•
•
•
•
•
•
•
•
•

Data Growing At a Faster Pace Than Ever
10

Internet of Things (IoT)
• Network of objects embedded with
software for collecting and sending data
over the Internet
• 5x more connected things than people by
2020

Industrial IoT
• Manufacturing
• Automotive
• Medical
• Data Center
• EVC
• Smart Meter
12
Glassbeam target market is focused on driving opera onal & business
analy cs value for connected product companies in Industrial IoT market
IT & Networks Medical & Health Care
Transporta on
EV Chargers & Smart Grid
Industrial & Mfg

Big Data Comes With Big Challenges
• Storage
• Processing
• Value

Storage Challenges
• Legacy SAN / NAS storage devices are expensive
• Traditional RDBMS were not designed for Big Data
• Cannot handle volume, velocity, variety of Big Data
14

Processing Challenges
• Diverse processing
• Organizations want do more than just BI / traditional analytics
• Go beyond SQL queries
• Timeliness
• Process data in reasonable amount of time
• Value of data decreases over time
15

How Much Data Can a Standard Server Process

•
•
17

• Large number of CPUs / cores
• Faster cores
• Large amount of memory
• Faster memory bus
• High-performance architecture
Scale-up with Powerful High-end Server
18

Disadvantages of Scale-up Architecture
• Proprietary
• Expensive
• Limited scalability
19

• Cluster of servers
• Commodity machines
• Pool together resources
• CPU
• Memory
• Disk
Scale-out Architecture
20

Benefits of Scale-out Architecture
• Relatively inexpensive
• Economical to scale
• No huge upfront investment
• Start small and expand cluster as workload increases
21

Challenges With Scale-out Architecture
• Writing distributed applications is very hard
• Split job into chunks that can be distributed across a cluster
• Schedule compute resources among different jobs
• Manage inter-node communication
• Handle network and node failures
• Hardware failures are more common at a cluster level
• Probability of a single node failing is low
• Probability of any one node in a large cluster failing is high
22

Getting Value Out of Big Data
• Traditional analytics / BI
• Custom processing
• Machine Learning
• Predictive analytics
• Automate complex tasks
• Stream processing
• Analyze in real-time/near real-time
• React in real-time
23

Traditional Analytics / BI
• What
• Customer growth for the last month/quarter/year
• Segmentation of customers by demographics
• Average time spent by mobile app users
• Why
• Sales growth slowed
• regional issue
• supply issue
• Profit dropped
• revenue dropped
• expenses increased
24

Custom Processing
• Index web pages
• Google
• Bing
• Process genome data
• Identify mutations linked to cancer, Alzheimer's and other disease
• Click analysis
• Log analysis
• 360-degree real time view of a customer
25

Predictive Analytics
• Advertisements that a visitor will most likely click
• Movies / songs / news that a customer will like
• Products that a customer will buy
• Patient will have an heart attack
26

• Virtual assistant
• Siri
• Google Now
• Autonomous machine
• Self-driving car
• Robots
• Tag Images
• Facebook
• Flickr
• Expert System
• Medical diagnosis
• Personalized medicine
• Security
• Fraud detection
• Network Security
• Music recognition
• Shazam
• SoundHound
Automate Complex Tasks
27

Big Data
Hadoop
Spark Core
Spark SQL
Spark Streaming
Machine Learning
GraphX

•
•
•
•
•
•
29

•
•
•
•
•
•
30

• Text
• CSV
• JSON
• XML
• Binary
• Sequence File
• Avro
• Parquet
• Optimized Row Columnar
(ORC)
File Formats
31

• Hive
• Impala
• Presto
• Drill
• Phoenix
• HAWQ
• Tajo
• Spark SQL
Distributed SQL Query Engine
32
Data Warehouse
Distributed
Storage
Distributed
Query Engine

•
•
•
•
•
•
33

•
•
34

•
•
35

Messaging Systems
• Kafka
• RabbitMQ
• ActiveMQ
• ZeroMQ
36

• Batch
• Hadoop MapReduce
• HPCC
• Stream
• Storm
• Samza
• Kafka Streams
• Batch and Stream
• Spark
• Flink
• Beam
• Apex
• Ignite
General-purpose Data Processing Frameworks
37

Big Data
Hadoop
Spark Core
Spark SQL
Spark Streaming
Machine Learning
GraphX

Hadoop

•
•
•
•
40

•
•
•
•
•

Hadoop is Not a Single Product
45

Hadoop Core Components
46
=

Big Data
Hadoop
Spark Core
Spark SQL
Spark Streaming
Machine Learning
GraphX

•
•
•
48

•
•
•
•
•
•
•
49

Spark
Fast, easy-to-use, general-purpose cluster computing framework
for processing large datasets using a simple programming model
51
• • •

Benefits
• Scale
• Fault-tolerance
• Abstracts distributed computing
• Hides the messy details of writing distributed applications
• Allows developers to just focus on the data processing logic
• Same code works on a laptop or a cluster of servers
• Ease-of-use
• Speed
• Flexibility
52

Easy To Use
• Library with an expressive API
• Scala, Java, Python, R
• 80+ operators
• Interactive development
• spark-shell
• notebooks
53

• Batch processing
• Interactive analytics
• Stream analysis
• Machine learning
• Graph analytics
Integrated Libraries For a Variety of DP Tasks
Spark Core
Spark
SQL
GraphX
Spark
Streaming
MLlib

Unified Platform for a Variety of Data Processing
• Solve a variety of problems with a single toolkit
• No need to learn different tools for each use case
• Avoid code and data duplication
• Achieve operational simplicity

Why is Spark Fast
• Advanced job execution engine
• Allows applications to cache data in memory
56

Advanced Job Execution Engine
• Directed Acyclic Graph (DAG) of stages
• simple job can contain just one stage
• complex job can contain many stages
• eliminates expensive operations between multiple jobs
• synchronization
• serialization/deserialization
• disk I/O
• Lazy operator evaluation
• Pipelined operations
57

Why Caching Makes Applications Run Faster
60
100 MB/s
500 MB/s
10 GB/s

Read Latency Comparison
61
0
50
100
150
200
1 TB
Time (Min)
Data Read
HDD
SSD
RAM

Primary Data Abstractions
• Resilient Distributed Dataset (RDD)
• Dataset

Resilient Distributed Dataset (RDD)
• Represents data as an immutable partitioned collection of
elements, which can be processed in parallel
• Provides the methods / operations for processing data
• Conceptually similar to a Scala collection, except for two things:
• represents a partitioned dataset, which may be distributed over a
cluster
• operations are lazily evaluated

Dataset
• Introduced in Spark 1.6.0
• Represents data as an immutable, partitioned, strongly-typed
collection of elements, which can be processed in parallel
• Benefits from Spark SQL optimizations
• performance optimizations
• memory optimizations (cached dataset uses 5x less space)
• Provides compile-time type-safety similar to the RDD API

Dataset Uses a Specialized Encoder
• Highly optimized
• Runtime code generation for serialization and deserialization
• Significantly faster than Java or Kryo serialization
• Encoded data can be significantly smaller (up to 2x)
• Serialized data is in the Tungsten binary format
• many operations can be done in-place without deserializing
• Built-in support for automatically generating encoders for
• primitive types (e.g. String, Integer, Long)
• Scala case classes
• Java Beans

Relationship Between Dataset and DataFrame
• DataFrame = Dataset[Row]
• Dataset is a partitioned collection of strongly-typed elements
• DataFrame is a partitioned collection of generic Row objects
• DataFrame can be transformed into a Dataset by calling
df.as[ElementType]
• Dataset can be transformed into a DataFrame by calling
df.toDF()

High-level Architecture
67
Master
Driver
Program
Executor
Task
Worker
Executor
Task
Worker
Executor
Task
Worker
Cache
Cache
Cache

Ideal Applications
• Complex data processing
• multi-step pipeline
• Iterative algorithm
• Machine Learning
• Graph analytics
• Ad hoc analysis
• Interactive

Spark Does Not Provide Storage
• Works with a variety of data sources
• No need to import data into Spark
• Scale compute and storage cluster independently

Process Data From a Variety Of Data Sources
And Many More

Spark Does Not Replace Hadoop
71
= =

Hadoop is Optional
72
= =

Big Data
Hadoop
Spark Core
Spark SQL
Spark Streaming
Machine Learning
GraphX

Spark SQL
• Spark library for processing structured data
• Provides more than just a SQL interface to Spark
• Increases developer productivity
• Makes applications run faster
• Uniform API for processing data from a variety of sources
Spark core
Spark SQL

Increases Productivity with Higher-level API
• SQL
• Declarative language
• HiveQL
• SQL-like
• Hive not required
• Language integrated queries
• DataFrame
• Scala, Java, Python and R

Makes Applications Run Faster
• Query optimization
• Python, Scala, Java, and SQL code all use the same optimizer
• Reduced disk I/O
• skip partitions
• skip columns (columnar storage)
• skip rows (using statistical information)
• in-memory columnar caching
• Predicate pushdown
• push operations to a database / data source
• Code generation
• generate Java bytecode

Uniform Data Access
Spark SQL

Primary Data Abstraction is DataFrame
• Represents a partitioned collection organized into named
columns
• Equivalent to a table in a relation database
• Inspired by DataFrames in Python / R
• Provides methods / operations for language integrated queries
in Scala, Java, Python, and R
• Constructed from structured data files, Hive tables, RDBMS,
NoSQL datastores, or RDDs

• Distributed SQL query engine
• JDBC / ODBC support
• BI and visualization app
• Tableau, Zoomdata, Qlik
• Run SQL / HiveQL queries
• No Scala, Java, R, or Python
BI / Visualization App
JDBC / ODBC
Spark SQL
HDFS /
S3
NoSQL RDBMS
Thrift JDBC/ODBC server

Big Data
Hadoop
Spark Core
Spark SQL
Spark Streaming
Machine Learning
GraphX

Spark Streaming
• Extends Spark for data stream processing
• Higher-level operators for streams analytics
• Integrates with other Spark libraries
• Spark SQL
• MLlib
• GraphX
• High throughput
Spark Core
Spark
Streaming

• Data stream split into micro-batches
• Batch interval specified by application
• Stream processed as a sequence of
micro-batch jobs
• Each batch is processed as an RDD
by Spark core
• Results of RDD operations are
returned in batches
High-level Architecture
Spark
Streaming
Spark Core
Input stream
Result
stream
Batches of
x seconds

Primary Abstraction is DStream
• Represents a data stream
• Implemented as a sequence of RDDs
• Provides operators for processing streams
• Basic operators similar to RDD operators
• Window operations
• Stateful stream processing

Spark Streaming Sources And Destinations
85

Big Data
Hadoop
Spark Core
Spark SQL
Spark Streaming
Machine Learning
GraphX

Machine Learning Applications
• Regression
• Home price valuation
• Classification
• Image classification
• Clustering
• Customer segmentation
• Anomaly detection
• Fraud
• Recommendation
• Movie
• Dimensionality reduction
87

What is Machine Learning
• Program software to learn from data
• Logic is not explicitly programmed
• Infer patterns in a dataset
• Generalize beyond the training set
• Mathematics provides the foundation
88

Broad Categorization of ML Algorithms
• Supervised
• Labeled dataset
• Examples: Linear Regression, SVM, Neural Networks
• Unsupervised
• Unlabeled dataset
• Example: K-Means
89

MLlib
Machine Learning with Spark
Spark Core
spark.mllib spark.ml

spark.mllib
• Extends Spark for ML and statistical analysis
• Provides higher-level API than Spark Core for ML
• Dataset represented by an RDD
91

spark.ml
• Another machine learning library that runs on top of Spark
• Less mature than spark.mllib
• Dataset represented by a DataFrame
• Higher-level abstraction
• Feature engineering
• Model training, tuning and evaluation
• Makes it easy to create ML pipelines / workflows
93

MLlib Supports Machine Learning Workflow
• Statistical analysis
• Feature extraction
• Feature transformation
• Dimensionality reduction
• Machine learning algorithms
• Model tuning
• Model evaluation
95

Statistical Analysis
• Summary statistics
• Correlations
• Stratified sampling
• Hypothesis testing
• Random data generation
96

• Feature Extraction and
Transformation
• TF-IDF
• Word2Vec
• StandardScaler
• Normalizer
• ChiSqSelector
• ElementwiseProduct
• PCA
• Dimensionality Reduction
• Singular value decomposition
(SVD)
• Principal component analysis
(PCA)
Feature Engineering
97

• Regression and Classification
• Linear regression, Isotonic
regression
• Logistic regression, SVM
• Naive Bayes
• Random Forest, Gradient-
Boosted Trees
• Multilayer perceptron
(Feedforward NN)
• Recommendation
• Collaborative Filtering (ALS)
• Clustering
• K-means
• Gaussian mixture
• Power iteration clustering (PIC)
• Latent Dirichlet allocation
(LDA)
• Bisecting k-means
• Streaming k-means
• Frequent pattern mining
• FP-growth
• Association rules
• PrefixSpan
Machine Learning Algorithms
98

Big Data
Hadoop
Spark Core
Spark SQL
Spark Streaming
Machine Learning
GraphX

Graph Intro
• Composed of vertices and edges
• Graphs are ubiquitous
• Web pages
• Social networks
• Transportation hubs
• Graph theory provides efficient algorithms for graph-oriented
data
100

• Extends Spark for analyzing large-scale
graph-oriented data
• Provides higher-level abstractions than
Spark core for graph analytics
• Unifies collections and graphs as first-
class composable objects
GraphX
101
Spark core
GraphX

• Fundamental graph operators
• filter
• reverse
• subgraph
• joinVertices
• mapVertices
• mapEdges
• aggregateMessages
• Advanced graph operators
• pregel
• pageRank
• connectedComponents
• stronglyConnectedComponents
• triangleCount
Primary Data Abstraction is Graph
102

Introduction to big data and apache spark

Recommended

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to Introduction to big data and apache spark (20)

Recently uploaded (20)

Introduction to big data and apache spark