Introduction to Big Data

& Basic Data Analysis

Big Data EveryWhere!
Lots of data is being collected
and warehoused
Web data, e-commerce
purchases at department/
grocery stores
Bank/Credit Card
transactions
Social Network
How much data?
Google processes 20 PB a day (2008)
Wayback Machine has 3 PB + 100 TB/month (3/2009)
Facebook has 2.5 PB of user data + 15 TB/day (4/2009)
eBay has 6.5 PB of user data + 50 TB/day (5/2009)
CERNs Large Hydron Collider (LHC) generates 15 PB a
year




640K ought to be
enough for anybody.
Maximilien Brice, CERN
The Earthscope
The Earthscope is the world's
largest science project. Designed to
track North America's geological
evolution, this observatory records
data over 3.8 million square miles,
amassing 67 terabytes of data. It
analyzes seismic slips in the San
Andreas fault, sure, but also the
plume of magma underneath
Yellowstone and much, much more.
(http://www.msnbc.msn.com/id/44
363598/ns/technology_and_science
-future_of_technology/#.TmetOdQ-
-uI)

1.

Type of Data
Relational Data (Tables/Transaction/Legacy
Data)
Text Data (Web)
Semi-structured Data (XML)
Graph Data
Social Network, Semantic Web (RDF),

Streaming Data
You can only scan the data once



What to do with these data?
Aggregation and Statistics
Data warehouse and OLAP
Indexing, Searching, and Querying
Keyword based search
Pattern matching (XML/RDF)
Knowledge discovery
Data Mining
Statistical Modeling


Statistics 101
Random Sample and Statistics
Population: is used to refer to the set or universe of all entities
under study.
However, looking at the entire population may not be
feasible, or may be too expensive.
Instead, we draw a random sample from the population, and
compute appropriate statistics from the sample, that give
estimates of the corresponding population parameters of
interest.
Statistic
Let Si denote the random variable corresponding to
data point xi , then a statistic is a function : (S1,
S2, , Sn) R.

If we use the value of a statistic to estimate a
population parameter, this value is called a point
estimate of the parameter, and the statistic is called
as an estimator of the parameter.
Empirical Cumulative Distribution Function
Where
Inverse Cumulative Distribution Function
Example
Measures of Central Tendency (Mean)
Population Mean:
Sample Mean (Unbiased, not robust):
Measures of Central Tendency
(Median)
Population Median:
or
Sample Median:
Example
Measures of Dispersion (Range)
Range:
Not robust, sensitive to extreme values
Sample Range:
Measures of Dispersion (Inter-Quartile Range)
Inter-Quartile Range (IQR):
More robust
Sample IQR:
Measures of Dispersion
(Variance and Standard Deviation)
Standard Deviation:
Variance:
Measures of Dispersion
(Variance and Standard Deviation)
Standard Deviation:
Variance:
Sample Variance & Standard Deviation:
Univariate Normal Distribution
Multivariate Normal Distribution
OLAP and Data Mining
Warehouse Architecture
23
Client
Client
Warehouse
Source
Source Source
Query & Analysis
Integration
Metadata
24
Star Schemas
A star schema is a common organization for
data at a warehouse. It consists of:
1. Fact table : a very large accumulation of facts
such as sales.
w Often insert-only.
2. Dimension tables : smaller, generally static
information about the entities involved in the
facts.
Terms
Fact table
Dimension tables
Measures
25
sale
orderId
date
custId
prodId
storeId
qty
amt
customer
custId
name
address
city
product
prodId
name
price
store
storeId
city
Star
26
customer custId name address city
53 joe 10 main sfo
81 fred 12 main sfo
111 sally 80 willow la
product prodId name price
p1 bolt 10
p2 nut 5
store storeId city
c1 nyc
c2 sfo
c3 la
sale oderId date custId prodId storeId qty amt
o100 1/7/97 53 p1 c1 1 12
o102 2/7/97 53 p2 c1 2 11
105 3/8/97 111 p1 c3 5 50
Cube
27
sale prodId storeId amt
p1 c1 12
p2 c1 11
p1 c3 50
p2 c2 8
c1 c2 c3
p1 12 50
p2 11 8
Fact table view:
Multi-dimensional cube:
dimensions = 2
3-D Cube
28
sale prodId storeId date amt
p1 c1 1 12
p2 c1 1 11
p1 c3 1 50
p2 c2 1 8
p1 c1 2 44
p1 c2 2 4
day 2
c1 c2 c3
p1 44 4
p2
c1 c2 c3
p1 12 50
p2 11 8
day 1
dimensions = 3
Multi-dimensional cube: Fact table view:
ROLAP vs. MOLAP
ROLAP:
Relational On-Line Analytical Processing
MOLAP:
Multi-Dimensional On-Line Analytical
Processing
29
Aggregates
30
sale prodId storeId date amt
p1 c1 1 12
p2 c1 1 11
p1 c3 1 50
p2 c2 1 8
p1 c1 2 44
p1 c2 2 4
Add up amounts for day 1
In SQL: SELECT sum(amt) FROM SALE
WHERE date = 1
81
Aggregates
31
sale prodId storeId date amt
p1 c1 1 12
p2 c1 1 11
p1 c3 1 50
p2 c2 1 8
p1 c1 2 44
p1 c2 2 4
Add up amounts by day
In SQL: SELECT date, sum(amt) FROM SALE
GROUP BY date
ans date sum
1 81
2 48
Another Example
32
sale prodId storeId date amt
p1 c1 1 12
p2 c1 1 11
p1 c3 1 50
p2 c2 1 8
p1 c1 2 44
p1 c2 2 4
Add up amounts by day, product
In SQL: SELECT date, sum(amt) FROM SALE
GROUP BY date, prodId
sale prodId date amt
p1 1 62
p2 1 19
p1 2 48
drill-down
rollup
Aggregates
Operators: sum, count, max, min,
median, ave
Having clause
Using dimension hierarchy
average by region (within store)
maximum by month (within date)
33
What is Data Mining?
Discovery of useful, possibly unexpected,
patterns in data
Non-trivial extraction of implicit, previously
unknown and potentially useful information
from data
Exploration & analysis, by automatic or
semi-automatic means, of large quantities of
data in order to discover meaningful patterns

Data Mining Tasks
Classification [Predictive]
Clustering [Descriptive]
Association Rule Discovery [Descriptive]
Sequential Pattern Discovery [Descriptive]
Regression [Predictive]
Deviation Detection [Predictive]
Collaborative Filter [Predictive]

Classification: Definition
Given a collection of records (training set )
Each record contains a set of attributes, one of the
attributes is the class.
Find a model for class attribute as a function
of the values of other attributes.
Goal: previously unseen records should be
assigned a class as accurately as possible.
A test set is used to determine the accuracy of the
model. Usually, the given data set is divided into
training and test sets, with training set used to
build the model and test set used to validate it.
Decision Trees
37
sale custId car age city newCar
c1 taurus 27 sf yes
c2 van 35 la yes
c3 van 40 sf yes
c4 taurus 22 sf yes
c5 merc 50 la no
c6 taurus 25 la no
Example:
Conducted survey to see what customers were
interested in new model car
Want to select customers for advertising campaign
training
set
Clustering
38
age
income
education
K-Means Clustering
39
Association Rule Mining
40
tran1 cust33 p2, p5, p8
tran2 cust45 p5, p8, p11
tran3 cust12 p1, p9
tran4 cust40 p5, p8, p11
tran5 cust12 p2, p9
tran6 cust12 p9
sales
records:
Trend: Products p5, p8 often bough together
Trend: Customer 12 likes product p9
market-basket
data
Association Rule Discovery
Marketing and Sales Promotion:
Let the rule discovered be
{Bagels, } --> {Potato Chips}
Potato Chips as consequent => Can be used to
determine what should be done to boost its sales.
Bagels in the antecedent => can be used to see which
products would be affected if the store discontinues
selling bagels.
Bagels in antecedent and Potato chips in consequent
=> Can be used to see what products should be sold
with Bagels to promote sale of Potato chips!
Supermarket shelf management.
Inventory Managemnt
Collaborative Filtering
Goal: predict what movies/books/ a person may be interested in,
on the basis of
Past preferences of the person
Other people with similar past preferences
The preferences of such people for a new movie/book/
One approach based on repeated clustering
Cluster people on the basis of preferences for movies
Then cluster movies on the basis of being liked by the same clusters of
people
Again cluster people based on their preferences for (the newly created
clusters of) movies
Repeat above till equilibrium
Above problem is an instance of collaborative filtering, where users
collaborate in the task of filtering information to find information of
interest
42
Other Types of Mining
Text mining: application of data mining to textual
documents
cluster Web pages to find related pages
cluster pages a user has visited to organize their visit
history
classify Web pages automatically into a Web directory
Graph Mining:
Deal with graph data
43
Data Streams
What are Data Streams?
Continuous streams
Huge, Fast, and Changing
Why Data Streams?
The arriving speed of streams and the huge amount of data
are beyond our capability to store them.
Real-time processing
Window Models
Landscape window (Entire Data Stream)
Sliding Window
Damped Window
Mining Data Stream



44
A Simple Problem
Finding frequent items
Given a sequence (x
1
,x
N
) where x
i
[1,m], and a real
number between zero and one.
Looking for x
i
whose frequency >
Nave Algorithm (m counters)
The number of frequent items 1/
Problem: N>>m>>1/
45
P(N) N
KRP algorithm
Karp, et. al (TODS 03)
46
=0.35
1/ = 3
N=30 m=12
N/ (1/) N
Streaming Sample Problem
Scan the dataset once
Sample K records
Each one has equally probability to be sampled
Total N record: K/N