A.A.

04-05 Datawarehousing & Datamining 1

Data Warehousing
and
Data Mining
A.A. 04-05 Datawarehousing & Datamining 2
Outline
1. Introduction and Terminology
2. Data Warehousing
3. Data Mining
Association rules
Sequential patterns
Classification
Clustering
A.A. 04-05 Datawarehousing & Datamining 3
Introduction and Terminology
Evolution of database technology
File processing (60s)
Relational DBMS (70s)
Advanced data models e.g.,
Object-Oriented,
application-oriented (80s)
Web-Based Repositories (90s)
Data Warehousing and
Data Mining (90s)
Global/Integrated Information Systems (2000s)
A.A. 04-05 Datawarehousing & Datamining 4
Introduction and Terminology
Major types of information systems within an organization
TRANSACTION
PROCESSING
SYSTEMS
Enterprise Resource Planning (ERP)
Customer Relationship Management (CRM)
KNOWLEDGE
LEVEL
SYSTEMS
Office automation, Groupware, Content
Distribution systems, Workflow
mangement systems
MANAGEMENT
LEVEL
SYSTEMS
Decision Support Systems (DSS)
Management Information Sys. (MIS)
EXECUTIVE
SUPPORT
SYSTEMS
Few
sophisticated
users
Many un-
sophisticated
users
A.A. 04-05 Datawarehousing & Datamining 5
Introduction and Terminology
Transaction processing systems:
Support the operational level of the organization, possibly
integrating needs of different functional areas (ERP);
Perform and record the daily transactions necessary to the
conduct of the business
Execute simple read/update operations on traditional
databases, aiming at maximizing transaction throughput
Their activity is described as:
OLTP (On-Line Transaction Processing)
A.A. 04-05 Datawarehousing & Datamining 6
Introduction and Terminology
Knowledge level systems: provide digital support for managing
documents (office automation), user cooperation and
communication (groupware), storing and retrieving
information (content distribution), automation of business
procedures (workflow management)
Management level systems: support planning, controlling and
semi-structured decision making at management level by
providing reports and analyses of current and historical data
Executive support systems: support unstructured decision
making at the strategic level of the organization
A.A. 04-05 Datawarehousing & Datamining 7
Introduction and Terminology
Decision
Making
Data Presentation
Reporting/Visualization engines
Data Analysis
OLAP, Data Mining engines
Data Warehouses / Data Marts
Data Sources
Transactional DB, ERP, CRM, Legacy systems
Multi-Tier architecture for
Management level and
Executive support systems
PRESENTATION
BUSINESS
LOGIC
DATA
A.A. 04-05 Datawarehousing & Datamining 8
Introduction and Terminology
OLAP (On-Line Analytical Processing):
Reporting based on (multidimensional) data analysis
Read-only access on repositories of moderate-large size
(typically, data warehouses), aiming at maximizing
response time
Data Mining:
Discovery of novel, implicit patterns from, possibly
heterogeneous, data sources
Use a mix of sophisticated statistical and high-
performance computing techniques
A.A. 04-05 Datawarehousing & Datamining 9
Outline
1. Introduction and Terminology
2. Data Warehousing
3. Data Mining
Association rules
Sequential patterns
Classification
Clustering
A.A. 04-05 Datawarehousing & Datamining 10
Data Warehousing
DATA WAREHOUSE
Database with the following distinctive characteristics:
Separate from operational databases
Subject oriented: provides a simple, concise view on one or
more selected areas, in support of the decision process
Constructed by integrating multiple, heterogeneous data
sources
Contains historical data: spans a much longer time horizon
than operational databases
(Mostly) Read-Only access: periodic, infrequent updates
A.A. 04-05 Datawarehousing & Datamining 11
Data Warehousing
Types of Data Warehouses
Enterprise Warehouse: covers all areas of interest for an
organization
Data Mart: covers a subset of corporate-wide data that is
of interest for a specific user group (e.g., marketing).
Virtual Warehouse: offers a set of views constructed on
demand on operational databases. Some of the views could
be materialized (precomputed)
A.A. 04-05 Datawarehousing & Datamining 12
Data Warehousing
Multi-Tier Architecture
DB DB
Data Warehouse
Server
Analysis
Reporting
Data Mining
Data sources Data Storage
OLAP
engine
Front-End
Tools
Cleaning
extraction
A.A. 04-05 Datawarehousing & Datamining 13
Data Warehousing
Multidimensional (logical) Model
Data are organized around one or more FACT TABLEs. Each
Fact Table collects a set of omogeneous events (facts)
characterized by dimensions and dependent attributes
Example: Sales at a chain of stores
100
30
Units
9000 2qtr St3 S1 P2
1500 1qtr St1 S1 P1
Sales Period Store Supplier Product
dimensions dependent attributes
A.A. 04-05 Datawarehousing & Datamining 14
Data Warehousing
Multidimensional (logical) Model (contd)
Each dimension can in turn consist of a number of attributes.
In this case the value in the fact table is a foreign key referring
to an appropriate dimension table
Address
Name
Code
supplier
Description
Code
product
Address
Manager
Name
Code
Store
Units
Store
Period
Sales
Supplier
Product
Fact table
dimension
table
dimension
table
dimension
table
STAR
SCHEMA
A.A. 04-05 Datawarehousing & Datamining 15
Data Warehousing
Conceptual Star Schema (E-R Model)
FACTs
PRODUCT
(1,1) (0,N) (0,N)
(0,N)
(1,1)
(1,1)
STORE
SUPPLIER
PERIOD
(1,1)
(0,N)
A.A. 04-05 Datawarehousing & Datamining 16
Data Warehousing
OLAP Server Architectures
They are classified based on the underlying storage layouts
ROLAP (Relational OLAP): uses relational DBMS to store
and manage warehouse data (i.e., table-oriented
organization), and specific middleware to support OLAP
queries.
MOLAP (Multidimensional OLAP): uses array-based data
structures and pre-computed aggregated data. It shows higher
performance than OLAP but may not scale well if not properly
implemented
HOLAP (Hybird OLAP): ROLAP approach for low-level raw
data, MOLAP approach for higher-level data (aggregations).
A.A. 04-05 Datawarehousing & Datamining 17
Data Warehousing
MOLAP Approach
Period
P
r
o
d
u
c
t
R
e
g
i
o
n
CD
TV
DVD
PC
1Qtr 2Qtr 3Qtr 4Qtr
Europe
North America
Middle East
Far East
DATA CUBE
representing a
Fact Table
Total sales of PCs in
europe in the 4th
quarter of the year
A.A. 04-05 Datawarehousing & Datamining 18
Data Warehousing
OLAP Operations: SLICE
Fix values for one or
more dimensions
E.g. Product = CD
A.A. 04-05 Datawarehousing & Datamining 19
Data Warehousing
OLAP Operations: DICE
Fix ranges for one or
more dimensions
E.g.
Product = CD or DVD
Period = 1Qrt or 2Qrt
Region = Europe or
North America
A.A. 04-05 Datawarehousing & Datamining 20
Data Warehousing
OLAP Operations: Roll-Up
Period
P
r
o
d
u
c
t
Region
CD
TV
DVD
PC
year
Europe
North America
Middle East
Far East
Aggregate data by grouping
along one (or more)
dimensions
E.g.: group quarters
Drill-Down = (Roll-Up)
-1
A.A. 04-05 Datawarehousing & Datamining 21
Data Warehousing
Cube Operator: summaries for each subset of dimensions
North America
Middle East
CD
TV
DVD
PC
1Qtr 2Qtr 3Qtr 4Qtr
Europe
Far East
SUM
SUM
SUM
Yearly sales of
electronics in the
middle east
Yearly sales of PCs
in the middle east
A.A. 04-05 Datawarehousing & Datamining 22
Data Warehousing
Cube Operator: it is equivalent to computing the following
lattice of cuboids
FACT TABLE
product, period product, region period, region
C
product period region
product, period, region
A.A. 04-05 Datawarehousing & Datamining 23
Data Warehousing
Cube Operator In SQL:
SELECT Product, Period, Region, SUM(Total Sales)
FROM FACT-TABLE
GROUP BY Product, Period, Region
WITH CUBE
A.A. 04-05 Datawarehousing & Datamining 24
Data Warehousing
* ALL ALL
* ALL ALL ALL
* Europe ALL ALL
*
* ALL 1qtr ALL
*
* ALL ALL CD
* ALL
* Europe ALL CD
* ALL
* Europe 1qtr ALL
* ALL
* ALL 1qtr CD
*
* Europe 1qtr CD
Tot. Sales Region Period Product
All combinations of Product,
Period and Region
All combinations of Product
and Period
All combinations of Product
A.A. 04-05 Datawarehousing & Datamining 25
Data Warehousing
Roll-up (= partial cube) Operator In SQL:
SELECT Product, Period, Region, SUM(Total Sales)
FROM FACT-TABLE
GROUP BY Product, Period, Region
WITH ROLL-UP
A.A. 04-05 Datawarehousing & Datamining 26
Data Warehousing
* ALL ALL ALL
* ALL ALL
* ALL ALL CD
* ALL
* ALL 1qtr CD
*
* Europe 1qtr CD
Tot. Sales Region Period Product
All combinations of Product,
Period and Region
All combinations of Product
and Period
All combinations of Product
Reduces the complexity from exponential to linear in the
number of dimensions
A.A. 04-05 Datawarehousing & Datamining 27
Data Warehousing
it is equivalent to computing the following subset of the lattice
of cuboids
product, period product, region period, region
C
product period region
product, period, region
A.A. 04-05 Datawarehousing & Datamining 28
Outline
1. Introduction and Terminology
2. Data Warehousing
3. Data Mining
Association rules
Sequential patterns
Classification
Clustering
A.A. 04-05 Datawarehousing & Datamining 29
Data Mining
Data Explosion: tremendous amount of data accumulated in
digital repositories around the world (e.g., databases, data
warehouses, web, etc.)
Production of digital data /Year:
3-5 Exabytes (10
18
bytes) in 2002
30% increase per year (99-02)
We are drowning in data, but starving for knowledge
See:
www.sims.berkeley.edu/how-much-info
A.A. 04-05 Datawarehousing & Datamining 30
Data Mining
Knowledge Discovery in Databases
Data Warehouse
DB DB
Information repositories
Training
data
KNOWLEDGE
Data cleaning
& integration
Selection &
transformation
Data Mining
Evaluation &
presentation
Domain
knowledge
A.A. 04-05 Datawarehousing & Datamining 31
Data Mining
Typologies of input data:
Unaggregated data (e.g., records, transactions)
Aggregated data (e.g., summaries)
Spatial, geographic data
Data from time-series databases
Text, video, audio, web data
A.A. 04-05 Datawarehousing & Datamining 32
Data Mining
DATA MINING
Process of discovering interesting patterns or
knowledge from a (typically) large amount of data
stored either in databases, data warehouses, or
other information repositories
Alternative names: knowledge discovery/extraction,
information harvesting, business intelligence
In fact, data mining is a step of the more general process
of knowledge discovery in databases (KDD)
Interesting: non-trivial, implicit, previously unknown,
potentially useful
A.A. 04-05 Datawarehousing & Datamining 33
Data Mining
Interestingness measures
Purpose: filter irrelevant patterns to convey concise and
useful knowledge. Certain data mining tasks can produce
thousands or millions of patterns most of which are
redundant, trivial, irrelevant.
Objective measures: based on statistics and structure of
patterns (e.g., frequency counts)
Subjective measures: based on users belief about the
data. Patterns may become interesting if they confirm or
contradict a users hypothesis, depending on the context.
Interestingness measures can employed both after and during
the pattern discovery. In the latter case, they improve the
search efficiency
A.A. 04-05 Datawarehousing & Datamining 34
Data Mining
Data Mining
Algorithms
Multidisciplinarity of Data Mining
Statistics
High-
Performance
Computing
Machine
learning
DB
Technology
Visualization
A.A. 04-05 Datawarehousing & Datamining 35
Data Mining
Data Mining Problems
Association Rules: discovery of rules X Y (objects that
satisfy condition X are also likely to satisfy condition Y). The
problem first found application in market basket or transaction
data analysis, where objects are transactions and
conditions are containment of certain itemsets
A.A. 04-05 Datawarehousing & Datamining 36
Association Rules
Statement of the Problem
I =Set of items
D =Set of transactions: t e D then t _ I
For an itemset X _ I,
support(X) = fraction or number of transactions containing X
ASSOCIATION RULE: X-Y Y, with Y c X _ I
support = support(X)
confidence = support(X)/support(X-Y)
PROBLEM: find all association rules with support > than min
sup and confidence > than min confidence
A.A. 04-05 Datawarehousing & Datamining 37
Association Rules
Market Basket Analysis:
Items = products sold in a store or chain of stores
Transactions = customers shopping baskets
Rule X-Y Y = customers who buy items in X-Y are likely to
buy items in Y
Of diapers and beer ..
Analysis of customers behaviour in a supermarket chain has
revealed that males who on thursdays and saturdays buy
diapers are likely to buy also beer
. Thats why these two items are found close to each
other in most stores
A.A. 04-05 Datawarehousing & Datamining 38
Association Rules
Applications:
Cross/Up-selling (especially in e-comm., e.g., Amazon)
Cross-selling: push complemetary products
Up-selling: push similar products
Catalog design
Store layout (e.g., diapers and beer close to each other)
Financial forecast
Medical diagnosis
A.A. 04-05 Datawarehousing & Datamining 39
Association Rules
Def.: Frequent itemset = itemset with support > min sup
General Strategy to discover all association rules:
1. Find all frequent itemsets
2. frequent itemset X, output all rules X-Y Y, with YcX,
which satisfy the min confindence constraint
Observation:
Min sup and min confidence are objective measures of
interestingness. Their proper setting, however, requires
users domain knowledge. Low values may yield exponentially
(in |I|) many rules, high values may cut off interesting rules
A.A. 04-05 Datawarehousing & Datamining 40
Association Rules
Example
For rule {A} {C} :
support = support({A,C}) = 50%
confidence = support({A,C})/support({A}) = 66.6%
For rule {C} {A} (same support as {A} {C}):
confidence = support({A,C})/support({C}) = 100%
Min. sup 50%
Min. confidence 50%
B, E, F 40
A, D 30
A, C 20
A, B, C 10
Items bought Transaction-id
50% {A, C}
50% {C}
50% {B}
75% {A}
Support Frequent Itemsets
A.A. 04-05 Datawarehousing & Datamining 41
Association Rules
Dealing with Large Outputs
Observation: depending on the values of min sup and on
the dataset, the number frequent itemsets can be
exponentially large but may contain a lot of redundancy
Goal: determine a subset of frequent itemsets of
considerably smaller size, which provides the same
information content, i.e., from which the complete set of
frequent itemsets can be derived without further information.
A.A. 04-05 Datawarehousing & Datamining 42
Association Rules
Notion of Closedness
I (items), D (transactions), min sup (support threshold)
Closed Frequent Itemsets =
{X_I : supp(X)>min sup & supp(Y)<supp(X) I_YX}
Its a subset of all frequent itemsets
For every frequent itemset X there exists a closed
frequent itemset Y_X such that supp(Y)=supp(X), i.e., Y
and X occur in exactly the same transcations
All frequent itemsets and their frequencies can be derived
from the closed frequent itemsets without further
information
A.A. 04-05 Datawarehousing & Datamining 43
Association Rules
Notion of Maximality
Maximal Frequent Itemsets =
{X_I : supp(X)>min sup & supp(Y)<min sup I_YX}
Its a subset of the closed frequent itemsets, hence of all
frequent itemsets
For every (closed) frequent itemset X there exists a
maximal frequent itemset Y_X
All frequent itemsets can be derived from the maximal
frequent itemsets without further information, however
their frequencies must be determined from D
information loss
A.A. 04-05 Datawarehousing & Datamining 44
Association Rules
B, L, G, H 40
B, A, D, F 50
B, A, D, F, L, G 60
B, A, D, F, L, H 30
B, L 20
B, A, D, F, G, H 10
Items Tid
YES
YES
YES
YES
NO
Maximal
3
3
4
4
6
Support
10, 40, 60 B, G
10, 30, 40 B, H
20, 30, 40, 60 B, L
10, 30, 50, 60 B, A, D, F
all B
Supporting
Transactions
Closed Frequent
Itemsets
Min sup = 3
DATASET
Example of closed and
maximal frequent itemsets
A.A. 04-05 Datawarehousing & Datamining 45
Association Rules
(B,6) (A,4) (D,4) (F,4) (L,4) (G,3) (H,3)
(B,4) (B,4) (A,4) (B,4) (A,4) (D,4) (B,4) (B,3) (B,3)
(B,4) (B,4) (B,4) (A,4)
(B,4)
(,6)
= closed & maximal
= closed but not maximal
(All frequent) VS (closed frequent) VS (maximal frequent)
A.A. 04-05 Datawarehousing & Datamining 46
Sequential Patterns
Data Mining Problems
Sequential Patterns: discovery of frequent subsequences in
a collection of sequences (sequence database), each
representing a set of events occurring at subsequent times.
The ordering of the events in the subsequences is relevant.
A.A. 04-05 Datawarehousing & Datamining 47
Sequential Patterns
Sequential Patterns
PROBLEM: Given a set of sequences, find the complete set
of frequent subsequences
sequence database
A sequence: <(ef)(ab)(df)(c)(b)>
An element may contain a set of items.
Items within an element are unordered
and are listed alphabetically.
<(a)(bc)(d)(c)>
is a subsequence of
<( <(a)(abc)(ac)(d)(cf)>
For min sup = 2, <(ab)(c)> is a frequent subsequence
<(e)(g)(af)(c)(b)(c)> 40
<(ef)(ab)(df)(c)(b)> 30
<(ad)(c)(bc)(ae)> 20
<(a)(abc)(ac)(d)(cf)> 10
sequence SID
A.A. 04-05 Datawarehousing & Datamining 48
Sequential Patterns
Applications:
Marketing
Natural disaster forecast
Analysis of web log data
DNA analysis
A.A. 04-05 Datawarehousing & Datamining 49
Classification
Data Mining Problems
Classification/Regression: discovery of a model or function
that maps objects into predefined classes (classification) or
into suitable values (regression). The model/function is
computed on a training set (supervised learning)
A.A. 04-05 Datawarehousing & Datamining 50
Classification
Statement of the Problem
Training Set: T = {t1, , tn} set of n examples
Each example ti
characterized by m features (ti(A
1
), , ti(A
m
))
belongs to one of k classes (C
i
: 1 s i s k)
GOAL
From the training data find a model to describe the classes
accurately and synthetically using the datas features. The
model will then be used to assign class labels to unknown
(previously unseen) records
A.A. 04-05 Datawarehousing & Datamining 51
Classification
Applications:
Classification of (potential) customers for: Credit approval,
risk prediction, selective marketing
Performance prediction based on selected indicators
Medical diagnosis based on symptoms or reactions to
therapy
A.A. 04-05 Datawarehousing & Datamining 52
Classification
Classification process
Training Set
Test Data
Unseen Data
VALIDATE
BUILD
PREDICT
class labels
model
A.A. 04-05 Datawarehousing & Datamining 53
Classification
Observations:
Features can be either categorical if belonging to
unordered domains (e.g., Car Type), or continuous, if
belonging to ordered domains (e.g., Age)
The class could be regarded as an additional attribute of
the examples, which we want to predict
Classification vs Regression:
Classification: builds models for categorical classes
Regression: builds models for continuous classes
Several types of models exist: decision trees, neural
networks, bayesan (statistical) classifiers.
A.A. 04-05 Datawarehousing & Datamining 54
Classification
Classification using decision trees
Definition: Decision Tree for a training set T
Labeled tree
Each internal node v represents a test on a feature. The
edges from v to its children are labeled with mutually
exclusive results of the test
Each leaf w represents the subset of examples of T
whose features values are consistent with the test
results found along the path from the root to w. The
leaf is labeled with the majority class of the examples it
contains
A.A. 04-05 Datawarehousing & Datamining 55
Classification
Example:
examples = car insurance applicants, class = insurance risk
high family 20
low truck 32
low family 68
high sports 43
high sports 18
high family 23
Risk Car Type Age
features class
Age < 25
Car type e{sports}
High
High
Low
Model
(decision tree)
A.A. 04-05 Datawarehousing & Datamining 56
Clustering
Data Mining Problems
Clustering: grouping objects into classes with the
objective of maximizing intra-class similarity and
minimizing inter-class similarity (unsupervised learning)
A.A. 04-05 Datawarehousing & Datamining 57
Clustering
Statement of the Problem
GIVEN: N objects, each characterized by p attributes
(a.k.a. variables)
GROUP: the objects into K clusters featuring
High intra-cluster similarity
Low inter-cluster similarity
Remark: Clustering is an instance of unsupervised learning or
learning by observations, as opposed to supervised learning
or learning by examples (classification)
A.A. 04-05 Datawarehousing & Datamining 58
Clustering
Example
outlier
attribute X
attribute Y
attribute X
attribute Y
object
A.A. 04-05 Datawarehousing & Datamining 59
Clustering
Several types of clustering problems exist depending
on the specific input/output requirements, and on the
notion of similarity:
The number of clusters K may be provided in input or not
As output, for each cluster one may want a representative
object, or a set of aggregate measurements, or the
complete set of objects belonging to the cluster
Distance-based clustering: similarity of objects is related to
some kind of geometric distance
Conceptual clustering: a group of objects forms a cluster if
they define a certain concept
A.A. 04-05 Datawarehousing & Datamining 60
Clustering
Applications:
Marketing: identify groups of customers based on their
purchasing patterns
Biology: categorize genes with similar functionalities
Image processing: clustering of pixels
Web: clustering of documents in meta-search engines
GIS: identification of areas of similar land use
A.A. 04-05 Datawarehousing & Datamining 61
Clustering
Challenges:
Scalability: strategies to deal with very large datasets
Variety of attribute types: defining a good notion of
similarity is hard in the presence of different types of
attributes and/or different scales of values
Variety of cluster shapes: common distance measures
provide only spherical clusters
Noisy data: outliers may affect the quality of the clustering
Sensitivity to input ordering: the ordering of the input data
should not affect the output (or its quality)
A.A. 04-05 Datawarehousing & Datamining 62
Clustering
Main Distance-Based Clustering Methods
Partitioning Methods: create an initial partition of the objects
into K clusters, and refine the clustering using iterative
relocation techniques. A cluster is represented either by the
mean value of its component objects or by a centrally located
component object (medoid)
Hierarchical Methods: start with all objects belonging to
distinct clusters and then successively merge the pair of
closest clusters/objects into one single cluster (agglomerative
approach); or start with all objects belonging to one cluster
and successively split up a cluster into smaller ones (divisive
approach)