Chapter 3: Data Warehousing and

OLAP Technology: An Overview

 What is a data warehouse?

 Data warehouse architecture

 From data warehousing to data mining

Data Mining: Concepts and

January 29, 2025 Techniques 1
 What is Data Warehouse?
 Defined in many different ways, but not rigorously.
 A decision support database that is maintained
separately from the organization’s operational database
 Support information processing by providing a solid
platform of consolidated, historical data for analysis.
 “A data warehouse is a subject-oriented, integrated, time-
variant, and nonvolatile collection of data in support of
management’s decision-making process.”—W. H. Inmon
 Data warehousing:
 The process of constructing and using data warehouses

Data Mining: Concepts and

January 29, 2025 Techniques 2
 Data Warehouse—Subject-Oriented

 Organized around major subjects, such as

customer, product, sales
 Focusing on the modeling and analysis of data for
decision makers, not on daily operations or
transaction processing
 Provide a simple and concise view around
particular subject issues by excluding data that
are not useful in the decision support process

Data Mining: Concepts and

January 29, 2025 Techniques 3
 Data Warehouse—Integrated
 Constructed by integrating multiple,
heterogeneous data sources
 relational databases, flat files, on-line

transaction records
 Data cleaning and data integration techniques
are applied.
 Ensure consistency in naming conventions,

encoding structures, attribute measures, etc.

among different data sources

E.g., Hotel price: currency, tax, breakfast covered,
etc.
 When data is moved to the warehouse, it is
converted.
Data Mining: Concepts and
January 29, 2025 Techniques 4
 Data Warehouse—Time Variant
 The time horizon for the data warehouse is
significantly longer than that of operational systems
 Operational database: current value data
 Data warehouse data: provide information from a
historical perspective (e.g., past 5-10 years)
 Every key structure in the data warehouse
 Contains an element of time, explicitly or
implicitly
 But the key of operational data may or may not
contain “time element”
Data Mining: Concepts and
January 29, 2025 Techniques 5
 Data Warehouse—Nonvolatile
 A physically separate store of data transformed
from the operational environment
 Operational update of data does not occur in the
data warehouse environment
 Does not require transaction processing,
recovery, and concurrency control mechanisms
 Requires only two operations in data
accessing:

initial loading of data and access of data

Data Mining: Concepts and

January 29, 2025 Techniques 6
 Data Warehouse vs. Heterogeneous
DBMS
 Traditional heterogeneous DB integration: A query driven approach
 Build wrappers/mediators on top of heterogeneous databases
 When a query is posed to a client site, a meta-dictionary is used
to translate the query into queries appropriate for individual
heterogeneous sites involved, and the results are integrated into
a global answer set

Complex information filtering, compete for resources
 Data warehouse: update-driven, high performance

Information from heterogeneous sources is integrated in
advance and stored in warehouses for direct query and analysis

Data Mining: Concepts and

January 29, 2025 Techniques 7
 Data Warehouse vs. Operational
DBMS
 OLTP (on-line transaction processing)
 Major task of traditional relational DBMS
 Day-to-day operations: purchasing, inventory, banking,
manufacturing, payroll, registration, accounting, etc.
 OLAP (on-line analytical processing)
 Major task of data warehouse system
 Data analysis and decision making
 Distinct features (OLTP vs. OLAP):
 User and system orientation: customer vs. market
 Data contents: current, detailed vs. historical, consolidated
 Database design: ER + application vs. star + subject
 View: current, local vs. evolutionary, integrated
 Access patterns: update vs. read-only but complex queries
Data Mining: Concepts and
January 29, 2025 Techniques 8
 OLTP vs. OLAP
OLTP OLAP
users clerk, IT professional knowledge worker
function day to day operations decision support
DB design application-oriented subject-oriented
data current, up-to-date historical,
detailed, flat relational summarized, multidimensional
isolated integrated, consolidated
usage repetitive ad-hoc
access read/write lots of scans
index/hash on prim. key
unit of work short, simple transaction complex query
# records accessed tens millions
#users thousands hundreds
DB size 100MB-GB 100GB-TB
metric transaction throughput query throughput, response

Data Mining: Concepts and

January 29, 2025 Techniques 9
 Why Separate Data Warehouse?
 High performance for both systems
 DBMS— tuned for OLTP: access methods, indexing,
concurrency control, recovery
 Warehouse—tuned for OLAP: complex OLAP queries,
multidimensional view, consolidation
 Different functions and different data:
 missing data: Decision support requires historical data which
operational DBs do not typically maintain
 data consolidation: DS requires consolidation (aggregation,
summarization) of data from heterogeneous sources
 data quality: different sources typically use inconsistent data
representations, codes and formats which have to be
reconciled
 Note: There are more and more systems which perform OLAP
analysis directly on relational databases
Data Mining: Concepts and
January 29, 2025 Techniques 10
 Chapter 3: Data Warehousing and
OLAP Technology: An Overview

 What is a data warehouse?

 Data warehouse architecture

 From data warehousing to data mining

Data Mining: Concepts and

January 29, 2025 Techniques 11
 Design of Data Warehouse: A
Business Analysis Framework
 Four views regarding the design of a data
warehouse

Top-down view

allows selection of the relevant information necessary
for the data warehouse

Data source view

exposes the information being captured, stored, and
managed by operational systems

Data warehouse view

consists of fact tables and dimension tables

Business query view

sees the perspectives of data in the warehouse from
the view of end-user
Data Mining: Concepts and
January 29, 2025 Techniques 12
 Data Warehouse Design
Process
 Top-down, bottom-up approaches or a combination of both

Top-down: Starts with overall design and planning (mature)

Bottom-up: Starts with experiments and prototypes (rapid)
 From software engineering point of view

Waterfall: structured and systematic analysis at each step before
proceeding to the next

Spiral: rapid generation of increasingly functional systems, short
turn around time, quick turn around
 Typical data warehouse design process

Choose a business process to model, e.g., orders, invoices, etc.

Choose the grain (atomic level of data) of the business process

Choose the dimensions that will apply to each fact table record

Choose the measure that will populate each fact table record

Data Mining: Concepts and

January 29, 2025 Techniques 13
 Data Warehouse: A Multi-Tiered Architecture

Monitor
Metadata & OLAP Server
Other
sources Integrator

Analysis
Operational Extract Query
DBs Transform Data Serve Reports
Load
Refresh
Warehouse Data mining

Data Marts

Data Sources Data Storage OLAP Engine Front-End Tools

Data Mining: Concepts and
January 29, 2025 Techniques 14
 Three Data Warehouse Models
 Enterprise warehouse
 collects all of the information about subjects spanning

the entire organization

 Data Mart
 a subset of corporate-wide data that is of value to a

specific groups of users. Its scope is confined to

specific, selected groups, such as marketing data mart

Independent vs. dependent (directly from warehouse) data
mart
 Virtual warehouse
 A set of views over operational databases

 Only some of the possible summary views may be

materialized
Data Mining: Concepts and
January 29, 2025 Techniques 15
 Data Warehouse
Development: A
Recommended Approach
Multi-Tier Data
Warehouse
Distributed
Data Marts

Data Data Enterprise

Mart Mart Data
Warehouse

Model refinement Model refinement

Define a high-level corporate data model

Data Mining: Concepts and
January 29, 2025 Techniques 16
 Data Warehouse Back-End Tools and
Utilities
 Data extraction
 get data from multiple, heterogeneous, and external

sources
 Data cleaning
 detect errors in the data and rectify them when

possible
 Data transformation
 convert data from legacy or host format to

warehouse format
 Load
 sort, summarize, consolidate, compute views, check

integrity, and build indicies and partitions

 Refresh
 propagate the updates from the data sources to the

warehouse Data Mining: Concepts and

January 29, 2025 Techniques 17
 Metadata Repository
 Meta data is the data defining warehouse objects. It stores:
 Description of the structure of the data warehouse

schema, view, dimensions, hierarchies, derived data defn, data
mart locations and contents
 Operational meta-data

data lineage (history of migrated data and transformation path),
currency of data (active, archived, or purged), monitoring
information (warehouse usage statistics, error reports, audit
trails)
 The algorithms used for summarization
 The mapping from operational environment to the data warehouse
 Data related to system performance

warehouse schema, view and derived data definitions
 Business data

business terms and definitions, ownership of data, charging
policies Data Mining: Concepts and
January 29, 2025 Techniques 18
 OLAP Server Architectures
 Relational OLAP (ROLAP)

Use relational or extended-relational DBMS to store and
manage warehouse data and OLAP middle ware

Include optimization of DBMS backend, implementation of
aggregation navigation logic, and additional tools and services

Greater scalability
 Multidimensional OLAP (MOLAP)

Sparse array-based multidimensional storage engine

Fast indexing to pre-computed summarized data
 Hybrid OLAP (HOLAP) (e.g., Microsoft SQLServer)

Flexibility, e.g., low level: relational, high-level: array
 Specialized SQL servers (e.g., Redbricks)

Specialized support for SQL queries over star/snowflake
schemas
Data Mining: Concepts and
January 29, 2025 Techniques 19
 Chapter 3: Data Warehousing and
OLAP Technology: An Overview

 What is a data warehouse?

 Data warehouse architecture

 From data warehousing to data mining

Data Mining: Concepts and

January 29, 2025 Techniques 20
 Data Warehouse Usage
 Three kinds of data warehouse applications

Information processing

supports querying, basic statistical analysis, and
reporting using crosstabs, tables, charts and graphs

Analytical processing

multidimensional analysis of data warehouse data

supports basic OLAP operations, slice-dice, drilling,
pivoting

Data mining

knowledge discovery from hidden patterns

supports associations, constructing analytical models,
performing classification and prediction, and presenting
the mining results using visualization tools
Data Mining: Concepts and
January 29, 2025 Techniques 21
 From On-Line Analytical Processing
(OLAP)
to On Line Analytical Mining (OLAM)
 Why online analytical mining?

High quality of data in data warehouses

DW contains integrated, consistent, cleaned
data

Available information processing structure
surrounding data warehouses

ODBC, OLEDB, Web accessing, service facilities,
reporting and OLAP tools

OLAP-based exploratory data analysis

Mining with drilling, dicing, pivoting, etc.

On-line selection of data mining functions

Integration and swapping of multiple mining
functions, algorithms, and tasks
Data Mining: Concepts and
January 29, 2025 Techniques 22
 An OLAM System Architecture
Mining query Mining result Layer4
User Interface
User GUI API
Layer3
OLAM OLAP
Engine Engine OLAP/OLAM

Data Cube API

Layer2
MDDB
MDDB
Meta
Data
Filtering&Integration Database API Filtering
Layer1
Data cleaning Data
Databases Data
Data Data
integration Warehouse
Mining: Concepts and Repository
January 29, 2025 Techniques 23
 Chapter 3: Data Warehousing and
OLAP Technology: An Overview
 What is a data warehouse?

 A multi-dimensional data model

 Data warehouse architecture

 Data warehouse implementation

 From data warehousing to data mining

 Summary
Data Mining: Concepts and
January 29, 2025 Techniques 24
 Summary: Data Warehouse and OLAP
Technology

 Why data warehousing?

 Data warehouse architecture
 From OLAP to OLAM (on-line analytical mining)

Data Mining: Concepts and

January 29, 2025 Techniques 25
 References (I)
 S. Agarwal, R. Agrawal, P. M. Deshpande, A. Gupta, J. F. Naughton, R.
Ramakrishnan, and S. Sarawagi. On the computation of multidimensional
aggregates. VLDB’96
 D. Agrawal, A. E. Abbadi, A. Singh, and T. Yurek. Efficient view maintenance in
data warehouses. SIGMOD’97
 R. Agrawal, A. Gupta, and S. Sarawagi. Modeling multidimensional databases.
ICDE’97
 S. Chaudhuri and U. Dayal. An overview of data warehousing and OLAP
technology. ACM SIGMOD Record, 26:65-74, 1997
 E. F. Codd, S. B. Codd, and C. T. Salley. Beyond decision support. Computer
World, 27, July 1993.
 J. Gray, et al. Data cube: A relational aggregation operator generalizing group-by,
cross-tab and sub-totals. Data Mining and Knowledge Discovery, 1:29-54, 1997.
 A. Gupta and I. S. Mumick. Materialized Views: Techniques, Implementations, and
Applications. MIT Press, 1999.
 J. Han. Towards on-line analytical mining in large databases. ACM SIGMOD
Record, 27:97-107, 1998.
 V. Harinarayan, A. Rajaraman, and J. D. Ullman. Implementing data cubes
efficiently. SIGMOD’96 Data Mining: Concepts and
January 29, 2025 Techniques 26
 References (II)
 C. Imhoff, N. Galemmo, and J. G. Geiger. Mastering Data Warehouse Design:
Relational and Dimensional Techniques. John Wiley, 2003
 W. H. Inmon. Building the Data Warehouse. John Wiley, 1996
 R. Kimball and M. Ross. The Data Warehouse Toolkit: The Complete Guide to
Dimensional Modeling. 2ed. John Wiley, 2002
 P. O'Neil and D. Quass. Improved query performance with variant indexes.
SIGMOD'97
 Microsoft. OLEDB for OLAP programmer's reference version 1.0. In
http://www.microsoft.com/data/oledb/olap, 1998
 A. Shoshani. OLAP and statistical databases: Similarities and differences. PODS’00.
 S. Sarawagi and M. Stonebraker. Efficient organization of large multidimensional
arrays. ICDE'94
 OLAP council. MDAPI specification version 2.0. In
http://www.olapcouncil.org/research/apily.htm, 1998
 E. Thomsen. OLAP Solutions: Building Multidimensional Information Systems. John
Wiley, 1997
 P. Valduriez. Join indices. ACM Trans. Database Systems, 12:218-246, 1987.
 J. Widom. Research problems in data warehousing. CIKM’95.
Data Mining: Concepts and
January 29, 2025 Techniques 27