Scribd
Upload
36 views

Distributed Databases Data Warehousing: CPS 216 Advanced Database Systems

Distributed databases allow data to be partitioned across multiple sites for availability, scalability and performance. There are two main approaches for building distributed databases - top-down and bottom-up. Query optimization in distributed databases involves deciding query plans including parallel operations and determining where to execute each operation while considering communication costs. Concurrency control uses techniques like two-phase commit to coordinate transactions across multiple sites. Data warehousing integrates data from multiple sources into a central repository to support analytical queries. Materialized views in data warehouses must be incrementally maintained as the source data changes.

Uploaded by

Rahul Deka
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
36 views

Distributed Databases Data Warehousing: CPS 216 Advanced Database Systems

Distributed databases allow data to be partitioned across multiple sites for availability, scalability and performance. There are two main approaches for building distributed databases - top-down and bottom-up. Query optimization in distributed databases involves deciding query plans including parallel operations and determining where to execute each operation while considering communication costs. Concurrency control uses techniques like two-phase commit to coordinate transactions across multiple sites. Data warehousing integrates data from multiple sources into a central repository to support analytical queries. Materialized views in data warehouses must be incrementally maintained as the source data changes.

Uploaded by

Rahul Deka
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 11

Distributed Databases

Data Warehousing
CPS 216
Advanced Database Systems

Review
Distributed DBMS
Top-down approach
Data partitioning
Query processing
Query optimization
Concurrency control and recovery

Bottom-up approach
Query processing and optimization
2

Optimizing distributed queries


What is different from optimizing centralized queries?
New strategies: parallel joins, semijoins,
Plans have a new property: interesting sites
Communication cost is a big factor besides I/O
Per-message cost, per-byte cost, CPU cost to pack/unpack data

Parallelism: response time versus total resource


consumption
Plan B
70

Plan A
100

40
50
3

Example: two-step optimization


Step 1 (compile time): decide the join order, join
methods, and access paths
Same complexity as in a centralized DBMS

Step 2 (run time): decide where to execute each operator


Can cope with changing load and network characteristics
Can use data that has been dynamically allocated to a site
(caching or replication)
Site 2
Better: Site 2
Step 1: ba
Step 2: ba
ba

ba

ba

ba

Site 1

ba

ba

R1 R2 R3 R4 R1 R2 R3 R4
Site 1

ba

R1 R4 R2 R3

Site 1

Concurrency control & recovery in


Distributed DBMS
Rich and interesting field
We will just sample the field by looking at the
problem of distributed transaction commit
Transaction T
op3

op1; op2

op3; op4

Site 1

Site 2

Site 3

commit

abort

commit

Two-phase commit
Coordinator

Participant

Initial

T.commit
prepare*
Wait

OK*
commit*

Initial

NOK
abort*

prepare
OK
Abort

Ready

commit
-

Commit

Notation:

prepare
NOK
abort
-

Abort

Commit

Incoming message
Outgoing message

* = everyone
6

Key points of 2PC


By sending OK a participant promises the
coordinator to commit
But it can only commit when instructed to do so by
the coordinator
The coordinator could tell it to abort instead

After sending NOK a participant can abort


unilaterally
Coordinator can decide to commit only if all
participants have responded OK
Logging of all messages are required at each site
7

Bottom-up approach to
Building a distributed DBMS
Data already in various sources
Build a distributed DBMS to provide global,
uniform access to all data
How to integrate data?
How to deal with heterogeneous and autonomous
sources?
Mediation approach
8

Wrapper/mediator architecture
Client

Client

Mediator

Catalog

Wrapper

Wrapper

Wrapper

Database

Database

Database

Mediator

Client

Client

Mediator
Catalog
Accept queries from clients
Rewrite and optimize queries
Wrapper
Wrapper
Wrapper
Send subplans to be executed
by wrappers
Database
Database
Database
Combine results from
wrappers and perform any additional local processing
necessary
Mediator catalog stores global schema and external
schema of sources as exported by wrappers
No source-specific code in a mediator!
10

Wrapper
Hide heterogeneity away
from mediator
Translate mediator requests
so that they are understood
by sources

Client

Client

Mediator

Catalog

Wrapper

Wrapper

Wrapper

Database

Database

Database

Example: SELECT * FROM


Books WHERE title LIKE %Databases; a form-based
search request for books with title matching *Databases

Translate results returned by a source so that they are


compliant with its external schema
Example: result HTML page Books tuples
11

Query optimization with wrappers


Basic questions
Capability: What types of subplans can be
handled by a wrapper?
How do we enumerate valid plans?

Cost: What is the cost of executing a subplan by a


wrapper?
How do we pick the optimal plan?
12

Example: Garlic query optimization


Haas et al., VLDB 1997
Incorporated in DB2
Rules for generating valid plans
Supplied by wrappers and mediator
Plugged into the optimizer

Plans have interesting properties

Order (as in Selinger)


Site (where the output is produced)
Columns (in the output)
Predicates (that have been applied)
Cost, etc.

13

Example rules for a DBMS source


wrap_access(table, columns, predicates) =
SCANDBMS(table, columns, predicates)
Condition: table is at my site
I can handle any projection and selection (by converting them
to a single-table SELECT-FROM-WHERE SQL statement)

wrap_join(subplan1, subplan2, predicates) =


JOINDBMS(subplan1, subplan2, predicates)
Condition: subplan1.site = subplan2.site = my site
I can handle any local join (by converting it to a multi-table
SELECT-FROM-WHERE SQL statement)
14

Example rules for a Web source


wrap_access(table, columns, predicates) =
FETCHWeb(Books, title LIKE string)
Condition: table = Books, (title LIKE string) predicates
I can search books by title (with wildcards); no projection

wrap_access(table, columns, predicates) =


FETCHWeb(Books, author = string)
Condition: table = Books, (author = string) predicates
I can search books by exact author names; no projection
I cannot search books by title and author at the same time

No wrap_join rule
I cannot handle process joins

15

Example rules for the mediator


med_pushdown(subplan) = RECEIVE(SEND(subplan))
Condition: subplan.site mediator

med_pushdown(subplan) = subplan
Condition: subplan.site = mediator

med_access(table, columns, predicates) =


plan wrap_access(table, columns, predicates):
FILTERmed(med_pushdown(plan)),
predicates plan.predicates)
I can get the result of a single-table scan from a wrapper and
then evaluate remaining selection predicates
16

More rules for the mediator


med_join(subplan1, subplan2, predicates) =
plan wrap_join(subplan1, subplan2, predicates):
med_pushdown(plan)
Condition: subplan1.site = subplan2.site mediator
I can push down a join to a wrapper

med_join(subplan1, subplan2, predicates) =


JOINmed(med_pushdown(subplan1),
med_pushdown(subplan2), predicates)
I also can handle a join locally

And more

17

Plan enumeration
Call all wrap_access and med_access rules to generate
single-table access plans
Repeatedly call all wrap_join and med_join rules to
generate multi-table join plans
Example final plans
FILTERmed(
RECEIVE(SEND(FETCHWeb(Books, title LIKE string))),
author = string), versus
FILTERmed(
RECEIVE(SEND(FETCHWeb(Books, author = string))),
title LIKE string)
RECEIVE(SEND(JOINDBMS(R, S))), versus
JOINmed(RECEIVE(SEND(R)), RECEIVE(SEND(S)))

18

Costing
Wrapper-supplied cost model
Lots of work for wrapper developers

Calibration
Define a generic cost model with parameters for all
wrappers
Example: cost = c (# of tuples)

Run test queries to measure the parameters for each


wrapper

Learning curve
Use recent statistics to adjust cost estimates
Example: cost = average over last three runs

19

Summary of wrapper/mediator
Not all sources are created equal!
Whats in a source?
Wrapper: source schema external schema
Mediator: external schema global schema

What can it do?


Wrappers and mediators supply rules describing their
query processing capabilities

How much does it cost?


Wrappers supply cost model, or
Mediator calibrates or learns the cost model
20

Data warehousing
Data resides in many distributed, heterogeneous
OLTP (On-Line Transaction Processing) sources
Sales, inventory, customer,
NC branch, NY branch, CA branch,

Need to support OLAP (On-Line Analytical


Processing) over an integrated view of the data
Store the integrated data at a central repository
called the data warehouse
21

OLTP versus OLAP


OLTP

OLAP

Mostly updates
Short, simple transactions
Clerical users
Goal: ACID, transaction
throughput

Mostly reads
Long, complex queries
Analysts, decision makers
Goal: fast queries

22

Warehousing versus mediation


Warehousing
Eager integration
In advance: before queries
Answer could be stale

Copy data from sources


Need to maintain
consistency
Query processing is local
to the warehouse

Mediation
Lazy integration
On demand: at query time
Answer is more up-to-date

Leave data at sources


No need to maintain
consistency
Sources participate in
query processing

Faster
Can operate when sources
are unavailable

23

Maintaining a data warehouse


Buzz word: the ETL process
Extraction: extract relevant data and/or changes
from sources
Transformation: transform data to match the
warehouse schema
Loading: integrate data/changes into the
warehouse
Can still use a wrapper/mediator architecture

24

Warehouse data = materialized views


If the transformation process can be captured by
SQL, then warehouse data can be seen as a view
CREATE VIEW warehouse_table AS
SELECT
FROM source_table1, source_table2,
WHERE ;

Except the view is materialized


That is, the result is stored
And needs to be maintained when source data changes
25

Maintaining materialized views


Vold = Q(Rold, )
Change detected: Rnew Rold R R
What is Vnew?
Recomputation: Vnew Q(Rnew, )

Done periodically, e.g., every night


What if there is no night, e.g., an international organization?
What if recomputation takes longer than a day?

Incremental maintenance
Compute only the changes to V: V and V
Apply the changes to Vold: Vnew Vold V V
Potentially much faster if changes are small
26

Incremental maintenance
Example: V = p R
Change: Rnew Rold R
Vnew = p Rnew = p (Rold R) = p Rold p R
= Vold V

Change: Rnew Rold R


Vnew = p Rnew = p (Rold R) = p Rold p R
= Vold V
Change propagation equations
27

Change propagation
More change propagation equations
(R R)
S =
(R
S) (R
S)
(R R)
S =
(R
S) (R
S)

Repeatedly apply change propagation equations


to factor out changes
(pr (R R))
prs ps S =
(pr R pr R)
prs ps S =
(pr R
prs ps S) (pr R
prs ps S)
28

Self-maintainability
A warehouse is self-maintainable if it can be
maintained without accessing the sources
Self-maintainable: V = p R
Not self-maintainable: V = R
S
R and R need to be joined with S
S and S need to be joined with R
Problem: need to query the source for maintenance
What if the source/network is slow?
What if the source/network is down?
What if the source has been updated again?

29

Making warehouse self-maintainable


Add auxiliary views
Example: Order
O.OID = L.OID AND O.month = nov AND L.product =
book Lineitem
Nave approach: add base tables O and L
Better approach: push selections down and then add
selection views month = nov O and product = book L
Use constraints
The join is a foreign-key join (L.OID references O.OID), so
only month = nov O is needed
If we only insert matching orders and lineitems together, then
no auxiliary view is needed
30

10

Next time
Warehouse design
Data cube
ROLAP versus MOLAP

31

11

You might also like