Storage, Files & Indexing

(R&G Chapters 8, 9)

1
 The BIG picture
Relational Relational
Want to Model Algebra, SQL
store data
Want to
Result access data
Conceptual Logical
Design Design
Query Optimization
and Execution
ER to
ER Relational Relational Operators
Models
Schema Access Methods
Refinement Buffer Management
Disk Space
Management

Indexing Physical
Design Database Storage, Files
Storage 2
 File and access layer
Database as a "file of records"
• Create/delete files
• insert/delete/modify record
• retrieve one particular record (point access)
– specified using record id
• retrieve range of records (range access)
– satisfying some conditions
• retrieve all records (scan)

3
 File and access methods layer
• File Organization
– How is data organized in files?
• Indexing
– How to make data access efficient?
• Storage
– How data is physically stored on disk?

4
 File Organization & Indexing
• File Organization

• Heap & Sorted Files

• Indexing

• Meta-data
– System Catalog

5
 File Organization
Page
Students
sid name login age gpa
50000 Dave dave@cs 19 3.3
Fields
53666 Jones jones@cs 18 3.4
53688 Smith smit@ee 18 3.2
… … … … …
Records
…

Files
…

… File
Storage Pages

6
Page Format (N-ary Storage Model)
• Page = collection of slots
• Each slot stores one record
– Record identifier: <page_id, slot_number>
– Option 2: <uniq> -> <page_id, slot_number>
• Page format should support
– Fast searching, insertion, deletion
• Page format depends on record format
– Fixed-Length
– Variable-Length

7
 Record Formats: Fixed-Length
F1 F2 F3 F4

L1 L2 L3 L4

Base address (B) Address = B+L1+L2

• Schema is stored in system catalog

– Number of fields is fixed for all records of a table
– Domain is fixed for all records of a table
• Each field has fixed length
• Finding ith field is done via arithmetic.
8
Page Format: Fixed-Length Records

•Record id = <page id, slot #>

•In the packed case, moving records for free space
management changes rid; maybe unacceptable.

9
 Record Formats: Variable-Length
F1 F2 F3 F4
$ $ $ $
Fields Delimited by Special Symbols
F1 F2 F3 F4

Array of Field Offsets

• Array of field offsets is typically superior
– Direct access to fields
– Clean way of handling NULL values
– Oracle 8: length—data pairs, DB2: Array of offsets
10
Page Format: Variable-Length Records
Data

Slot Array

• Need to move records in a page

• Allocation/deletion must find/release free space
• Maintain slot directory with <record offset, record length> pairs
• Records can move on page without changing rid
• Useful for freely moving fixed-length records (e.g., sorting)
11
 Variable-Length Records: Issues
• If a field grows and no longer fits?
– shift all subsequent fields
• If record no longer fits in page?
– Move a record to another page after modification
• What if record size > page size?
– SQL Server record size = 8KB
– DB2 record size = page size

12
Alternate Page Formats:
Column store

13
 Page Format for Column Store
Decomposition Storage Model (DSM)

Decompose a relational table to sub-tables per attribute

ü Saves IO by bringing only the relevant attributes
14
 Alternate Page Formats:
Partition Attributes Across (PAX)
Decompose a slotted-page
internally in mini-pages
per attribute
ü Cache-friendly
ü Compatible with
slotted-pages
ü Brings only relevant
attributes to cache

15
 File Organization & Indexing
• File Organization

• Heap & Sorted Files

• Indexing

• Meta-data
– System Catalog

16
 Alternative File Organizations
Many alternatives exist, each good for some
situations, and not so good in others:
•Heap files: Suitable when typical access is a file
scan retrieving all records.
•Sorted Files: Best for retrieval in some order,
or for retrieving a range of records.
•Index File Organizations: (will cover shortly..)

17
 Heap (Unordered) Files
• Simplest file structure
– contains records in no particular order
– Need to be able to scan, search based on rid
• As file grows and shrinks, disk pages are
allocated and de-allocated.
– Need to manage free space
 Heap File Implemented Using Lists
Data Data Data
… Full Pages
Page Page Page
Header
Page
Data Data Data
… Pages with
Page Page Page
Free Space

• <Heap file name, header page id> stored somewhere

• Each page contains 2 pointers plus data.
• Manage free pages using free list
– What if most pages have some space?
 Heap File Using a Page Directory
Data
Header Page 1
Page
Data
Page 2

…
Data
DIRECTORY Page N
• The directory is a collection of pages
– linked list implementation is just one alternative.
• The entry for a page can include the number of free
bytes on the page.
– Much smaller than linked list of all HF pages!
 Heap File vs. Sorted File
• Which is better?
– Let us design a cost model to find out
• Simplified cost model:
– Based only on IO cost
• IO is the dominating cost
• Ignore CPU and other overheads
• Ignore effect of prefetching and sequential access
– Consider only average case

21
 More Assumptions…
• Single record insert and delete.
• Equality search - exactly one match (e.g.,
search on key)
– Question: what if more or fewer?
• Heap Files:
– Insert always appends to end of file.
• Sorted Files:
– Files compacted after deletions.
– Search done on file-ordering attribute.
 Cost of Operations (in # of I/O’s)
B: Number of data pages
Heap File
Heap File Sorted File File
Sorted notes…
notes…
Scanall
Scan all B B
B B
records
records
Equality
Equality 0.5B log2 B assumes
assumesexactly
exactly
0.5B log2 B oneone
match!
Search
Search match!
Range
Range B (log22 B)
(log B) ++ (#match
B
Search
Search (#match pages)
pages)
Insert 2 (log 2B)B)++ 2*(B/2)
must R & W
Insert 2 (log 2 2*(B/2) must R & W

Delete
Delete 0.5B + 1 (log2B) + 2*(B/2) must R & W
 File Organization & Indexing
• File Organization

• Heap & Sorted Files

• Indexing

• Meta-data
– System Catalog

24
 Indexing
Employee
SSN Name Age Salary
Index Index
• Display employees in increasing on SSN on Salary
order of age
• age>30 111 John 28 100K
• Updates? … … 28 ..
• Increasing order of salary? … … 29 ..
… … 30 ..
• Salary>120K?
… … 31 …
• Indexing … … 34 ..
– Multiple efficient access paths
… … … ...

Sorted File on Age

25
 Indexing
•Sometimes, we want to retrieve records by
specifying the values in one or more fields, e.g.,
–Find all students in the "CS" department
–Find all students with a gpa > 3
•An index on a file speeds up selections on the
search key fields for the index.
–The search key is NOT necessarily a key (e.g., no need to
be unique).
•Index and keys
–Any subset of the fields in a relation can be the search
key for an index on the relation.
–Can have multiple indices on any number of fields
26
 Index Search Conditions
• Search condition =
<search key, comparison operator>
Examples…
(1) Condition: Department = "CS"
– Search key: Department
– Comparison operator: equality (=)
(2) Condition: GPA > 3
– Search key: GPA
– Comparison operator: greater-than (>)
 Example: Simple Index on GPA

2 2.5 3 3.5

Data entries:
1.2* 1.7* 1.8* 1.9* 2.2* 2.4* 2.7* 2.7* 2.9* 3.2* 3.3* 3.3* 3.6* 3.8* 3.9* 4.0*

(Index File)
(Data file)

Data Pages containing Data Records

An index contains a collection of data entries, and supports

efficient retrieval of records matching a given search condition.
 Data Entry Representation
How are data entries represented in an index?
• i.e., what is at the bottom of the index?
Assume a data entry k*
There are 3 alternative representations:
1) Actual data record (with key value k)
2) <k, rid of matching data record>
3) <k, list of rids of matching data records>

29
 Alternative 1
Actual data record (with key value k)
• The index structure becomes the file organization.
– Similar to heap files or sorted files.
• At most one index can use Alt. 1.
• Efficient but can be expensive to maintain.
– Insertions and deletion modify the data file.
Alternative 1: Example

31
 Alternatives 2 & 3
<k, rid of matching data record>
<k, list of rids of matching data records>
• Easier to maintain than Alt. 1.
• If a file has several indexes, only one can use Alt. 1;
the others must use Alt. 2 or 3.
• Alt. 3 is more compact than Alt. 2, but leads to
variable sized data entries.
– even for search keys of fixed length.
• Data entries with long rid lists may span many pages!
Alternative 2: Example

33
 File Organization & Indexing
• File Organization
• Index Classification
• Clustered/Unclustered
• Heap & Sorted Files • Sparse/Dense
• Types of Indexes
• Primary
• Indexing • Clustering
• Secondary Key
• Secondary Non-Key
• Meta-data • Indexing techniques
– System Catalog • Hash vs tree
• Choosing search key

34
 Index Classification
Some of the material for this
topic is taken from "Database
Systems", Elmasri, Navathe.

• Clustered vs Unclustered
• Dense vs Sparse
• Indexing field
– Key
2
– Non-Key
2X2=4
• Physical ordering of the file
– Ordered on indexing field
2
– Not ordered on indexing field
Physical Ordering on Indexing Field
Ordered Not Ordered
Indexing Key Primary Index Secondary Index (Key)
Field Non Key Clustering Index Secondary Index (Non Key)
35
 File Organization & Indexing
• File Organization
• Index Classification
• Clustered/Unclustered
• Heap & Sorted Files • Sparse/Dense
• Types of Indexes
• Primary
• Indexing • Clustering
• Secondary Key
• Secondary Non-Key
• Meta-data • Indexing Technique
– System Catalog • Tree vs Hash
• Choosing search key

36
 Index Classification - Clustering
• Clustered vs. unclustered: If the order of the
data records is similar to the order of the index
data entries, then the index is clustered.

Index entries
CLUSTERED direct search for UNCLUSTERED
data entries

Data Data
entries (Index File) entries
(Data file)

Data Records Data Records

Clustered vs unclustered: Example

Alt 1. Clustered Alt 2. Unclustered

Qn: Can we have Alt 1. unclustered?

Qn: Can we have Alt 2. clustered? 38
 Clustered vs. Unclustered Index
Assuming Alt. 2 for data entries and data records
stored in a Heap file:
• To build clustered index, first sort the Heap file.
– Keeping some free space on each page for future inserts.
• Overflow pages may be needed for inserts.
à The order of data records is similar—but not identical to—the sort order.
Index entries
CLUSTERED direct search for UNCLUSTERED
data entries

Data Data
entries (Index File) entries
(Data file)

Data Records Data Records

 Clustered vs. Unclustered Index

• Cost of retrieving records found in range scan:

– Clustered: cost = # pages in file w/matching records
– Unclustered: cost ≈ # of matching index data entries
• What are the tradeoffs?
– Clustered Pros:
• Efficient for range searches
– Clustered Cons:
• Expensive to maintain (on the fly or sloppy with
reorganization)
 Dense vs. Sparse Index
Sparse Dense
• Dense: at least one Index Data File
Index
entry per key value on Name
Ashby, 25, 3000
On Age
– Alt. 1 è dense index. Basu, 33, 4003
22
25
Bristow, 30, 2007
• Sparse: an entry per Ashby
30
33

data page in file

Cass Cass, 50, 5004
Smith Daniels, 22, 6003
Jones, 40, 6003 40
– Always clustered! 44
44
– Smaller than a dense index; Smith, 44, 3000
Tracy, 44, 5004
50

but some optimizations are

based on dense indexes
 File Organization & Indexing
• File Organization
• Index Classification
• Clustered/Unclustered
• Heap & Sorted Files • Sparse/Dense
• Types of Indexes
• Primary
• Indexing • Clustering
• Secondary Key
• Secondary Non-Key
• Meta-data • Indexing Technique
– System Catalog • Tree vs Hash
• Choosing search key

42
 Primary Index
• Indexing Field = Key
• File is physically sorted
on indexing field
• One index entry per
block
• Index pointers can be
block pointers (anchors)
• Sparse Index

43
 Clustering Index
• Indexing Field = Non-Key
• File is physically sorted
on indexing field
• One index entry per
distinct value
• Index pointer is block
pointer to first block
with the value
• Sparse Index

44
Clustering Index - Alternative

45
 Secondary Key Index
• Indexing Field = Key
• File is NOT physically
sorted on indexing field
• One index entry per
record
• Index pointer is record
pointer
• Dense Index

46
 Secondary Non-Key Index
• Indexing Field = Non-Key
• File is NOT physically
sorted on indexing field
• One index entry per
record
• Index pointer
‒ One per record (dense)
‒ One per value (sparse)
o Variable-length record
o Extra level of indirection

47
 Index Classification: Summary
File physically
Indexing Index Sparse or
Type of Index sorted on Index Pointers
Field Entries Dense?
indexing field?
One per
Primary Key Yes Block anchor Sparse
block
One per
Clustering Non-Key Yes Block pointer Sparse
value
One per
Secondary Key Key No Record pointer Dense
record
One per Record pointer/
Secondary Sparse or
Non-Key No record/ Variable length/
Non-Key Dense
value indirection

48
 File Organization & Indexing
• File Organization
• Index Classification
• Clustered/Unclustered
• Heap & Sorted Files • Sparse/Dense
• Types of Indexes
• Primary
• Indexing • Clustering
• Secondary Key
• Secondary Non-Key
• Meta-data • Indexing Technique
– System Catalog • Tree vs Hash
• Choosing search keys

49
 Hash-based Index
• Good for equality selections.
– File = a collection of buckets. Bucket = primary page plus 0 or
more overflow pages.
– Hash function h: h(r.search_key) = bucket for record r.

50
 Tree-based Index
• Good for range selections.
– Leaves contain data entries sorted by search key value
– B+ tree: all root->leaf paths have equal length (height)

51
 Index, index, everywhere
Select E.dno
From Employees E
Where E.age > 40

• Would you build a b+tree index or hash index?

– Hint: it’s not an equality query
• Would you build a clustered index?
• When would a B+tree be suboptimal?
– Hint: think about selectivity
 File Organization & Indexing
• File Organization
• Index Classification
• Clustered/Unclustered
• Heap & Sorted Files • Sparse/Dense
• Types of Indexes
• Primary
• Indexing • Clustering
• Secondary Key
• Secondary Non-Key
• Meta-data • Indexing Technique
– System Catalog • Tree vs Hash
• Choosing search key

53
 Choosing Search Key
Select sID
From Student
Where sName = ‘Mary’ And GPA > 3.9

• Can build index on sname

– What index would this be? Hash or tree?
• Can build index on GPA
– What index would this be? Hash or tree?
• How about <sname, gpa> together?
 Composite Search Keys
• Search on field combination. Examples of
– Equality query: Every field value composite key indexes using
is equal to a constant. E.g. wrt lexicographic order
<sal,age> index:
11,80 11
• age=12 and sal =75 12

ag e
12,10

m
e
l
– Range query: Some field value is

sa
na
12,20 12
not a constant. E.g.: 13,75 bob 12 10 13
• age =12; or age=12 and sal > 20 <age, sal> cal 11 80 <age>
joe 12 20
• Data entries in index sorted 10,12 sue 13 75 10
by search key for range 20,12 20
Data records
queries. 75,13 75
80,11 sorted by name 80
– Lexicographic order.
<sal, age> <sal>
Composite Search Key - Tradeoffs
Select AVG(E.sal)
From Employees E
Where E.age = 25
AND E.sal BETWEEN 3000 AND 5000

• Do we build index on <age,sal>, <sal,age>?

• What index would this be? Hash or tree?
• Do we really need data file?
– “index-only evaluation” is possible
 File Organization & Indexing
• File Organization

• Heap & Sorted Files

• Indexing

• Meta-data
– System Catalog

57
 System Catalogs
• For each relation:
– name, file name, file structure (e.g., Heap file)
– attribute name and type, for each attribute
– index name, for each index
– integrity constraints
• For each index:
– structure (e.g., B+ tree) and search key fields
• For each view:
– view name and definition
• Plus stats, authorization, buffer pool size, etc.
Catalogs are themselves stored as relations!
 System Catalog in Oracle
> desc user_tables
Name Null? Type
----------------------------------------- -------- ----------------------------
TABLE_NAME NOT NULL VARCHAR2(128)
TABLESPACE_NAME VARCHAR2(30)
CLUSTER_NAME VARCHAR2(128)
IOT_NAME VARCHAR2(128)
STATUS VARCHAR2(8)
PCT_FREE NUMBER
PCT_USED NUMBER
...

> desc user_tab_columns

Name Null? Type
----------------------------------------- -------- ----------------------------
OWNER NOT NULL VARCHAR2(128)
TABLE_NAME NOT NULL VARCHAR2(128)
COLUMN_NAME NOT NULL VARCHAR2(128)
DATA_TYPE VARCHAR2(128)
DATA_TYPE_MOD VARCHAR2(3)
DATA_TYPE_OWNER VARCHAR2(128)
DATA_LENGTH NOT NULL NUMBER
DATA_PRECISION NUMBER
DATA_SCALE NUMBER
NULLABLE VARCHAR2(1)
...

59
 System Catalog in Oracle
> select * from student;

SID NAME AGE

---------- -------------------- ----------
A1234 John 23
B1 Xavier 24
A21341 Scott 21
C12948291 Benjamin 25
1948 Ben 29

> select table_name, num_rows, blocks, avg_row_len from user_tables;

TABLE_NAME NUM_ROWS BLOCKS AVG_ROW_LEN

-------------------- ---------- ---------- -----------
STUDENT 5 5 20

> select table_name, column_name, data_type, data_length from user_tab_columns;

TABLE_NAME COLUMN_NAME DATA_TYPE DATA_LENGTH

---------- ----------- ---------- -----------
STUDENT SID CHAR 10
STUDENT NAME VARCHAR2 32
STUDENT AGE NUMBER 22

60
 Summary
• Database organized as a collection of files
– Several file organizations (heap, sorted, …) with tradeoffs
• Files are a collection of pages
– Several page layouts (NSM, DSM, …) with tradeoffs
• Pages contain a collection of records
– Several record formats (fixed, variable length…) with
tradeoffs
• Index is a quick way to find records
– Several index types with tradeoffs

One size does not fit all!