Native XML Support in

DB2 Universal Database

Matthias Nicola, Bert van der Linden

IBM Silicon Valley Lab
Presented by Mo Liu , Frate, Joseph and John Russo

Some material in the talk is adapted from

the slides of this paper’s conference talk.
Agenda
 What is DB2 9 (Viper)?
 Native XML in the forthcoming version of
DB2
 Native XML Storage
 XML Schema Support
 XML indexes
 Querying XML data in DB2
 Summery
What is DB2 9 (Viper)?
 IBM DB2 9 is the next-generation hybrid
data server with optimized management of
both XML and relational data.
 IBM extended DB2 to include:
• New storage techniques for efficient
management of hierarchical structures
inherent in XML documents.
• New indexing technology
 New query language support (for XQuery), a new
graphical query builder (for XQuery), and new query
optimization techniques

 New support for validating XML data based on user-

supplied schemas

 New administrative capabilities, including extensions to

key database utilities

 Integration with popular application programming

interfaces (APIs)
XML Databases
 XML-enabled Databases
 The core data model is not XML (but e.g. relational)
 Mapping between XML data model and DB’s data
 model is required, or XML is stored as text
 E.g.: DB2 XML Extender v8

 Native XML Databases

 Use the hierarchical XML data model to store and
process XML internally
 No mapping, no storage as text
 Storage format = processing format
 E.g.: Forthcoming version of DB2
XML in Relational Databases –
Today's Challenge
Today’s Challenge:
XML must be force fit into relational data model – 2 choices
1. Shredding or decomposing
− Mapping from XML to relational often too complex
− Loses hierarchical dependencies
− Loses digital signature
− Often requires dozens or hundreds of tables
− Difficult to change original XML document
2. Large Object (BLOB, CLOB, Varchar)
It allows for fast insert and retrieval of full documents but it needs XML
parsing at query execution time.
− SLOW performance
− Search performance is slow (must parse at search time)
− Retrieval of sub-documents is expensive
− Update inside the document is slow
− Indexing is inefficient (based on relative position)
− Difficult to join with relational
− Costs get worse as document size increases
DB2 Hybrid XML Engine - Overview
Integration of XML & Relational
Capabilities in DB2
 Native XML data type
(not Varchar, not CLOB, not object-
relational)
 XML Capabilities in all DB2 components
 Applications combine XML & relational
data
Integrating XML and Relational in DB2
DB2 Hybrid XML Engine - Interfaces
 Data Definition
create table dept(deptID int, deptdoc xml);
 Insert
insert into dept(deptID, deptdoc) values (?,?)
 Index
create index xmlindex1 on dept(deptdoc)
generate key using xmlpattern ‘/dept/name’ as varchar(30);
 Retrieve
select deptdoc from dept where deptID = ?
 SQL based Query
select deptID, xmlquery('$d/dept/name' passing deptdoc as “d") from dept where
deptID <> “PR27”;
 XQuery based Query
for $book in db2-fn:xmlcolumn('BOOKS')/book
for $entry in db2-fn:xmlcolumn('REVIEWS')/entry
where $book/title = $entry/title
return <review> {$entry/review/text()} </review>;
Native XML Storage
Efficient Document Tree Storage
Information for Every Node
 Tag name, encoded as unique StringID
 A nodeID
 Node kind (e.g. element, attribute, etc.)
 Namespace / Namespace prefix
 Type annotation
 Pointer to parent
 Array of child pointers
 Hints to the kind & name of child nodes
 (for early-out navigation)
 For text/attribute nodes: the data itself
XML Node Storage Layout
XML Storage: “Regions Index”
XML Indexes in DB2
 Need index support to manage millions of
XML documents
 Path-specific value indexes on XML
columns to index frequently used elements
and attributes
 XML-aware full-text indexing
 XML Value Indexes
 Table DEPT has two fields: “id” and “dept_doc”
 Field “dept_doc” is an XML document:

<dept>
<employee id=901>
<name>John Doe</name>
<phone>408 555 1212</phone>
<office>344</office>
</employee>
</dept>

 CREATE INDEX idx1 ON DEPT(deptdoc) GENERATE KEY USING

XMLPATTERN ‘/dept/employee/name’ AS SQL VARCHAR(35)
 Creates XML value index on employee name for all documents
XML Value Indexes (continued)
 “xmlpattern” identifies the XML nodes to be
indexed
 Subset of XPath language
 Wildcards,namespaces allowed
 XPath predicates such as /a/b[c=5] not supported
 “AS SQL” necessary to define data type, since
DB2 does not require single XML schema for all
documents in a table (so DB2 may not know
data type to use for index)
XML Value Indexes: Data Types

 Allowed data types for indexes:

 VARCHAR(n)
 VARCHAR HASHED,
 DOUBLE
 DATE
 TIMESTAMP

 DB2 index manager enhanced to handle special

XML types (e.g., +0, -0, +INF, -INF, NaN)
XML Value Indexes (continued)
 Node does not cast to the index type
 No error is raised
 No index entry created for that node

 Single document (e.g., XML field from

single record) may contain 0, 1, or multiple
index entries
 Different than relational index
XML Value Indexes: unique indexes

 Unique indexes enforced within a

document, and across all documents
 Example of unique index on employee id:
CREATE UNIQUE INDEX idx2 ON
DEPT(deptdoc) GENERATE KEY
USING XMLPATTERN
‘/dept/employee/@id’ AS SQL
DOUBLE
XML Value Indexes: multiple elements
or attributes
 Can create indexes on multiple elements
or attributes
 Example: create index on all text nodes:
CREATE INDEX idx3 ON DEPT(deptdoc)
GENERATE KEY USING XMLPATTERN
‘//text()’ AS SQL VARCHAR(hashed)
 Example: create index on all attributes
CREATE INDEX idx4 ON DEPT(deptdoc)
GENERATE KEY USING XMLPATTERN
‘//@*’ AS SQL DOUBLE
XML Value Indexes: namespaces

 Can index in a particular namespace

 XMLPATTERN can contain namespace
declarations and prefixes
 Example:
CREATE INDEX idx5 ON DEPT(deptdoc)
GENERATE KEY USING XMLPATTERN
‘DECLARE NAMESPACE
m=http://www.me.com/;/m:dept/m:employee/
m:name’ AS SQL VARCHAR(45)
XML Value Indexes: internal
 For each XML document, each unique path
mapped to an integer PathID (like StringID for
tags)
 Each index entry includes:
 PathID to identify path of indexed node
 Value of the node cast to the index type
 RowID
 Identify rows containing the matching documents

 NodeID
 Identify matching nodes and regions within the documents
XML Value Indexes: atomic vs. non-atomic

 Atomic Node:
 if it is an attribute, or
 if it is a text node, or
 if it is an element that has no child elements
and exactly one text node child
 Indexes typically defined for atomic nodes
 Possible to define index on non-atomic
nodes, e.g. index on ‘/dept/employee’
XML Value Indexes: atomic vs. non-atomic

 ‘/dept/employee’ non-atomic since it has

child elements
 Single index entry for all of “employee”
element, on all text nodes under
“employee” (concatenation)
 Can be useful for mixed content in text-
oriented XML, e.g.:
 <title>The benefits of <bold>XML</bold></title>
XML Full Text Indexes
 Allows full-text search of XML columns
 Can be fully indexed or partially indexed
 Example of full index:
CREATE INDEX myIndex FOR TEXT ON DEPT(deptdoc)
FORMAT XML CONNECT TO PERSONNELDB
 Example query:
SELECT deptdoc FROM dept WHERE
CONTAINS(deptdoc,’SECTIONS(“/dept/comment”) “Brazil” ‘)
=1
Internal index structure
System RX: One Part Relational, One Part XML
Kevin Beyer, Roberta J Cochrane,
Vanja Josifovski, Jim Kleewein, George Lapis,
Guy Lohman, Bob Lyle, Fatma Özcan,
Hamid Pirahesh, Normen Seemann,
Tuong Truong, Bert Van der Linden, Brian Vickery,
Chun Zhang
Internal index structure
 XML index implemented with two B+ trees
 Path index
 Value Index
Internal index structure: Path Index
 Path Index maps reverse path (revPath) to a
generated path identifier (pathId)
 A “reverse path” is a list of node labels from leaf
to root
 Compressed into vector of label identifiers
 Analogy to COLUMNS catalog from relational
database
 Used for efficient processing of descendent
queries
 Example: “//name” query
Internal index structure: Value
Index
 Value Index used to represent nodes
 Cconsists of the following key:
 PathId
 value
 nodeId
 rid
Internal index structure: Value
Index
 “value” is representation of the node’s data
value when cast to the index’s data type
 “rid” identifies the row in the table (used for
locking)
 “nodeId” identifies a node within the
 uses a Dewey node identifier
 can provide quick access to a node in the
XML store
 “pathId” to retrieve specific path queries
Internal index structure: Tradeoffs
of Value Index key fields
 Order of keys is a tradeoff
 pathId first allows quick retrieval of specific
queries
 e.g.,index on //name might match many paths
 query on /book/author/name still has consecutive
index entries
 but, query like //name=‘Maggie’ will need to examine
every location in the index per matching path
XML Schema Support
 Optional XML Schema validation
 Insert, Update, Query
 Limited support for DTDs an external
entities
 Type annotation produced by validation
persisted with document (query execution)
 Conforms to XML Query standard, XML
Schema standard, XML standard
XML Schema Support
 Register XML Schemas and DTDs in DB
 DB then stores type-annotated documents
on disk, compiles execution plans with
references to the XML Schemas
 Schemas stored in DB itself, for
performance
 XML Schema Repository (XSR)
XML Schema Support: XSR
 XSR consists of several new database
catalog tables:
 Original XML schema documents for XML
schema
 Binary representation of the schema for fast
reference
XML Schema Support: Registration

 Example:
REGISTER XMLSCHEMA
http://my.dept.com FROM dept.xsd AS
departments.deptschema complete
 Schema URI is http://my.dept.com
 File with schema document is “dept.xsd”
 Schema identifier in DB is “deptschema”
 Belongs to relational DB schema
“departments”
XML Schema Support: Validation

 “XMLVALIDATE” function to validate

documents in SQL statements
 Schema for validation
 isspecified explicitly, or
 can be deduced from the schemaLocation
hints in the instance documents
 Referenced by Schema URI or by
identifier
XML Schema Support: Validation

 Example (explicit by URI):

INSERT INTO DEPT(detpdoc)
VALUES xmlvalidate(?according to
xmlschema uri ‘http://my.dept.com’)
 Example (explicit by ID):
INSERT INTO DEPT(deptdoc)
VALUES xmlvalidate(? according to
xmlschema id
departments.deptschema)
XML Schema Support: Validation

 Example (implcit)
 DB2 tries to deduce schema from input
document
INSERT INTO dept(deptdoc) VALUES
xmlvalidate(?)
 Try to find it in repository
XML Schema Support: First repository
design principle

 Repository will not

 require users to modify a schema before it is
being registered
 require users to modify XML documents
before they are inserted and validated
 Once document is validated in DB,it will
never require updates to remain valid
 Considered infeasible to bulk-update all
existing documents to become valid
XML Schema Support: Second repository
design principle

 Enable schema evolution

 Sequence of changes in an XML schema
over its lifetime
 New or evolving business needs
 How to accomplish schema evolution is
much-debated
 no standards
 business demands require it; so constrain
problem
XML Schema Support: Second repository
design principle

 Flexibility of schema repository

“paramount importance”
 DB2’s schema repository does not require
namespace or the schema URI of each
registered schema to be unique (user
does not have control)
 Database-specific Schema identifier must
be unique (user does have control)
XML Schema Support: Second repository
design principle

 Built-in support for one very simple type of

schema evolution
 If new schema is backwards-cmpatible with old
schema, then old schema can be replaced with
new schema in the schema repository
 DB2 verifies all possible elements and attributes
in old schema have same named types in the
new schema
Querying XML Data in DB2
 Options Supported
 XQuery/XPath as a stand-alone language
 SQL embedded in XQuery
 XQuery/XPath embedded in SQL/XML
 Plain SQL for full-document retrieval
 DB2 treats SQL and XQuery as primary query
languages.
 Bothwill operate independently on their data models
 Can also be integrated
Sample Tables
create table ship (
shipNo varchar(5) primary key not null,
capacity decimal(7,2),
class int,
purchDate date, Notice the
maintenance xml xml datatype
)

create table captain (

captID varchar(5) primary key not null,
lname varchar(20),
fname varchar(20),
DOB date,
contact xml
)
Sample XML Data
Ship.maintenance
<mrecord>
<log>
<mntid>2353</mntid>
<shipno>39</shipno>
<vendorid>2345</vendorid>
<captid>9875</captid>
<maintdate>01/10/2007</maintdate>
<service>Removed rust on hull </service>
<resolution>complete</resolution>
<cost>13450.96</cost>
<nextservice>01/10/2008</nextservice>
</log>
<log>
<mntid>1254</mntid>
<shipno>39</shipno>
<vendorid>1253</vendorid>
<captid>9234</captid>
<maintdate>09/20/2005</maintdate>
<service>Replace rudder</service>
<resolution>complete</resolution>
<cost>34532.21</cost>
<nextservice>NA</nextservice>
</log>
</mrecord>
Sample XML Data
Captain.contactinfo
<contactinfo>
<Address>
<street>234 Rolling Lane</street>
<city>Rockport</city>
<state>MA</state>
<zipcode>01210</zipcode>
</Address>
<phone>
<work>9783412321</work>
<home>9722342134</home>
<cell>9782452343</cell>
<satellite>2023051243</satellite>
</phone>
<email>[email protected]</email>
</contactinfo>
Standalone XQuery in DB2
for $s in db2-fn:xmlcolumn(‘ship.maintenance’)
let $ml:= $s//log Db2-fn:xmlcolumn returns
sequence of all documents
where $ml/cost = > 10000 in the XML column

order by $ml/shipno
return <MaintenanceLog>
{$ml/shipno,$ml}
</MaintenanceLog>
SQL Embedded in XQuery
for $m in db2-fn:sqlquery(‘select maintenance from ship where class = 1’)
let $ml := $m//log
order by $ml/shipno
return
<maintenanceLog>
{$ml}
</mantenanceLog>

This will return the documents for all class one ships.
Select Statement using XML
Column
Select shipno,class,maintenance
from ship
where class = 1

 This will produce the maintenance

document for each ship that is class 1.
 We can also create views this way
SQL/XML Queries
 Restricting results using XML element
values
 select captid,lname,fname from captain
where xmlexists(‘$c/contactinfo/Address[state=“MA”]’
passing captain.contact as “c”

• This will return the captid, lname and fname of

all captains who live in Massachusetts
SQL/XML Queries
 Projecting XML element values
 Two functions: XMLQuery and XMLTable
 XMLQuery retrieves value for 1 element
 XMLTable retrieves value for multiple elements
 XMLQuery example:
select xmlquery(‘$c/contactinfo/email’
passing contact as “c”)
from captain
where state = ‘MA’
 Thiswill return email addresses for all captains in
Massachusetts
SQL/XML Queries
XMLQuery (Continued)
 We could also look for only first email for each captain
by changing the first line:
select xmlquery(‘$c/contactinfo/email[1]’ …

 Similarly, we could use xmlexists to qualify:

select xmlquery(‘$c/contactinfo/email’
passing contact as “c”)
from captain
where state = ‘MA’
and xmlexists(‘$c/contactinfo/email’
passing contact as “c”)
SQL/XML Queries
XMLTable
 XMLTable retrieves XML elements
 Elements are mapped into result set
columns
 Maps XML data as relational data
SQL/XML Queries
XMLTable Example
select s.shipNo,sm.mid,sm.vid,sm.md,sm.cost
from ship s,
xmltable(‘$c/mrecord/log’ passing s.maintenance as “c”
columns varchar(4) mid path ‘mntid’,
varchar(4) vid path ‘vendorid’,
date md path ‘maintdate’,
decimal(7,2) cost path ‘cost’) as sm
This will produce a list of maintenance logs
for all ships
Joining XML and Relational Data
select c.captid,c.lname,c.fname
from captain, ship
where xmlexists(‘$s/mrec/log[captid=$c]’
passing ship.maintenance as “s”, captain.captid as “c”)

If the captain was the captain of any ship

when it underwent maintenance, he or she
will be listed
Using FLWR Expressions in
SQL/XML
select captid,
xmlquery(‘for $c in $cn/contactinfo
let $x := $c//city
return $x’ passing contact as “cn”)
from captain
where class = 1

Returns captid as well as city information

XMLElement
XML Element allows you to publish relational data as XML

select xmlelement(name “captain”,

xmlelement(name “captid”, captid),
xmlelement(name “lname”,lname),
xmlelement(name “fname”,fname),
xmlelement(name “class”,class))
from captain
where class <= 2
XMLElement
Output from previous command
<captain>
<captid>3563</captid>
<lname>Smith</lname>
<fname>John</fname>
<class>2</class>
</captain>
…
Aggregating and Grouping Data
select xmlelement(name “captainlist”,
xmlagg(xmlelement(name “captain”,
xmlforest(cid as “captid”,lname as “lname”,fname as “fname”,class
as “class”))
order by cid))
from captain
group by class

This query produces three captainlist

elements each with a number of captains.
Updating and Deleting XML Data

 Updates
 Use XMLParse command. You must specify
the entire XML column to update. If you
specify only 1 element to update, the rest of
the data will be lost.
 Deletion
 Same as standard SQL
 Can also use xmlexists to use XML as
qualifier
 Query Execution Plans

•Separate parsers for SQL and

XQuery statements

•Integrated query compiler for both

languages

•QGMX is an internal query graph

model

•Query execution plans contain

special operators for navigation
(XSCAN), XML index access
(XISCAN) and joins over XML Source: [2]
indexes (XANDOR)
Query Run-time Evaluation
 3 major components added for processing
queries over XML:
 XML Navigation
 XML Index Runtime
 XQuery Function Library
Summary
 Problems with CLOB and Shredded XML
storage
 Native XML support in DB2 offers:
 Hierarchical and parsed representation
 Path-specific XML indexing
 New XML join and query methods
 Integration of SQL and XQuery
References
[1] Nicola, M. and van der Linden, B. 2005. Native XML support in DB2 universal
database. In Proceedings of the 31st international Conference on Very Large Data
Bases (Trondheim, Norway, August 30 - September 02, 2005). Very Large Data
Bases. VLDB Endowment, 1164-1174.

[2] Beyer, K., Cochrane, R. J., Josifovski, V., Kleewein, J., Lapis, G., Lohman, G., Lyle,
B., Özcan, F., Pirahesh, H., Seemann, N., Truong, T., Van der Linden, B., Vickery, B.,
and Zhang, C. 2005. System RX: one part relational, one part XML. In Proceedings of
the 2005 ACM SIGMOD international Conference on Management of Data
(Baltimore, Maryland, June 14 - 16, 2005). SIGMOD '05. ACM Press, New York, NY,
347-358.
[3] http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0603saracco2/