Hotsos 2013 - Creating Structure in Unstructured Data
5 likes9,505 views
This document discusses creating structure from unstructured XML data and optimizing XML performance in Oracle databases. It provides examples of structuring Wikipedia XML data and indexing it in various ways using XMLType, binary XML, structured and unstructured XML indexes. The key is choosing the right storage and indexing approach depending on the query patterns and data structure. Proper design can significantly outperform default XML handling.
Hotsos 2013 - Creating Structure in Unstructured Data
- 1. Creating Structure in Unstructured Data What is possible, today…? Marco Gralike
- 5. “Big Data” = XML ?
- 6. Challenges are! Ahum, the problems are!
- 7. WikiPedia • One string of XML data with structured and unstructured data sections • Language: English • Size : 42,15 GB • Pages : 12.961.997 • Date : 21 Dec 2012
- 9. Setup • VirtualBox VM – OEL 5U8 (64) – 8 GB RAM • LaCie Little Big Disk – RAID 0 – Thunderbolt • Database – SGA 4GB – PGA 2GB
- 10. My new LaCie LBD is really fast -
- 11. Defeat?! - 1.000.000 pages only
- 12. Status of Technology used
- 13. XML - Where are we…? Gartner
- 14. Achieved…?
- 15. On the Horizon! • JSoniq • Zorba
- 17. Oracle XML DB • NO cost option • C (native / embedded kernel) • (XQuery) Standards • Code maintained by Oracle
- 18. XQuery XMLType Abstraction DB XQuery Procedural XQuery XQuery Rewrite Pushdown XVM (use “no query rewrite”) Relational Streaming XPath DOM Tree Evaluation Model Access SQL Execution Methods XMLIndex Object-Relational Binary XML Relational Storage Secure Files Source: S317428: Building Really Scalable XML Applications with Oracle XML DB and Oracle Text
- 19. So about what are we talking ?
- 27. WikiPedia • Structured & Unstructured bits and pieces • A lot of “unbounded” elements • Not a lot of restrictions • The bit with value is in element “tekst”
- 28. How do we get this Structured?
- 31. Strings = small & defined (12c?) Ename pointer += 100;
- 33. Flexible, Humans No Design Patterns
- 35. <verybigggr> <empno>1</empno><ename>Marco</ename> <empno>2</empno> </verybigggr> <small/><verybigggr/><bigger/>
- 40. We need options!
- 41. “XMLType” Container In Memory CLOB (document) (document) Object Relational Binary XML (data) (data)
- 42. XMLType In Memory (document) XOB XML Schema
- 43. XMLType Binary XML Securefile (document/content) Post Parse LOB Index
- 44. XMLType Object Relational (content) Fully Shredded Indexes
- 45. Something else to Realize !
- 46. “What is the fastest way to get this stuff in the database…?”
- 47. “…it depends…”
- 48. “So what is the fastest way to get XML in the database…?”
- 49. “…it depends…”
- 50. “So what is the fastest way to get XML in the database… … and useful in my case…?”
- 51. Garbage IN – Garbage OUT
- 52. WikiPedia • SQL*Loader • Parallel or Direct • Securefile LOB Column • 2.5 hours And no (performant) way to get the details out… a.k.a “completely useless”
- 53. WikiPedia • SQL*Loader • Parallel or Direct • Securefile Binary XML • …2.5 hours ???
- 54. XML Parsing • SAX - Simple API for XML • DOM - Document Object Module
- 55. fast insert performance CLOB XMLType CLOB (domain) indexes XMLType Binary XML XMLType Object Relational fast select performance
- 57. XML Partitioning • Object Relational Partitioning – Equi-Partitioning since version Oracle 11.1.0.7.0 • Binary XML Partitioning – Range, List, Hash • Local partitioned XMLIndex – LOCAL keyword in XMLIndex create syntax • Partition Key on virtual column (Binary XML) • Partition Key on column (Object Relational)
- 58. XMLType Binary XML Securefile (document/content) Post Parse LOB Index
- 59. Driving access on CONTENT BTre e Index bookstore Function based Index (XPath) book whitepaper title author author chapter title author id paragraph Unstructured Structured XMLIndex XMLIndex content structured content BTree Oracle XML Index Text Index
- 60. Structured Data
- 61. Structured XMLIndex (SXI) • CONTENT TABLE(s) • Based on XMLTABLE syntax Structured XMLIndex • XMLTable construct can be f (x) nested: – VIRTUAL column alias • Can be maintained manually • Secondary indexes possible Content Tables
- 62. Describe CONTENT TABLE • A “regular” heap table with columns… • Ideal for secondary indexes, if needed.
- 63. CONTENT TABLE(s) Structured XMLIndex f (x) Content Tables
- 65. Unstructured XMLIndex (UXI) • PATH TABLE • Use Path Subsetting Unstructured – Full Blown XMLIndex can be BIG XMLIndex f (x) • Token Tables (XDB.X$......) – Query re-write on Tokens – Fuzzy Searches, // – Optimizer Statistics • Can be maintained manually – Recorded in Pending Table Path Table • Secondary indexes possible
- 67. What’s hidden…
- 68. PATH TABLE Unstructured XMLIndex f (x) Path Table
- 69. Binary XML – No Index
- 70. Binary XML + XMLIndex (SXI)
- 71. Binary XML + XMLIndex + Sec.Ind.
- 72. Binary XML + XMLIndex + Sec.Ind.
- 74. XML Full Tekst Index • Based on Oracle Text Index, XQuery Full Text • XML Namespace Aware • XML Semantic aware full text search – Full-Tekst Selection Expression – contains text – Logical Full Text Operator – ftor, ftand, ftMildNot – Context Aware full text search
- 85. Balanced Design • Inserts, Updates & Deletes – XML Future Changes – Index Maintenance In Memory On Disk • Selects – In Memory – Via Indexes • XML Validation – Strict, Lazy – Client Side Possibilities
- 86. Reward • Optimal performance • Out performing XML • Proper design will give performance increase over XML handling… …proper design is still key…
- 88. References Oracle XML DB – http://www.oracle.com/pls/db112/homepage XML DB FAQ Thread – http://forums.oracle.com/forums/thread.jspa?thr eadID=410714 Personal Blog – http://www.xmldb.nl – http://technology.amis.nl
- 89. References Daniela Florescu, Oracle Corporation Advances in XML and XQuery Sam Idicula, Oracle XML DB Development Team Binary XML Storage and Query Processing in Oracle Jinyu Wang, Scott Brewton Making XML Technology Easier to Use Joel Spolsky - Joel on Software Back to Basics
- 90. References Oracle XML DB Main page material • Oracle XML DB : Best Practices to Get Optimal Performance out of XML Queries (PDF) • Oracle XML DB : Choosing the Best XMLType Storage Option for Your Use Case (PDF) • A Request for Comments for the Oracle Binary XML Format
Editor's Notes
- #19: See also OOW 2010, S317428: Building Really Scalable XML Applications with Oracle XML DB and Oracle Text – Nipun Agarwal, Oracle