Real-Time Data Warehousing using Oracle Database and OBIEE - Collaborate'11

Real-Time DW with OBIEE and Oracle Database
Stewart Bryson, Managing Director, Rittman Mead America
Collaborate, Orlando, April 2011

T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com

Thursday, 21 April 2011 1

Who am I?



• Stewart Bryson, Managing Director, Rittman
Mead America
• Oracle BI/DW Architect and Delivery Specialist
• Oracle Database and BI Stack since 1996
• Regular Community Speaker
• Writer for Rittman Mead Blog:
http://www.rittmanmead.com/blog
• Developer of Transcend Framework
• Email : stewart.bryson@rittmanmead.com
• Twitter : @stewartbryson



Who is Rittman Mead?



• Oracle BI&DW Experts
• Consulting, Training and Support
• Global Company, Offices in the US, UK and India
• Clients in the UK, USA, Europe, Middle-East
• Voted UKOUG BI Partner of the Year 2008
• 16 Sessions at Oracle Open World 2010
• Twitter : @rittmanmead



Rittman Mead BI Forum, May 2011



Registrations open for the Rittman Mead BI Forum, May 2011
• Brighton, UK, May 18-20th 2011
• Atlanta, GA, May 25-27th 2011
• Covers OBIEE, DW, EPM
• Guest speakers including:
‣ Jeff McQuigg
‣ Kevin McGinley
‣ Jean-Pierre Dijcks
‣ Mike Durran
‣ Shyam Varan Nath
‣ Philippe Lions
‣ Emiel van Bockel
• OBIEE 11g Masterclass with
Mark Rittman and Tony Heljula
• Full details at http://www.rittmanmead.com/biforum2011



What will this session answer?



• What do I mean by “real-time data warehousing?”
‣ Zero latency, or just reduced latency?
‣ Do I really mean “right-time data warehousing?”
• What can I use to reduce latency?
‣ What does OBIEE offer?
‣ What does the Oracle Database (and OWB)
offer?
• Can I blend these different offerings?
‣ Of course you can!
‣ It’s the name of the presentation



What do I mean by “real-time”?



• Traditional data warehouses
‣ Daily to weekly batch-load cycles
‣ Key-metrics are always delayed
‣ Information may not be “actionable”
• Real-time data warehouses
‣ Up to the minute or hour reporting
‣ Key-metrics are moving
‣ Information is actionable



An interval will always exist between the occurrence of a measurable
event and our ability to process that event as a “reportable fact”



L1&-6)&/(&)7#'.@@)&)2-#-"0)*#(@#.2$(/.23#'%-%#$%2#6%?)#'.@@)&.23#+%-)2$.)*=#L(&#)I%/0+)7#%#2)B#-&%2*%
+%-)2$"#-6%2#%#$6%23)#-(#%#&)3.(2%+#3&(10.23#(@#*-(&)*#-6%-#/%"#2))'#-(#>)#0&(0%3%-)'#-6&(136#/%2
%33&)3%-.(2#-%>+)*#4*))#L.31&)#M<=

K6)#'.?.*.(2#>)-B))2#'%-%#%$E1.*.-.(2#%2'#'%-%#0&)*)2-%-.(2#0&)*)2-*#%#D)"#E1)*-.(2#-6%-#%2"#$(/0%
$(2-)/0+%-.23#/(?.23#-(#%#&)%+,-./)#95#*"*-)/N#%&)#"(1#O1*-#.2-)&)*-)'#.2#0&(?.'.237#%*#&%0.'+"#%*#0(*
An interval will always exist between the occurrence of a measurable
(2#%+&)%'"#%$E1.&)'#'%-%7#(&#'(#"(1#%+*(#2))'#-(#&)0(&-#(2#2)B+"#%'')'#@%$-1%+#.2@(&/%-.(2C
event and our ability to process that event as a “reportable fact”
L(&#)I%/0+)7#%#B(&D)&#.2#%#/(>.+)
FIGURE 1: BI latency
$)2-&)#/%"#+.D)#-(#*))#.2@(&/%-.(2#
Event Report $1*-(/)&#.*P#%2'#-6)#+.D)+.6(('#-6%
$1*-(/)&P*#')/(3&%06.$#B(1+'#*B
*100+.)&=

K6.*#.2@(&/%-.(2#2))'*#-(#>)#0&)*)2
%&)#*0)%D.23#-(#-6)/<#%2'#0&)@)&%>
+.2),(@,>1*.2)**#%00+.$%-.(2#-6%-#/%
.2-)&%$-.(2=#R(B)?)&#.-#.*#12+.D)+"#-6
Recording Propagation Aggregation Query >"#-6)#$%++#$)2-&)#%3)2-#@(&#$1*-(/)
0&)'.$-)'#$61&2#B(1+'#>)#.2@+1)2$)
6(1&*=

S(2?)&*)+"7#%#*"*-)/#-6%-#.*#1*.23#
Real-time query latency -(#%+)&-#-(#0(-)2-.%+#@&%1'#4*-($D#/%&
$(/0%2.)*#%&)#0(**.>+)#)I%/0+)#1*
D2(B#%>(1-#10,-(,-6),/(/)2-#%$-.?

***+,-./0.12345,416,+537


Business Case



• We want to be able to segment our reporting
between “real-time” needs and static reports
‣ Most users will want real-time numbers
delivered with dashboards
‣ Some users (financial-minded) don’t want
numbers to change each time they run a report
• Use the standard Oracle BI Stack
‣ Oracle Database Enterprise Edition
‣ Oracle Business Intelligence Enterprise Edition



Latency versus Query Performance



Latency versus Query Performance

Latency

Query Performance



Oracle Next-Generation Reference Architecture



Requirement: Identifying and
Capturing Change



Requirement: Change Data Capture




• Oracle CDC
• OWB Trickle-feed
mappings
• Trigger-based
capture
• Recording
modification dates
• MINUS Command



• Oracle CDC • Oracle CDC
• OWB Trickle-feed • OWB Trickle-feed
mappings mappings
• Trigger-based • Message Queues
capture • Views over
• Recording database links
modification dates • File movements
• MINUS Command



• Oracle CDC • OWB Mappings
• Oracle CDC
and Process Flows
mappings • Other ETL Tools
mappings
• Message Queues • Custom ETL code
• Trigger-based
capture • Views over • Custom transaction
database links code
• Recording
• MINUS Command


Oracle Asynchronous Change Data Capture




Hot Log
Online redo logs mined locally
Change tables are updated and stored locally

Hot Log Distributed
Online redo logs are mined locally
Change tables are updated and stored remotely

Autolog Online
Online redo logs are mined remotely

Autolog Archive
Archived redo logs are mined remotely




Hot Log
Online redo logs mined locally
Change tables are updated and stored locally

Hot Log Distributed
Online redo logs are mined locally
Latency


Autolog Online
Online redo logs are mined remotely

Autolog Archive
Archived redo logs are mined remotely




• Oracle CDC • OWB Mappings
• Oracle CDC
and Process Flows
mappings • Other ETL Tools
mappings
• Message Queues • Custom ETL code
• Trigger-based
capture • Views over • Custom transaction
database links code
• Recording
• MINUS Command



• Trickle-feed mappings allow the development of Streams
Capture and Apply processes as continuously running
mappings.



Federated OLTP/EDW Reporting



• Layering in OLTP data to provide the real-time
component

• General approach is to use a BI tool such as
OBIEE

• No real database or modeling involvement



Classic Batch Load Star Schema: Process Flow



Classic Batch Load Star Schema: Fact Table Load



Federation with OBIEE



OBIEE Metadata and Data Layers




• Physical Data Sources, such
as relational databases, Relational, OLAP, File and
Application Data Sources
OLAP Cubes and application
source adapters




• A Semantic Model made up
of Metrics, dimensions,
Enterprise Semantic Model
hierarchies and calculations,
derived from...

source adapters




• The Web Catalog contains
definitions of reports, alerts
Web Catalog
dashboards, KPIs,
based on...

• A Semantic Model made up
of Metrics, dimensions,
Enterprise Semantic Model
hierarchies and calculations,
derived from...

source adapters



Flow of Data Through the Three-Layer Semantic Model



Flow of Data Through the Three-Layer Semantic Model

Simplification of the Data Model
Integration of Disparate DataSources
Addition of Business Logic and Calculations



Logical Table Sources Map to Physical Models



Logical Table Sources Respect the Physical Model



WITH
SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2,
T43901.CUST_LAST_NAME as c3,
T43971.SAL_AMT as c4
from
GCBC_CRM.CUSTOMERS T43901,
GCBC_POS.POS_TRANS T43971,
GCBC_POS.POS_TRANS_HEADER T43978
where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE
('2010-09-18 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T43978.TRANS_DATE )
union all
select T44042.CUSTOMER_FIRST_NAME as c2,
T44042.CUSTOMER_LAST_NAME as c3,
T44105.AMOUNT as c4
from
GCBC_EDW.CUSTOMER_DIM T44042,
GCBC_EDW.SALES_FACT T44105
where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )),
SAWITH1 AS (select sum(D3.c4) as c1,
D3.c2 as c2,
D3.c3 as c3
from
SAWITH0 D3
group by D3.c2, D3.c3)
select distinct 0 as c1,
D2.c2 as c2,
D2.c3 as c3,
D2.c1 as c4
from
SAWITH1 D2



WITH
SAWITH0 AS (select sum(T44105.AMOUNT) as c1,
T44042.CUSTOMER_FIRST_NAME as c2,
T45416.CALENDAR_YEAR as c4
from
GCBC_EDW.DATE_DIM T45416,
where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and
T45416.SQL_DATE = TO_DATE('2003-01-09 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') )
group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR)
D1.c2 as c2,
D1.c3 as c3,
D1.c4 as c4,
D1.c1 as c5
from
SAWITH0 D1



WITH
SAWITH0 AS (select sum(T44105.AMOUNT) as c1,
T44042.CUSTOMER_FIRST_NAME as c2,
T45416.CALENDAR_YEAR as c4
from
GCBC_EDW.DATE_DIM T45416,
where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and
T45416.SQL_DATE = TO_DATE('2003-01-09 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') )
group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR)
D1.c2 as c2,
D1.c3 as c3,
D1.c4 as c4,
D1.c1 as c5
from
SAWITH0 D1

• The BI Server understands from the Fragmentation
content how to “optimize” the query
• Only one LTS is needed to provide the answer



Demonstration
Fragmentation with OLTP and EDW



• Strengths
‣ Low latency
‣ No complex modeling or ETL
‣ Incorporated into any standard dimensional
EDW
• Weaknesses
‣ Performance
‣ Difficult to conform multiple systems



EDW with a Real-Time Component



• Modeling complimentary real-time structures in an
EDW

• Use functionality of the Oracle Database and OWB

• Complex ETL and Database interaction



The Real-Time Partition

Fact Table




Fact Real-Time
Fact Table
Partition




• All ETL processing of fact records are directed at
the Real-Time Partition using Micro-Batch
• No indexes, small data set
• Partition Exchange at the end of the day



Early-Arriving Facts



What if Type 2 dimension changes haven’t arrived yet?

• Grab the most recent dimension record when that
entity already exists
• Use a default “Not Known Yet” record when they
don’t
• Prior to the partition exchange, we run our late-
arriving dimension loads
• We update the Real-Time Partition with the correct
surrogate keys
‣ OWB generates an efficient MERGE
‣ Small data sets in the Real-Time Partition



Real-Time Integration: Fact Table RT Load



Real-Time Integration: Correct Dimensions



Real-Time Integration: “Closing the Books” on the Day



Plugging it together with OBIEE (Non-Fragmentation)

• Using standard database
functionality (UNION ALL)
• Could be done with a view
• Using Surrogate Keys in the
RT Partition
• OBIEE doesn’t have to
manage different LTS’s



Demonstration
EDW with a Real-Time Component



Summary



• Latency versus performance and scalability

• Latency and performance versus degree of
difficulty

• Different variations on the two test cases



Real-Time DW with OBIEE and Oracle Database
Stewart Bryson, Managing Director, Rittman Mead America
Collaborate, Orlando, April 2011



Real-Time Data Warehousing using Oracle Database and OBIEE - Collaborate'11

Recommended

More Related Content

Viewers also liked (10)

Similar to Real-Time Data Warehousing using Oracle Database and OBIEE - Collaborate'11 (20)

More from Mark Rittman (20)

Recently uploaded (20)

Real-Time Data Warehousing using Oracle Database and OBIEE - Collaborate'11