SlideShare a Scribd company logo
1 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Data Warehouse Migration to
Oracle Data Integrator 11g
Michael Rainey
Data Warehouse Developer, Business Information Systems
Email: michael.rainey@pnnl.gov
LinkedIn: http://www.linkedin.com/in/rainey
Web: http://www.pnnl.gov
2 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Pacific Northwest National Laboratory
• Department of Energy research laboratory,
operated by Battelle
– $1.1 billion in business volume in fiscal year 2010
• Mission Outcomes
– Strengthen U.S. scientific foundations for innovation
– Increase U.S. energy capacity and reduce dependence on
imported oil
– Prevent and counter terrorism and proliferation of weapons of
mass destruction
– Reduce environmental effects of human activity and create
sustainable systems
3 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
PNNL Data Warehouse
• Industry standard hub & spoke architecture
• Millions of rows processed daily
– 40+ source applications
– Standardization & transformations performed within the DW
– Final tables replicated to 70+ targets for operational/analytical use
• Legacy software
– Visual Basic 6
– SQL Server 2000 Data Transformation Services (DTS)
Legacy System
4 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Migration to Oracle Data Integrator
• Minimize implementation risk
– Data warehouse was mature, well designed
– High level of integration within data transformations
• Typical migration of this size estimated to take 2-3 years
– 4900+ DTS packages to migrate
– Interface building & testing estimated based on SQL code
complexity and inexperience with ODI
The Challenge
5 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Migration to ODI – Conversion Estimate
The Algorithm
Complexity
Level
Point
Analysis
Worst Case -
Hours To
Convert
Best Case -
Hours To
Convert
Transform Gathers and
Distributions
Non-
Standard
Total
Processes
Worst Case -
Est. lbr
hours
Best Case -
Est. lbr
hours
Level 1 1-6 1 0.5 285 4187 0 4472 4472 2236
Level 2 7-11 3 2 137 99 0 236 708 472
Level 3 12-17 5 5 83 32 0 115 575 575
Level 4 18 - 28 10 8 56 16 35 107 1070 856
Level 5 Above 28 15 15 42 7 0 49 735 735
Total 7560 4874
Tight schedule & limited resources led to use of ODI SDK
6 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Migration to ODI – The Solution
Overview
SQL parse tool
ODI SDK - Java
Build ODI objects
7 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Migration to ODI – The Solution
Metadata Stored as SQL
SELECT ...
FROM LOCATION_HIERARCHY LH LEFT OUTER JOIN
(
SELECT F.FACILITY_ID, F.FACILITY_ID AS VALUE, ‘ID' AS SOURCE
FROM FACILITY_S1 F
WHERE F.DW_CURRENT_FLAG = 'Y'
UNION
SELECT G.FACILITY_ID, G.ATTRIBUTE_VALUE AS VALUE, 'ALIAS' AS SOURCE
FROM FACILITY_ATTRIBUTE G
WHERE G.ATTRIBUTE_ID = 'BUILDING'
AND G.FACILITY_ID <> G.ATTRIBUTE_VALUE
AND G.DW_CURRENT_FLAG = 'Y'
UNION
SELECT H.FACILITY_ID, H.ATTRIBUTE_VALUE AS VALUE, ‘CORP' AS SOURCE
FROM FACILITY_ATTRIBUTE H
WHERE H.ATTRIBUTE_ID = ‘BUILDINGA'
AND H.FACILITY_ID <> H.ATTRIBUTE_VALUE
AND H.DW_CURRENT_FLAG = 'Y'
UNION
SELECT J.FACILITY_ID, J.LOCATION AS VALUE, ‘MAP' AS SOURCE
FROM LOC_TO_FAC_MAPPING J
WHERE J.DW_CURRENT_FLAG = 'Y'
AND J.FACILITY_ID <> J.LOCATION
) LOCFIS ON LH.LOCATION = LOCFIS.VALUE LEFT OUTER JOIN
( SELECT K.FACILITY_ID, K.FACILITY_ID AS VALUE, ‘ID' AS SOURCE
FROM FACILITY_S1 K
WHERE K.DW_CURRENT_FLAG = 'Y'
UNION
...
8 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Migration to ODI – The Solution
Interface Built in ODI
9 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Migration to ODI – Success!
• ODI SDK was a key component to a successful migration
• Large volume of ODI objects built in a short period of time
• Risk of human error minimized by automating Interface
creation
• Continued use of ODI SDK for day-to-day operations
10 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Thank you!
Michael Rainey
Data Warehouse Developer, Business Information Systems
Email: michael.rainey@pnnl.gov
LinkedIn: http://www.linkedin.com/in/rainey
Web: http://www.pnnl.gov
Ad

More Related Content

Viewers also liked (15)

A Walk Through the Kimball ETL Subsystems with Oracle Data Integration
A Walk Through the Kimball ETL Subsystems with Oracle Data IntegrationA Walk Through the Kimball ETL Subsystems with Oracle Data Integration
A Walk Through the Kimball ETL Subsystems with Oracle Data Integration
Michael Rainey
 
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data StreamingOracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Michael Rainey
 
Data Migration Between MongoDB and Oracle
Data Migration Between MongoDB and OracleData Migration Between MongoDB and Oracle
Data Migration Between MongoDB and Oracle
ChihYung(Raymond) Wu
 
Oracle 12 c new-features
Oracle 12 c new-featuresOracle 12 c new-features
Oracle 12 c new-features
Navneet Upneja
 
Oracle To Sql Server migration process
Oracle To Sql Server migration processOracle To Sql Server migration process
Oracle To Sql Server migration process
harirk1986
 
GoldenGate and Oracle Data Integrator - A Perfect Match- Upgrade to 12c
GoldenGate and Oracle Data Integrator - A Perfect Match- Upgrade to 12cGoldenGate and Oracle Data Integrator - A Perfect Match- Upgrade to 12c
GoldenGate and Oracle Data Integrator - A Perfect Match- Upgrade to 12c
Michael Rainey
 
awesome groovy
awesome groovyawesome groovy
awesome groovy
Paul King
 
SMES Wrap Up
SMES Wrap UpSMES Wrap Up
SMES Wrap Up
Rosetta Eun Ryong Lee
 
MADA Holding engineering industries business unit - metal division - distri...
MADA Holding  engineering industries business unit - metal division -  distri...MADA Holding  engineering industries business unit - metal division -  distri...
MADA Holding engineering industries business unit - metal division - distri...
Mohamed Adel.M.Sc.,MIMBA,PMP ®
 
Delivering the Data Factory, Data Reservoir and a Scalable Oracle Big Data Ar...
Delivering the Data Factory, Data Reservoir and a Scalable Oracle Big Data Ar...Delivering the Data Factory, Data Reservoir and a Scalable Oracle Big Data Ar...
Delivering the Data Factory, Data Reservoir and a Scalable Oracle Big Data Ar...
Mark Rittman
 
GoldenGate and Oracle Data Integrator - A Perfect Match...
GoldenGate and Oracle Data Integrator - A Perfect Match...GoldenGate and Oracle Data Integrator - A Perfect Match...
GoldenGate and Oracle Data Integrator - A Perfect Match...
Michael Rainey
 
Real-time Data Warehouse Upgrade – Success Stories
Real-time Data Warehouse Upgrade – Success StoriesReal-time Data Warehouse Upgrade – Success Stories
Real-time Data Warehouse Upgrade – Success Stories
Michael Rainey
 
Real-Time Data Replication to Hadoop using GoldenGate 12c Adaptors
Real-Time Data Replication to Hadoop using GoldenGate 12c AdaptorsReal-Time Data Replication to Hadoop using GoldenGate 12c Adaptors
Real-Time Data Replication to Hadoop using GoldenGate 12c Adaptors
Michael Rainey
 
Single phase tansformer
Single phase tansformerSingle phase tansformer
Single phase tansformer
Md Abdullah Al Mayin
 
In-Memory Computing: How, Why? and common Patterns
In-Memory Computing: How, Why? and common PatternsIn-Memory Computing: How, Why? and common Patterns
In-Memory Computing: How, Why? and common Patterns
Srinath Perera
 
A Walk Through the Kimball ETL Subsystems with Oracle Data Integration
A Walk Through the Kimball ETL Subsystems with Oracle Data IntegrationA Walk Through the Kimball ETL Subsystems with Oracle Data Integration
A Walk Through the Kimball ETL Subsystems with Oracle Data Integration
Michael Rainey
 
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data StreamingOracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Michael Rainey
 
Data Migration Between MongoDB and Oracle
Data Migration Between MongoDB and OracleData Migration Between MongoDB and Oracle
Data Migration Between MongoDB and Oracle
ChihYung(Raymond) Wu
 
Oracle 12 c new-features
Oracle 12 c new-featuresOracle 12 c new-features
Oracle 12 c new-features
Navneet Upneja
 
Oracle To Sql Server migration process
Oracle To Sql Server migration processOracle To Sql Server migration process
Oracle To Sql Server migration process
harirk1986
 
GoldenGate and Oracle Data Integrator - A Perfect Match- Upgrade to 12c
GoldenGate and Oracle Data Integrator - A Perfect Match- Upgrade to 12cGoldenGate and Oracle Data Integrator - A Perfect Match- Upgrade to 12c
GoldenGate and Oracle Data Integrator - A Perfect Match- Upgrade to 12c
Michael Rainey
 
awesome groovy
awesome groovyawesome groovy
awesome groovy
Paul King
 
MADA Holding engineering industries business unit - metal division - distri...
MADA Holding  engineering industries business unit - metal division -  distri...MADA Holding  engineering industries business unit - metal division -  distri...
MADA Holding engineering industries business unit - metal division - distri...
Mohamed Adel.M.Sc.,MIMBA,PMP ®
 
Delivering the Data Factory, Data Reservoir and a Scalable Oracle Big Data Ar...
Delivering the Data Factory, Data Reservoir and a Scalable Oracle Big Data Ar...Delivering the Data Factory, Data Reservoir and a Scalable Oracle Big Data Ar...
Delivering the Data Factory, Data Reservoir and a Scalable Oracle Big Data Ar...
Mark Rittman
 
GoldenGate and Oracle Data Integrator - A Perfect Match...
GoldenGate and Oracle Data Integrator - A Perfect Match...GoldenGate and Oracle Data Integrator - A Perfect Match...
GoldenGate and Oracle Data Integrator - A Perfect Match...
Michael Rainey
 
Real-time Data Warehouse Upgrade – Success Stories
Real-time Data Warehouse Upgrade – Success StoriesReal-time Data Warehouse Upgrade – Success Stories
Real-time Data Warehouse Upgrade – Success Stories
Michael Rainey
 
Real-Time Data Replication to Hadoop using GoldenGate 12c Adaptors
Real-Time Data Replication to Hadoop using GoldenGate 12c AdaptorsReal-Time Data Replication to Hadoop using GoldenGate 12c Adaptors
Real-Time Data Replication to Hadoop using GoldenGate 12c Adaptors
Michael Rainey
 
In-Memory Computing: How, Why? and common Patterns
In-Memory Computing: How, Why? and common PatternsIn-Memory Computing: How, Why? and common Patterns
In-Memory Computing: How, Why? and common Patterns
Srinath Perera
 

Similar to Data warehouse migration to oracle data integrator 11g (20)

How green standards are changing data center design and operations
How green standards are changing data center design and operationsHow green standards are changing data center design and operations
How green standards are changing data center design and operations
Schneider Electric
 
Eland: A Python client for data analysis and exploration
Eland: A Python client for data analysis and explorationEland: A Python client for data analysis and exploration
Eland: A Python client for data analysis and exploration
Elasticsearch
 
Delivering Business Value with the Next Generation Data Center
Delivering Business Value with the Next Generation Data CenterDelivering Business Value with the Next Generation Data Center
Delivering Business Value with the Next Generation Data Center
George Demarest
 
Shanish_SQL_PLSQL_Profile
Shanish_SQL_PLSQL_ProfileShanish_SQL_PLSQL_Profile
Shanish_SQL_PLSQL_Profile
Shanish Jain
 
Example of Fund Raising Presentation to Investors
Example of Fund Raising Presentation to InvestorsExample of Fund Raising Presentation to Investors
Example of Fund Raising Presentation to Investors
Jeremiah Josey
 
Shivaprasada_Kodoth
Shivaprasada_KodothShivaprasada_Kodoth
Shivaprasada_Kodoth
Shivaprasada Kodoth
 
Pinaki bhattachaya
Pinaki bhattachayaPinaki bhattachaya
Pinaki bhattachaya
Pinaki Bhattacharya
 
Veerapradeep_Apps_profile
Veerapradeep_Apps_profileVeerapradeep_Apps_profile
Veerapradeep_Apps_profile
Veerapradeep Ravi
 
Demystifying Data Virtualization: Why it’s Now Critical for Your Data Strategy
Demystifying Data Virtualization: Why it’s Now Critical for Your Data StrategyDemystifying Data Virtualization: Why it’s Now Critical for Your Data Strategy
Demystifying Data Virtualization: Why it’s Now Critical for Your Data Strategy
Denodo
 
mani_datastage
mani_datastagemani_datastage
mani_datastage
manishankar ray
 
Rio Networks Carrier Neutral Colocation Facility
Rio Networks Carrier Neutral Colocation FacilityRio Networks Carrier Neutral Colocation Facility
Rio Networks Carrier Neutral Colocation Facility
markbiltonsmith
 
Tracxn - Data Center Infrastructure Startup Landscape
Tracxn - Data Center Infrastructure Startup LandscapeTracxn - Data Center Infrastructure Startup Landscape
Tracxn - Data Center Infrastructure Startup Landscape
Tracxn
 
Collaborate 2012-business data transformation and consolidation for a global ...
Collaborate 2012-business data transformation and consolidation for a global ...Collaborate 2012-business data transformation and consolidation for a global ...
Collaborate 2012-business data transformation and consolidation for a global ...
Chain Sys Corporation
 
Collaborate 2012-business data transformation and consolidation
Collaborate 2012-business data transformation and consolidationCollaborate 2012-business data transformation and consolidation
Collaborate 2012-business data transformation and consolidation
Chain Sys Corporation
 
MT85 Challenges at the Edge: Dell Edge Gateways
MT85 Challenges at the Edge: Dell Edge GatewaysMT85 Challenges at the Edge: Dell Edge Gateways
MT85 Challenges at the Edge: Dell Edge Gateways
Dell EMC World
 
SplunkLive! Customer Presentation – Harris
SplunkLive! Customer Presentation – HarrisSplunkLive! Customer Presentation – Harris
SplunkLive! Customer Presentation – Harris
Splunk
 
Pinaki bhattachaya
Pinaki bhattachayaPinaki bhattachaya
Pinaki bhattachaya
Pinaki Bhattacharya
 
LAWAL ABDULLAHI (Resume)
LAWAL ABDULLAHI (Resume)LAWAL ABDULLAHI (Resume)
LAWAL ABDULLAHI (Resume)
Abdullahi Lawal
 
Refactoring the Factory: Building a Cloud-Native App Transformation Factory
Refactoring the Factory: Building a Cloud-Native App Transformation FactoryRefactoring the Factory: Building a Cloud-Native App Transformation Factory
Refactoring the Factory: Building a Cloud-Native App Transformation Factory
VMware Tanzu
 
Green Wave Project :: Liberate your balance sheet
Green Wave Project :: Liberate your balance sheetGreen Wave Project :: Liberate your balance sheet
Green Wave Project :: Liberate your balance sheet
AllenKnoll
 
How green standards are changing data center design and operations
How green standards are changing data center design and operationsHow green standards are changing data center design and operations
How green standards are changing data center design and operations
Schneider Electric
 
Eland: A Python client for data analysis and exploration
Eland: A Python client for data analysis and explorationEland: A Python client for data analysis and exploration
Eland: A Python client for data analysis and exploration
Elasticsearch
 
Delivering Business Value with the Next Generation Data Center
Delivering Business Value with the Next Generation Data CenterDelivering Business Value with the Next Generation Data Center
Delivering Business Value with the Next Generation Data Center
George Demarest
 
Shanish_SQL_PLSQL_Profile
Shanish_SQL_PLSQL_ProfileShanish_SQL_PLSQL_Profile
Shanish_SQL_PLSQL_Profile
Shanish Jain
 
Example of Fund Raising Presentation to Investors
Example of Fund Raising Presentation to InvestorsExample of Fund Raising Presentation to Investors
Example of Fund Raising Presentation to Investors
Jeremiah Josey
 
Demystifying Data Virtualization: Why it’s Now Critical for Your Data Strategy
Demystifying Data Virtualization: Why it’s Now Critical for Your Data StrategyDemystifying Data Virtualization: Why it’s Now Critical for Your Data Strategy
Demystifying Data Virtualization: Why it’s Now Critical for Your Data Strategy
Denodo
 
Rio Networks Carrier Neutral Colocation Facility
Rio Networks Carrier Neutral Colocation FacilityRio Networks Carrier Neutral Colocation Facility
Rio Networks Carrier Neutral Colocation Facility
markbiltonsmith
 
Tracxn - Data Center Infrastructure Startup Landscape
Tracxn - Data Center Infrastructure Startup LandscapeTracxn - Data Center Infrastructure Startup Landscape
Tracxn - Data Center Infrastructure Startup Landscape
Tracxn
 
Collaborate 2012-business data transformation and consolidation for a global ...
Collaborate 2012-business data transformation and consolidation for a global ...Collaborate 2012-business data transformation and consolidation for a global ...
Collaborate 2012-business data transformation and consolidation for a global ...
Chain Sys Corporation
 
Collaborate 2012-business data transformation and consolidation
Collaborate 2012-business data transformation and consolidationCollaborate 2012-business data transformation and consolidation
Collaborate 2012-business data transformation and consolidation
Chain Sys Corporation
 
MT85 Challenges at the Edge: Dell Edge Gateways
MT85 Challenges at the Edge: Dell Edge GatewaysMT85 Challenges at the Edge: Dell Edge Gateways
MT85 Challenges at the Edge: Dell Edge Gateways
Dell EMC World
 
SplunkLive! Customer Presentation – Harris
SplunkLive! Customer Presentation – HarrisSplunkLive! Customer Presentation – Harris
SplunkLive! Customer Presentation – Harris
Splunk
 
LAWAL ABDULLAHI (Resume)
LAWAL ABDULLAHI (Resume)LAWAL ABDULLAHI (Resume)
LAWAL ABDULLAHI (Resume)
Abdullahi Lawal
 
Refactoring the Factory: Building a Cloud-Native App Transformation Factory
Refactoring the Factory: Building a Cloud-Native App Transformation FactoryRefactoring the Factory: Building a Cloud-Native App Transformation Factory
Refactoring the Factory: Building a Cloud-Native App Transformation Factory
VMware Tanzu
 
Green Wave Project :: Liberate your balance sheet
Green Wave Project :: Liberate your balance sheetGreen Wave Project :: Liberate your balance sheet
Green Wave Project :: Liberate your balance sheet
AllenKnoll
 
Ad

More from Michael Rainey (14)

Data Warehouse - Incremental Migration to the Cloud
Data Warehouse - Incremental Migration to the CloudData Warehouse - Incremental Migration to the Cloud
Data Warehouse - Incremental Migration to the Cloud
Michael Rainey
 
Continuous Data Replication into Cloud Storage with Oracle GoldenGate
Continuous Data Replication into Cloud Storage with Oracle GoldenGateContinuous Data Replication into Cloud Storage with Oracle GoldenGate
Continuous Data Replication into Cloud Storage with Oracle GoldenGate
Michael Rainey
 
SQL on Hadoop for the Oracle Professional
SQL on Hadoop for the Oracle ProfessionalSQL on Hadoop for the Oracle Professional
SQL on Hadoop for the Oracle Professional
Michael Rainey
 
Going Serverless - an Introduction to AWS Glue
Going Serverless - an Introduction to AWS GlueGoing Serverless - an Introduction to AWS Glue
Going Serverless - an Introduction to AWS Glue
Michael Rainey
 
Offload, Transform, and Present - the New World of Data Integration
Offload, Transform, and Present - the New World of Data IntegrationOffload, Transform, and Present - the New World of Data Integration
Offload, Transform, and Present - the New World of Data Integration
Michael Rainey
 
Oracle Data Integrator 12c - Getting Started
Oracle Data Integrator 12c - Getting StartedOracle Data Integrator 12c - Getting Started
Oracle Data Integrator 12c - Getting Started
Michael Rainey
 
Streaming with Oracle Data Integration
Streaming with Oracle Data IntegrationStreaming with Oracle Data Integration
Streaming with Oracle Data Integration
Michael Rainey
 
Oracle data integrator 12c - getting started
Oracle data integrator 12c - getting startedOracle data integrator 12c - getting started
Oracle data integrator 12c - getting started
Michael Rainey
 
Oracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data StreamingOracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data Streaming
Michael Rainey
 
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data StreamingOracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Michael Rainey
 
Tame Big Data with Oracle Data Integration
Tame Big Data with Oracle Data IntegrationTame Big Data with Oracle Data Integration
Tame Big Data with Oracle Data Integration
Michael Rainey
 
A Picture Can Replace A Thousand Words
A Picture Can Replace A Thousand WordsA Picture Can Replace A Thousand Words
A Picture Can Replace A Thousand Words
Michael Rainey
 
A Walk Through the Kimball ETL Subsystems with Oracle Data Integration
A Walk Through the Kimball ETL Subsystems with Oracle Data IntegrationA Walk Through the Kimball ETL Subsystems with Oracle Data Integration
A Walk Through the Kimball ETL Subsystems with Oracle Data Integration
Michael Rainey
 
KScope14 - Real-Time Data Warehouse Upgrade - Success Stories
KScope14 - Real-Time Data Warehouse Upgrade - Success StoriesKScope14 - Real-Time Data Warehouse Upgrade - Success Stories
KScope14 - Real-Time Data Warehouse Upgrade - Success Stories
Michael Rainey
 
Data Warehouse - Incremental Migration to the Cloud
Data Warehouse - Incremental Migration to the CloudData Warehouse - Incremental Migration to the Cloud
Data Warehouse - Incremental Migration to the Cloud
Michael Rainey
 
Continuous Data Replication into Cloud Storage with Oracle GoldenGate
Continuous Data Replication into Cloud Storage with Oracle GoldenGateContinuous Data Replication into Cloud Storage with Oracle GoldenGate
Continuous Data Replication into Cloud Storage with Oracle GoldenGate
Michael Rainey
 
SQL on Hadoop for the Oracle Professional
SQL on Hadoop for the Oracle ProfessionalSQL on Hadoop for the Oracle Professional
SQL on Hadoop for the Oracle Professional
Michael Rainey
 
Going Serverless - an Introduction to AWS Glue
Going Serverless - an Introduction to AWS GlueGoing Serverless - an Introduction to AWS Glue
Going Serverless - an Introduction to AWS Glue
Michael Rainey
 
Offload, Transform, and Present - the New World of Data Integration
Offload, Transform, and Present - the New World of Data IntegrationOffload, Transform, and Present - the New World of Data Integration
Offload, Transform, and Present - the New World of Data Integration
Michael Rainey
 
Oracle Data Integrator 12c - Getting Started
Oracle Data Integrator 12c - Getting StartedOracle Data Integrator 12c - Getting Started
Oracle Data Integrator 12c - Getting Started
Michael Rainey
 
Streaming with Oracle Data Integration
Streaming with Oracle Data IntegrationStreaming with Oracle Data Integration
Streaming with Oracle Data Integration
Michael Rainey
 
Oracle data integrator 12c - getting started
Oracle data integrator 12c - getting startedOracle data integrator 12c - getting started
Oracle data integrator 12c - getting started
Michael Rainey
 
Oracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data StreamingOracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data Streaming
Michael Rainey
 
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data StreamingOracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming
Michael Rainey
 
Tame Big Data with Oracle Data Integration
Tame Big Data with Oracle Data IntegrationTame Big Data with Oracle Data Integration
Tame Big Data with Oracle Data Integration
Michael Rainey
 
A Picture Can Replace A Thousand Words
A Picture Can Replace A Thousand WordsA Picture Can Replace A Thousand Words
A Picture Can Replace A Thousand Words
Michael Rainey
 
A Walk Through the Kimball ETL Subsystems with Oracle Data Integration
A Walk Through the Kimball ETL Subsystems with Oracle Data IntegrationA Walk Through the Kimball ETL Subsystems with Oracle Data Integration
A Walk Through the Kimball ETL Subsystems with Oracle Data Integration
Michael Rainey
 
KScope14 - Real-Time Data Warehouse Upgrade - Success Stories
KScope14 - Real-Time Data Warehouse Upgrade - Success StoriesKScope14 - Real-Time Data Warehouse Upgrade - Success Stories
KScope14 - Real-Time Data Warehouse Upgrade - Success Stories
Michael Rainey
 
Ad

Recently uploaded (20)

Role of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered ManufacturingRole of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered Manufacturing
Andrew Leo
 
tecnologias de las primeras civilizaciones.pdf
tecnologias de las primeras civilizaciones.pdftecnologias de las primeras civilizaciones.pdf
tecnologias de las primeras civilizaciones.pdf
fjgm517
 
Linux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdfLinux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdf
RHCSA Guru
 
Complete Guide to Advanced Logistics Management Software in Riyadh.pdf
Complete Guide to Advanced Logistics Management Software in Riyadh.pdfComplete Guide to Advanced Logistics Management Software in Riyadh.pdf
Complete Guide to Advanced Logistics Management Software in Riyadh.pdf
Software Company
 
Drupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy ConsumptionDrupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy Consumption
Exove
 
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsAI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global Trends
InData Labs
 
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfSAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
Precisely
 
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your UsersAutomation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Lynda Kane
 
Automation Dreamin': Capture User Feedback From Anywhere
Automation Dreamin': Capture User Feedback From AnywhereAutomation Dreamin': Capture User Feedback From Anywhere
Automation Dreamin': Capture User Feedback From Anywhere
Lynda Kane
 
Big Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur MorganBig Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur Morgan
Arthur Morgan
 
Salesforce AI Associate 2 of 2 Certification.docx
Salesforce AI Associate 2 of 2 Certification.docxSalesforce AI Associate 2 of 2 Certification.docx
Salesforce AI Associate 2 of 2 Certification.docx
José Enrique López Rivera
 
How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?
Daniel Lehner
 
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)
Ortus Solutions, Corp
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
Lynda Kane
 
Asthma presentación en inglés abril 2025 pdf
Asthma presentación en inglés abril 2025 pdfAsthma presentación en inglés abril 2025 pdf
Asthma presentación en inglés abril 2025 pdf
VanessaRaudez
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
"Rebranding for Growth", Anna Velykoivanenko
"Rebranding for Growth", Anna Velykoivanenko"Rebranding for Growth", Anna Velykoivanenko
"Rebranding for Growth", Anna Velykoivanenko
Fwdays
 
Hands On: Create a Lightning Aura Component with force:RecordData
Hands On: Create a Lightning Aura Component with force:RecordDataHands On: Create a Lightning Aura Component with force:RecordData
Hands On: Create a Lightning Aura Component with force:RecordData
Lynda Kane
 
Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025
Splunk
 
Role of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered ManufacturingRole of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered Manufacturing
Andrew Leo
 
tecnologias de las primeras civilizaciones.pdf
tecnologias de las primeras civilizaciones.pdftecnologias de las primeras civilizaciones.pdf
tecnologias de las primeras civilizaciones.pdf
fjgm517
 
Linux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdfLinux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdf
RHCSA Guru
 
Complete Guide to Advanced Logistics Management Software in Riyadh.pdf
Complete Guide to Advanced Logistics Management Software in Riyadh.pdfComplete Guide to Advanced Logistics Management Software in Riyadh.pdf
Complete Guide to Advanced Logistics Management Software in Riyadh.pdf
Software Company
 
Drupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy ConsumptionDrupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy Consumption
Exove
 
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsAI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global Trends
InData Labs
 
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfSAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
Precisely
 
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your UsersAutomation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Lynda Kane
 
Automation Dreamin': Capture User Feedback From Anywhere
Automation Dreamin': Capture User Feedback From AnywhereAutomation Dreamin': Capture User Feedback From Anywhere
Automation Dreamin': Capture User Feedback From Anywhere
Lynda Kane
 
Big Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur MorganBig Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur Morgan
Arthur Morgan
 
Salesforce AI Associate 2 of 2 Certification.docx
Salesforce AI Associate 2 of 2 Certification.docxSalesforce AI Associate 2 of 2 Certification.docx
Salesforce AI Associate 2 of 2 Certification.docx
José Enrique López Rivera
 
How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?
Daniel Lehner
 
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)
Ortus Solutions, Corp
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
Lynda Kane
 
Asthma presentación en inglés abril 2025 pdf
Asthma presentación en inglés abril 2025 pdfAsthma presentación en inglés abril 2025 pdf
Asthma presentación en inglés abril 2025 pdf
VanessaRaudez
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
"Rebranding for Growth", Anna Velykoivanenko
"Rebranding for Growth", Anna Velykoivanenko"Rebranding for Growth", Anna Velykoivanenko
"Rebranding for Growth", Anna Velykoivanenko
Fwdays
 
Hands On: Create a Lightning Aura Component with force:RecordData
Hands On: Create a Lightning Aura Component with force:RecordDataHands On: Create a Lightning Aura Component with force:RecordData
Hands On: Create a Lightning Aura Component with force:RecordData
Lynda Kane
 
Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025
Splunk
 

Data warehouse migration to oracle data integrator 11g

  • 1. 1 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8 Data Warehouse Migration to Oracle Data Integrator 11g Michael Rainey Data Warehouse Developer, Business Information Systems Email: [email protected] LinkedIn: http://www.linkedin.com/in/rainey Web: http://www.pnnl.gov
  • 2. 2 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8 Pacific Northwest National Laboratory • Department of Energy research laboratory, operated by Battelle – $1.1 billion in business volume in fiscal year 2010 • Mission Outcomes – Strengthen U.S. scientific foundations for innovation – Increase U.S. energy capacity and reduce dependence on imported oil – Prevent and counter terrorism and proliferation of weapons of mass destruction – Reduce environmental effects of human activity and create sustainable systems
  • 3. 3 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8 PNNL Data Warehouse • Industry standard hub & spoke architecture • Millions of rows processed daily – 40+ source applications – Standardization & transformations performed within the DW – Final tables replicated to 70+ targets for operational/analytical use • Legacy software – Visual Basic 6 – SQL Server 2000 Data Transformation Services (DTS) Legacy System
  • 4. 4 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8 Migration to Oracle Data Integrator • Minimize implementation risk – Data warehouse was mature, well designed – High level of integration within data transformations • Typical migration of this size estimated to take 2-3 years – 4900+ DTS packages to migrate – Interface building & testing estimated based on SQL code complexity and inexperience with ODI The Challenge
  • 5. 5 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8 Migration to ODI – Conversion Estimate The Algorithm Complexity Level Point Analysis Worst Case - Hours To Convert Best Case - Hours To Convert Transform Gathers and Distributions Non- Standard Total Processes Worst Case - Est. lbr hours Best Case - Est. lbr hours Level 1 1-6 1 0.5 285 4187 0 4472 4472 2236 Level 2 7-11 3 2 137 99 0 236 708 472 Level 3 12-17 5 5 83 32 0 115 575 575 Level 4 18 - 28 10 8 56 16 35 107 1070 856 Level 5 Above 28 15 15 42 7 0 49 735 735 Total 7560 4874 Tight schedule & limited resources led to use of ODI SDK
  • 6. 6 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8 Migration to ODI – The Solution Overview SQL parse tool ODI SDK - Java Build ODI objects
  • 7. 7 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8 Migration to ODI – The Solution Metadata Stored as SQL SELECT ... FROM LOCATION_HIERARCHY LH LEFT OUTER JOIN ( SELECT F.FACILITY_ID, F.FACILITY_ID AS VALUE, ‘ID' AS SOURCE FROM FACILITY_S1 F WHERE F.DW_CURRENT_FLAG = 'Y' UNION SELECT G.FACILITY_ID, G.ATTRIBUTE_VALUE AS VALUE, 'ALIAS' AS SOURCE FROM FACILITY_ATTRIBUTE G WHERE G.ATTRIBUTE_ID = 'BUILDING' AND G.FACILITY_ID <> G.ATTRIBUTE_VALUE AND G.DW_CURRENT_FLAG = 'Y' UNION SELECT H.FACILITY_ID, H.ATTRIBUTE_VALUE AS VALUE, ‘CORP' AS SOURCE FROM FACILITY_ATTRIBUTE H WHERE H.ATTRIBUTE_ID = ‘BUILDINGA' AND H.FACILITY_ID <> H.ATTRIBUTE_VALUE AND H.DW_CURRENT_FLAG = 'Y' UNION SELECT J.FACILITY_ID, J.LOCATION AS VALUE, ‘MAP' AS SOURCE FROM LOC_TO_FAC_MAPPING J WHERE J.DW_CURRENT_FLAG = 'Y' AND J.FACILITY_ID <> J.LOCATION ) LOCFIS ON LH.LOCATION = LOCFIS.VALUE LEFT OUTER JOIN ( SELECT K.FACILITY_ID, K.FACILITY_ID AS VALUE, ‘ID' AS SOURCE FROM FACILITY_S1 K WHERE K.DW_CURRENT_FLAG = 'Y' UNION ...
  • 8. 8 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8 Migration to ODI – The Solution Interface Built in ODI
  • 9. 9 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8 Migration to ODI – Success! • ODI SDK was a key component to a successful migration • Large volume of ODI objects built in a short period of time • Risk of human error minimized by automating Interface creation • Continued use of ODI SDK for day-to-day operations
  • 10. 10 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8 Thank you! Michael Rainey Data Warehouse Developer, Business Information Systems Email: [email protected] LinkedIn: http://www.linkedin.com/in/rainey Web: http://www.pnnl.gov