Extended Data Warehouse - A New Data Architecture for Modern BI with Claudia Imhoff

Extended Data Warehouse -
A New Data Architecture
for Modern BI

Today’s Speakers
■ Paul Moxon
Senior Director, Product Management
Denodo Technologies
■ Claudia Imhoff
President, Intelligent Solutions
Founder, Boulder BI Brain Trust

Copyright © Intelligent Solutions, Inc. 2015 All Rights Reserved
Agenda
 Extending the Data Warehouse Architecture
 Use Cases for a Modern BI Environment
 Things to Ponder…
 XDW – Real World Examples
3

Next Generation BI
4Based on a concept by Shree Dandekar of Dell
Business
insights
Economics
New
technologies
Non-traditional
data sources
Increasing
data volumes
& data rates
Extended data
warehouse
Next
generation
BI
DRIVERS
FEATURES
Slide compliments of Colin White – BI Research, Inc.

A Complex BI Environment
5
Multiple user devices
Multiple output formats
Multiple deployment options
Sophisticated analytics
+ complex analytic workloadsMultiple data sources
Increasing data volumes
& data rates
DW historical
data
Web & social
content
Sensor
data
Operational
data
Text &
media files
Decision
management
Data
management
Data
integration
Data
analysis
Decision
management

The Extended Data Warehouse
Architecture (XDW)
6
Traditional EDW
environment
Investigative computing
platform
Analytic tools & applications
Other internal & external
structured & multi-structured data
Real-time streaming data
Courtesy of Colin White – BI Research, Inc.Operational real-time environment
RT analysis engineOperational systems
BI services
Data
refinery
Data integration
platform

Agenda
7

Operational Analytics Use
Case
Embedded or callable BI
services:
 Real-time fraud detection
 Real-time loan risk assessment
 Optimizing online promotions
 Location-based offers
 Contact center optimization
 Supply chain optimization
Real-time analysis engine:
 Traffic flow optimization
 Web event analysis
 Natural resource exploration
analysis
 Stock trading analysis
 Risk analysis
 Correlation of unrelated data
streams (e.g., weather effects on
product sales)
8
Operational real-time environment
RT analysis engine
Operational systems
BI services

Data Provisioning Use Case:
Data Integration
9
 Heavy lifting process of extracting,
transforming to standard format
and loading structured data –
mostly batch
 Physically consolidates data into
“trusted” EDW sets for analysis
 Invokes data quality processing
where needed
 Employs low-cost hardware and
software to enable large data
volumes to be combined and stored
 Requires more formal governance
policies to manage data security,
privacy, quality, archiving and
destruction
Traditional EDW
environment
platform
Data
refinery
Data integration
platform

Data Integration Cases
 For use with production analyses in the traditional
enterprise data warehouse
 Data is consolidated into higher quality, trusted sets
 Trickle feeds allow near real-time analytics
 Reliable, consistent, historical data for production reporting, multi-
dimensional analytics, advanced analytics
 Probably is part of formal data governance process
 Is conducted in persistent staging area
10

Data Provisioning Use Case:
Data Refinery
11
 Ingests raw detailed structured and
unstructured data in batch and/or
real-time into a managed data store
 Distills data into useful business
information and distributes the
results to downstream systems
 May also directly analyze certain
types of data
 Also employs low-cost hardware
and software to enable large
amounts of detailed data to be
managed cost effectively
 Requires (flexible) governance
policies to manage data security,
privacy, quality, archiving and
destruction
Traditional EDW
environment
platform
Data
refinery
Data integration
platform

Data Refinery Cases
 Many organizations use the data refinery to determine
what’s of value in big data
 Not all data is useful
 Quickly discover interesting data
 Perform rough analyses to determine valuable data
 Move valuable data only into the investigative computing platform
or to the data integration platform
 Probably not part of formal data governance process
 Can be considered part of the staging area
12

Traditional EDW Use Cases
13
Most BI environments today
 New technologies can be
incorporated into the EDW
environment to improve
performance, efficiency & reduce
costs
Use cases
 Production reporting
 Historical comparisons
 Customer analysis (next best offer,
segmentation,
life-time value scores,
churn analysis, etc.)
 KPI calculations
 Profitability analysis
 Forecasting
Traditional EDW
environment
Data
refinery
Data integration
platform
BI services

Investigative Computing Use
Cases
New technologies used here
include:
 Hadoop, in-memory computing,
columnar storage, data
compression, appliances, etc.
Use cases
 Data mining and predictive
modeling for EDW and real-
time environments
 Cause and effect analysis
 Data exploration (“Did this ever
happen?” “How often?”)
 Pattern analysis
 General, unplanned
investigations of data
14
Data
refinery
Data integration
platform
RT analysis engine
platform
Operational systems
BI services

All Components Must Work Together
Data Virtualization is Mandatory
15
analytic models
analyses
New sources of data Enterprise DW
Analytic tools
Investigative
computing platform
Data refinery Operational systems
existing
customer
data
next best
customer offer
3rd party data
location data
social data
feedback
RT analysis engine
call center dashboard
or web event stream
Slide created by Colin White – BI Research, Inc.

Need for Analytics
 Definition:
 Practice of iterative, methodical exploration of an organization’s
data with emphasis on [advanced] analytical techniques
 Business analytics are used by organizations committed to data-
driven decision-making
 Need:
 Analytics give us far more value from our data than simple
reporting or comparative diagnostics
 They are the only meaningful way to measure success or failure
 They give us more than just descriptions of what happened – why
did it happen, will it continue to happen, what should I do to either
stop it or continue the activity?
16

Four Forms of BI
17
Based on Delen, Dursun and Demirkan, Haluk, “Decision Support Systems, Data, information and analytics as services,”
from Elsevier, published online May 29, 2012
Business Analytics
Descriptive
(Reactive)
Prescriptive
(Proactive)
Predictive
(Proactive)
What happened?
What is happening?
•Business reporting
•Dashboards
•Scorecards
•Data warehousing
Well-defined
business problems
and opportunities
What will happen?
•Data mining
•Text mining
•Web/media mining
•Forecasting
Accurate projections
of the future states
and conditions
What should I do?
Why should I do it?
•Optimization
•Simulation
•Decision modeling
•Expert systems
Best possible
business decisions
and transactions
OutcomesEnablersQuestions
Diagnostic
(Reactive)
Why did it happen?
•Behavioral analysis
•Cause and effect
analysis
•Correlations
Cause and effects of
changes in business
activities

Agenda
 Things to Ponder
18

Things to Think About
 Understand advantages and disadvantages of data
virtualization
 Advantages:
 Quick and fast access to any data
 No physical movement of data needed
 Low or no latency in accessing data
 Disadvantages
 Data virtualization does not replace ETL for EDWs
 It can impact performance of operational systems
 If data quality and data transformations are complex (e.g.,
multi-path), data virtualization is not recommended
19

Things to Think About
 Understand security needs in a virtual world
 Data virtualization can give data architects a “choke point” to
enforce security policies
 Understand your failover and scale-up requirements
 Eliminate rogue or unneeded data marts
 The benefit of data virtualization is the reduced need of physical
instantiations of data
 Create virtual marts as a standard practice unless there is a
compelling reason for a physical one
 Integrate cloud and on-premises sources virtually
 Be sure you can virtualize relational and non-relational
data sources together
20

Business Is In The Driver’s
Seat
 Self-service BI – used to expand BI throughout the enterprise but…
 IT must be recognized as being important to the business
 A company that puts no thought into information management and
analysis won’t be around for long
 IT is a significant partner and enabler to business strategies
 Business must have healthy relationship with IT professionals – most
important aspect of becoming a data-driven company
 Business must be recognized as technologically-savvy
 Emergence of super-analyst: someone highly
skilled, highly empowered, and highly productive
when set free
 Analysts prefer using their own tools instead of
ones blessed by IT & sanctioned by the
organization
 Virtualization is an important technology here
21

Future: The Customer is in the
Driver’s Seat!
 Caution! Paradigm shift ahead!
 Customer’s mobile phone becoming their mobile wallet - and their
personal data warehouse
 When customers interact with companies, they get a copy of “their” data
 Shopping information
 Financial information
 Medical information
 Phone / Text information
 Only they have the 360 degree view
of their own data
 Questions
 Can they monetize their information?
 Can they put their needs out to bid?
 Can they virtualize their own data?
22

Final Thoughts
23
Need fast time to value to gain business benefits from big data technologies
o Impractical to use traditional enterprise DW approach for all solutions
o Need to extend the existing DW environment to support new capabilities
Need high performance solutions for supporting new BI analytic workloads
o One-size fits all data management is no longer viable
o Match technologies and costs to business needs and analytic workloads
Need to modify data modeling and integration approaches
o Need to support new data types, sources and platforms, and new approaches such
as data blending, schema-on-read and data refineries
Need to modify data governance approaches
o No longer practical to rigidly control and govern all forms of data – use different
levels of governance based on security, compliance, quality and retention needs

Agenda
24

Extended Data Warehouse Architecture
– Real World Examples

Extended Data Warehouse Architecture -
Recap
Traditional EDW
environment
platform
Slide created by Colin White – BI Research, Inc.
BI services
Data
refinery
Data integration
platform

Analyzes weather data to provide insurance to farmers who can lock in profits even in
the case of drought, excessive rains or other adverse weather conditions
• Aggregation of very large dynamic data sets
• Mix of cloud, web and internal data
• Excel and Tableau to generate reports on risk assessment, and recommend
coverage/price policies

Deployment in Amazon EC2 Cloud

Telematics - IoT
Data services access to machine generated data. Business use: predictive
maintenance services
Major Heavy Equipment Manufacturer

Telematics Project
Hadoop Cluster
OSI PI
Dealer
Maintenance
Parts
Inventory
Virtual Views
Dealer/Customer
Dashboards

Summary – Things to Remember
■ The traditional enterprise Data Warehouse
architecture needs to evolve to embrace Big Data
■ However this doesn’t mean that the enterprise
Data Warehouse is not needed
■ Match the technologies and costs to the business
needs
■ You will need to combine data from both the
traditional and new environments
■ Data Virtualization will be an essential component
of the Extended Data Warehouse

Data Virtualization – Next Steps
Move forward at your own pace
 Download Denodo Express –
The fastest way to Data Virtualization
 Denodo Community:
Documents, Videos, Tutorials, and more.
Move forward with one of our Data
Virtualization experts
 Phone: (+1) 877-556-2531 (NA)
 Phone: (+44) (0)20 7869 8053 (EMEA)
 Email: info@denodo.com | www.denodo.com

www.denodo.com info@denodo.com

Extended Data Warehouse - A New Data Architecture for Modern BI with Claudia Imhoff

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Extended Data Warehouse - A New Data Architecture for Modern BI with Claudia Imhoff (20)

More from Denodo (20)

Recently uploaded (20)

Extended Data Warehouse - A New Data Architecture for Modern BI with Claudia Imhoff