Bay Area Hadoop User Group

Accelerated Analytics for the Big Data Fabric
Bay Area Hadoop User Group

© 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555

AGENDA

 The Big Data Fabric
 Big Data Preparation – An Everyday Challenge
 Use-Case Scenario – Call Volume Analysis
 Solution Requirements
 Solution Workflow
 Phase I - Data Preparation & Visualization
 Phase II - Pentaho MapReduce & Orchestration
 Summary

2

The Big Data Fabric

Data Integration Big Analytics
Pentaho Business Analytics 3rd Party Tools
R
Visualization Dashboards 3rd Party BI Tools
Interactive Analysis Reports Applications

Data Integration Scheduling
Job Orchestration High Performance
Workflow Visual IDE

Hadoop Analytic Databases
NoSQL Databases

Big Data Mgmt
3

Preparing Big Data for Analysis
is an Everyday Challenge

• Very technical skills required
• Divide between M-R developers & analysts
• Beyond the reach of many organizations

4

Pentaho Visual MapReduce

Accessible by any ETL
developer, business analyst or data
scientist

Executes inside Hadoop as a native
Java MapReduce task
5

Pentaho Reporting & Analytics

Batch Reporting
and Ad Hoc Query
Data Visualization, Discovery
and Analysis

Hadoop NoSQL Hybrid
6

Use Case Scenario – Call Volume Analysis

• VOIP service provider has excess capacity and is
considering expansion to consumer markets
• Business Analyst: what are the top 10 states for
inbound calls on Fridays, Saturdays and Sundays?
• Research data available:
– Call records – date/timestamp & destination phone #
?
– NANP (North American Numbering Plan) data – area
code by country, state & time zone

7

Solution Requirements

• Data Preparation
– Access the call records in HDFS
– Extract the destination area code for each call
– Read the area code reference data
– Lookup country, state and time zone by area code, append to each
record
– Filter out records (non-U.S. calls, calls made on M-Tu-W-Th)
– Load to a relational database
– Generate metadata
• Analysis
– Explore data multi-dimensionally
– Find the top-10 states by inbound call volume
– Navigate via a geospatial interface
• Deployment
– Deploy in MapReduce to handle larger data volumes

8

Solution Workflow

• Phase I - Business Analysts
– Use a data extract to prepare and validate their analyses
– Iterate over requirements with executives and stake-holders

• Phase II - MapReduce Developers/Analysts
– Create production Pentaho MapReduce transformations
– Manage the deployment and orchestration between the
Hadoop cluster and the production database

9

Data Preparation (Phase I)

• The data pipeline implements the data preparation logic
• Each component has a “personality”– access, calculate, join, filter …
• Free-form design
– As many or as few inputs, transformations and outputs as needed
• Schema contract exists only between connected components
• Pipelined, multi-threaded for performance
• 100% Java-based for deployment flexibility

10

Data Pipeline – Input from HDFS

11

Data Pipeline - Calculator

12

Data Pipeline – Stream Lookup

13

Data Pipeline – Row Filter

14

Data Pipeline – Table Output

15

Visualization – Multi-Dimensional UX

16

Visualization – Geographic

17

Visualization - Heatmap

18

Deployment to Hadoop (Phase II)

• To process a larger set of data we can deploy the data pipeline via
MapReduce
– Input and output streams are encoded in key-value pairs
– Two specialized components provide an interface:

– A special job component deploys the data pipeline to the Hadoop
cluster:

19

Pentaho MapReduce – Inputs/Outputs

The core logic of the data pipeline is
identical … only the ends change

........

20

Pentaho MapReduce – Orchestration

21

Instant Analytics (Roadmap)

Choose a Big Data Source,
Answer a Few Questions,
Publish to Pentaho

Report, Explore and
Analyze

Customize Model
(Optional)
22

SUMMARY

1. The Big Data Fabric encompasses a large collection of Hadoop
distributions, NoSQL and analytical databases
2. A component-based approach to data access and integration can:
– Allow business analysts and data scientists to perform their own data
preparation
– Result in more rapid validation of business requirements & metrics
– Be used to create data pipelines that can be deployed directly to a
cluster, enabling analytics against much larger data sets
– Support orchestration across environments

23

Summary

24

Thank You
Join the conversation. You can find us on:

http://blog.pentaho.com

@Pentaho

Facebook.com/Pentaho

Pentaho Business Analytics


Bay Area Hadoop User Group

Recommended

More Related Content

What's hot (20)

Similar to Bay Area Hadoop User Group (20)

More from Pentaho (14)

Bay Area Hadoop User Group

Editor's Notes