Hadoop Demo eConvergence

Demo
BIGDATA Hadoop
eConvergence Inc.
www.econvergenceinc.com
Rajesh Kalasapati PMP
Data Architect
raj@econvergenceinc.com

What you learn from TODAY BIGDATA DEMO
 Evolution of Business Analytics
 Evolution of Data processing and Traditional RDBMS
systems
 Company data volumes and storage
 Architecture of Traditional RDBMS
 What is Hadoop?
 Why to use Hadoop?
 Hadoop Architecture
 Hadoop USE Cases

Analytics Evolution
 Questions
 Is Hadoop a replacement or compliment for R-DBMS today?
 How does this change in future?
 How does this differ from current RDBMS and MPP systems

Company Average Data volumes..

Data size table
 1,024 (e.g. one Kilobyte = 1,024 bytes).
 Bit = 1 bit
 Byte = 8 bits
 Kilobyte = 1024 bytes
 Megabyte = 1024 kilobytes
 Gigabyte = 1024 megabytes
 Terabyte = 1024 gigabytes
 Petabyte = 1,048,576 gigabytes
 Exabyte = 1,073,741,824 gigabytes
 Zettabyte = 1,099,511,627,776 gigabytes

RDBMS Products Architecture
Data
filesData
files
Data
files
Node 1
Process area
Background
processes
Node 2
Process area
Background
processes
Node N
Process area
Background
processes
Data
filesData
files
Data
files

Logical and Physical ..
Database
Logical Physical
Tablespace Data file
OS block
Segment
Extent
Oracle data
block
Schema

 Yes
 Relational
Databases
(RDBMSs) have
been around for
ages
 MySQL is the most
popular among
them
 Data stored in
tables
 Schema-based,
i.e., structured
tables
 Queried using SQL
SQL queries: SELECT user_id from users WHERE
username = “jbellis”
Example’s Source

What is Hadoop?
 Hadoop is a data storage and processing system
 Scalable, fault-tolerant and distributed
 Able to store structured or unstructured data
 Stores terabytes, petabytes of data inexpensively
 Robust and reliable and handles hardware and system failures automatically
WITHOUT loosing data or interrupting data analysis
 Hadoop runs on clusters of commodity servers. Each of those servers has local
CPU and storage

What is Commodity Hardware?
 In general:
 Average amount of computing resources.
 Does not imply low quality, but rather affordability
 Feature of commodity hardware is that over time it is widely used in roles for which it was not specifically
designed, as opposed to purpose-built hardware
 In context of Hadoop
 Hadoop clusters are run on servers
 Most commodity servers used in Hadoop clusters have average ratio of disk space to memory, as opposed to
being specialized servers with massively high memory or CPU
 Examples of commodity hardware
 1 TB hard disks in a JBOD(Just Bunch of Disk) configuration
 Two quad core CPUs, running at least 2-2.5 GHz
 16-24GBs of RAMs (24-32 GB if you are using Hbase)
 1 GB ethernet
 Powerful cluster
 Six 2TB hard disks with RAID 1 across two of the disks
 Two quad core CPUs
 32-64 GBs of ECC(Error correcting Code) RAM
 2-4 GB of ethernet

Goals of Hadoop-HDFS
 Very large distributed file system
 10K Nodes, 100 million files, 10 PB size
 Assumes Commodity Hardware
 Files are replicated to handle hardware failures
 Detect failures and recovers from them
 Optimized Batch processing
 Process will move to data storage than the data move to process in traditional DBs
 Provides very high aggregate bandwidth

Cat
Bat
Dog
Other
Words
(size:
TByte)
map
map
map
map
split
split
split
split
combine
combine
combine
reduce
reduce
reduce
part0
part1
part2
MapReduce
09/29/13
20

Traditional DWH Eco System
HR, Sales etc..
Information Sources
Operational DBs
(RDBMS)
Tools for extraction
Informatica / DataStage etc..
Oracle,
Teradata,
SQL Server,
Etc..
EDW
Data Marts
OLAP Servers
Supports
OLAP Tools
for queries
Data Mining
Client Tools
(Micro Strategy,
SAS,
SAP BO,
OBIEE,
Hyperian
Etc..)

Hadoop Eco System
HR, Sales etc..
Information Sources
Operational DBs
(RDBMS)
Oracle,
Teradata,
SQL Server
Etc..
EDW
Tools for extraction
Informatica / DataStage
Pig, Sqoop etc..
Datameer
Karmasphere
MicroStrategy
Platfora
Qlikview
Tableau
Tresata

BigData: Use Cases 1 – Risk Modeling
 How can banks better understand customers and markets?
 The Summary
 A large bank took separate data warehouses from multiple departments and combined them into a single
global repository in Hadoop for analysis. The bank used the Hadoop cluster to construct a new and more
accurate score of the risk in its customer portfolios. The more accurate score allowed the bank to manage its
exposure better and to offer each customer better products and advice. Hadoop increased revenue and
improved customer satisfaction.
 The Challenge
 A very large bank with several consumer lines of business needed to analyze customer activity across
multiple products to predict credit risk with greater accuracy. Over the years, the bank had acquired a
number of regional banks. Each of those banks had a checking and savings business, a home mortgage
business, credit card offerings and other financial products. Those applications generally ran in separate
silos—each used its own database and application software. As a result, over the years the bank had built up
a large number of independent systems that could not share data easily. With the economic downturn of
2008, the bank had significant exposure in its mortgage business to defaults by its borrowers. Understanding
that risk required the bank to build a comprehensive picture of its customers. A customer whose direct
deposits to checking had stopped, and who was buying more on credit cards, was likely to have lost a job
recently. That customer was at higher risk of default on outstanding loans as a result.

BigData: Use Cases 2 – CUSTOMER CHURN ANALYSIS
 Why do companies really lose customers?
 The Summary
 A large telecommunications provider analyzed call logs and complex data from multiple sources. It used
sophisticated predictive models across that data to predict the likelihood that any particular customer would
leave. Hadoop helped the telecommunications company build more valuable customer relationships and
reduce churn.
 The Challenge
 A large mobile carrier needed to analyze multiple data sources to understand how and why customers
decided to terminate their service contracts. Were customers actually leaving, or were they merely trading
one service plan for another? Were they leaving the company entirely and moving to a competitor? Were
pricing, coverage gaps, or device issues a factor? What other issues were important, and how could the
provider improve satisfaction and retain customers?

BigData: Use Cases 3 – RECOMMENDATION ENGINE
 How can companies predict customer preferences?
 The Summary
 A leading online dating service uses sophisticated analyses to measure the compatibility between individual
members, so that it can suggest good matches for a potential relationship. Hadoop helped customers find
romance.
 The Challenge
 When users sign up for the dating service, they fill in surveys that describe themselves and what they look
for in a romantic partner. The company combined that information with demographic and web activity to build
a comprehensive picture of its customers. The data included a mix of complex and structured information,
and the scoring and matching algorithms that used it were complex. Customers naturally wanted better
recommendations over time, so the analytical system had to evolve continually with new techniques for
assessing romantic fit. As the company added new subscribers, the amount of data it managed grew, and
the difficulty of comparing every possible pair of romantic partners in the network grew even faster.

BigData: Use Cases 4 – AD TARGETING
 How can companies increase campaign efficiency?
 The Summary
 Two leading advertising networks use Hadoop to choose the best ad to show to any given user.
 The Challenge
 Advertisement targeting is a special kind of recommendation engine. It selects ads best suited to a particular
visitor. There is, though, an additional twist: each advertiser is willing to pay a certain amount to have its ad
seen. Advertising networks auction ad space, and advertisers want their ads shown to the people most likely
to buy their products. This creates a complex optimization challenge.

BigData: Use Cases 5 – POINT-OF-SALE TRANSACTION ANALYSIS
 How do retailers target promotions guaranteed to make you buy?
 The Summary
 A large retailer doing Point-of-Sale transactional analysis needed to combine larger
quantities of PoS transaction analysis data with new and interesting data sources to forecast
demand and improve the return that it got on its promotional campaigns. The retailer built a
Hadoop cluster to understand its customers better and increased its revenues.
 The Challenge
 Retail analytics has been a core part of the data warehousing industry and has helped to
drive its growth. Today, retailers are able to collect much more data about their customers,
both in stores and online. Many want to combine this new information with recent and
historical sales data from PoS systems to increase sales and improve margins. Legacy data
warehousing systems are an expensive place to store complex data from new sources. They
do not, generally, support the kind of sophisticated analyses - sentiment, language
processing and others—that apply to this new data.

BigData: Use Cases 6 – ANALYZING NETWORK DATA TO PREDICT
FAILURE
 How can organizations use machine generated data to identify potential trouble?
 The Summary
 A very large public power company combined sensor data from the smart grid with a map of
the network to predict which generators in the grid were likely to fail, and how that failure
would affect the network as a whole.
 The Challenge
 Utilities run big, expensive and complicated systems to generate power. Each of the
generators includes sophisticated sensors that monitor voltage, current, frequency and other
important operating characteristics. Operating a single generator means paying careful
attention to all of the data streaming off of the sensors attached to it.
 Utilities operate many of these generators spread across multiple locations. The locations are
connected to one another, and then each utility is connected to the public power grid.
Monitoring the health of the entire grid requires capture and analysis of data from every
utility, and even from every generator, in the grid.
 The volume of data is enormous. A clear picture of the health of the grid depends on both
real-time and after-the-fact forensic analysis of all of it. Spotting facilities at risk of failure
early, and doing preventive maintenance or separating them from the grid, is critical to
preventing costly outages.

BigData: Use Cases 7 – THREAT ANALYSIS
 How can companies detect threats and fraudulent activity?
 The Summary
 Businesses have struggled with theft, fraud and abuse since long before computers existed.
Computers and on-line systems create new opportunities for criminals to act swiftly,
efficiently and anonymously. On-line businesses use Hadoop to monitor and combat criminal
behavior.
 The Challenge
 Online criminals write viruses and malware to take over individual computers and steal
valuable data. They buy and sell using fraudulent identities and use scams to steal money or
goods. They lure victims into scams by sending email or other spam over networks. In “pay-
per-click” systems like online advertising, they use networks of compromised computers to
automate fraudulent activity, bilking money from advertisers or ad networks.
 Online businesses must capture, store and analyze both the content and the pattern of
messages that flow through the network to tell the difference between a legitimate
transaction and fraudulent activity by criminals.

BigData: Use Cases 8 – TRADE SURVEILLANCE
 How can a bank spot the rogue trader?
 The Summary
 A large investment bank combines data about the parties that participate in a trade with the
complex data that describes relationships among those parties and how they interact with
one another. The combination allows the bank to recognize unusual trading activity and to
flag it for human review. Hadoop allows the bank to spot and prevent suspect trading activity.
 The Challenge
 The bank already captured trading activity and used that data to assess, predict, and
manage risk for both regulatory and non-regulatory purposes. The very large volume of data,
however, made it difficult to monitor trades for compliance, and virtually impossible to catch
“rogue” traders, who engage in trades that violate policies or expose the bank to too much
risk.
 The risk is enormous. At Barings Bank in 1995, a single trader named Nick Leeson made an
escalating series of money-losing trades in an attempt to cover losses from earlier ones. The
final cost to the bank, at nearly $1.3Bn, forced Barings out of business.

BigData: Use Cases 9 – SEARCH QUALITY
 What’s in your search?
 The Summary
 A leading online commerce company uses Hadoop to analyze and index its data and to
deliver more relevant, useful search results to its customers.
 The Challenge
 Good search tools have been a boon to web users. As the amount of data available online
has grown, organizing it has become increasingly difficult. Users today are more likely to
search for information with keywords than to browse through folders looking for what they
need.
 Good search tools are hard to build. They must store massive amounts of information, much
of it complex text or multimedia files. They must be able to process those files to extract
keywords or other attributes for searches. The amount of data and its complexity demand a
scalable and flexible platform for indexing.
 Besides the difficulty of handling the data, a good search engine must be able to assess the
intent and interests of the user when a search query arrives. The word “chips” in a query may
refer to fried food or to electronic components. Delivering meaningful results requires that the
system make a good guess between the two. Looking at the user’s recent activity and history
can help.

BigData: Use Cases 10 – Data sandbox
 What can you do with new data?
 The Summary
 Companies--even those with established enterprise data warehouses—often need a cost-
effective, flexible way to store, explore and analyze new types of complex data. Many
companies have created “data sandboxes” using Hadoop, where users can play with data,
decide what to do with it and determine whether it should be added to the data warehouse.
Analysts can look for new relationships in the data, mine it and use techniques like machine
learning and natural language processing on it to get new insights.
 The Challenge
 The variety, complexity and volume of data available to enterprises today are changing the
way that those enterprises think. There is value and insight locked up in complex data. The
best companies in the world are unlocking that value by building new analytical capacity.
 With shifting and complex data and emerging analytics, enterprises need a new platform to
store and explore the information they collect. Cost, scalability and flexibility are all forcing a
move away from the single-architecture data warehouse of the past, toward a more flexible
and comprehensive data management infrastructure.

Project handled……!!!!!!!!!!!

eConvergence Inc. 44 09/29/13
eConvergence, Inc
eConvergence Inc. is a software
development company and provides
cost-effective customer-centric IT
consulting services for customers
across the globe.

Strengths @ eConvergence
 Proven and experienced management team.
 Expertise in business transformation
outsourcing.
 Extensive industry expertise.
 Broad and evolving service offerings.

Key Domains/ Technology Expertise
 Data Warehousing
 Teradata
 Web Developments
 Business Intelligence
 Mobile Developments (iOS and Android)
 Manual & Automation testing
 Business Analyst
 Data Stage/Informatica
 BIGDATA/Hadoop
 PMP

Training Mode
 Onsite
 Online
 Batch size is limited
 Sessions are Live
 Sessions > Interactive

Batch Timings
Generally, the trainer covers 6 hrs/week
Weekdays
 9:00-11:00pm EST (6:00-8:00pm PST)
 Three days a week
Weekends
 3 hours each on Sat & Sunday
Based on Course, it can be 6-8Weeks

Technologies to Assist…
We use all the latest Training Methodologies to
make you learn in easier and best way:

GotoMeeting/Webex Conf. Software.
 Daily PPTs/Exercises forwarded after
the session.
 Recording of the sessions.
 Queries to be posted in Yahoo group.

Installation/Softwares
Our technical team will assist you in installation
of software required for training. :
 Installation in Desktop/Laptop.
 Server support for certain courses.
 Hands on Assistant.
 Technical Assistance.

Post training

Resume Preparation
 Interview Preparation
 Marketing Support
 Post Placement Support

Immigration Support
We provide assistance in all immigration
activities such as:
 H1 Transfers
 New H1s
 GC Processing
 eVerify
 F1 OPTs

Course Contents
 Session- 1
 DW Definition, DW Architecture
 Operation Data Bases(ODB)
 Data Modeling 3NF and Dimensional Modeling
 OLTP and OLAP, ETL concepts
 Top Down/Bottom up approaches
 Bill Inmon approach – advantages and
disadvantages
 Ralph Kimbell approach – advantages and
disadvantages
 Star and Snowflake schemas
 Dimension modeling design considerations
 Normalization techniques with live examples
 Data Mart project examples
 Customer, Products, Geo dimensional concepts
 Hierarchy structures
 Master Data Management systems
 Information Management systems
 NO SQL – BIG Data, ACID Model, CAP Model
 Session- 2
 HADOOP Architecture
 HDFS Architecture
 HDFS Features
 Intro Name node & Data Node
 File storage & Replication
 Build HADOOP Cluster–EC2/AWS
 Hadoop Configuration
 Session- 3
 MapReduce
 MapReduce Features
 MapReduce Job recovery
 MapReduce Job Check
 Cluster Rebalancing
 Secondary Name Node features
 Practice Hadoop Commands

Course Contents
 Session- 4
 Introduction to Hive
 Installation of Hive
 Hive SQLs
 Hive internal and external tables
 Hive Partitions
 Introduction to SQOOP
 Installation of Sqoop
 Sqoop practice with Hadoop and HBase
 Session- 5
 Introduction to Pig
 Installation of Pig
 Pig Relations, Bags, Tuples, Fields
 Pig- expressions
 Pig- Schemas
 Pig- Join and Split Optimization
 Pig- JSON
 Pig-Co Groups
 Session- 6
 Introduction to HBASE
 Architecture
 Install Hbase
 Region Servers , Master
 Hbase with Hive
 Hbase with Sqoop
 Hbase with PIG
 Hbase practice
 Session- 7
 Installation on VM Ware
 Installation of CDH4
 Introduction to R-Language and Revo Analytics
 Session- 8
 Performance Tuning
 Certification discussion – CCA-410
 Practical Example

Hadoop Demo eConvergence

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Hadoop Demo eConvergence (20)

Recently uploaded (20)

Hadoop Demo eConvergence