INTRODUCTION TO BIG DATA HADOOP
Download as PPTX, PDF1 like157 views
This document provides an overview of Big Data and Hadoop. It discusses how companies are generating large amounts of data daily. Hadoop was created to handle such large volumes of data across clusters of commodity hardware. Key aspects of Hadoop covered include its history, architecture, ability to scale out across clusters, and ability to process data in parallel across nodes. Hadoop also aims to abstract complexity and handle failures which are common given the large number of machines in clusters. The document compares Hadoop to relational databases and explains how Hadoop is better suited to semi-structured and unstructured data.
INTRODUCTION TO BIG DATA HADOOP
- 2. GOAL To learn about Big data and Hadoop Programming
- 3. AGENDA Big Data Hadoop Introduction History Comparison to Relational Databases Hadoop Eco-System and Distributions Resources
- 4. BIG DATA Information Data Corporation (IDC) estimates data created in 2010 to be Companies continue to generate large amounts of data, here are some 2011 stats: – Facebook ~ 6 billion messages per day – EBay ~ 2 billion page views a day, ~ 9 Petabytes of storage – Satellite Images by Skybox Imaging ~ 1 Terabyte per day
- 5. HADOOP Existing tools were not designed to handle such large amounts of data "The Apache™ Hadoop™ project develops open-source software for reliable, scalable, distributed computing." - http://hadoop.apache.org – Process Big Data on clusters of commodity hardware – Vibrant open-source community – Many products and tools reside on top of Hadoop
- 6. HADOOP JOBS
- 8. DATA STORAGE Storage capacity has grown exponentially but read speed has not kept up – 1990: • Store 1,400 MB • Transfer speed of 4.5MB/s • Read the entire drive in ~ 5 minutes– 2010: • Store 1 TB • Transfer speed of 100MB/s • Read the entire drive in ~ 3 hours • Hadoop - 100 drives working at the same time can read 1TB of data in 2 minutes
- 9. HADOOP CLUSTER A set of "cheap" commodity hardware • Networked together • Resides in the same location – Set of servers in a set of racks in a data center
- 10. USE COMMODITY HARDWARE “Cheap” Commodity Server Hardware – No need for super-computers, use commodity unreliable hardware – Not desktops
- 11. HADOOP SYSTEM PRINCIPLES Scale-Out rather than Scale-Up Bring code to data rather than data to code Deal with failures – they are common Abstract complexity of distributed and concurrent applications
- 12. SCALE OUT INSTEAD OF SCALE UP It is harder and more expensive to scale-up – Add additional resources to an existing node (CPU, RAM) – Moore’s Law can’t keep up with data growth – New units must be purchased if required resources can not be added – Also known as scale vertically • Scale-Out – Add more nodes/machines to an existing distributed application – Software Layer is designed for node additions or removal – Hadoop takes this approach - A set of nodes are bonded together as a single distributed system – Very easy to scale down as well
- 13. CODE TO DATA • Traditional data processing architecture – nodes are broken up into separate processing and storage nodes connected by high-capacity link – Many data-intensive applications are not CPU demanding causing bottlenecks in network Storage Node Processing Node Processing Node Storage Node Load Data Save Results Risk of bottleneck
- 14. CODE TO DATA Hadoop co-locates processors and storage – Code is moved to data (size is tiny, usually in KBs) – Processors execute code and access underlying local storage 16 Processor Storage Hadoop Node Processor Storage Hadoop Node Processor Storage Hadoop Node Processor Storage Hadoop Node
- 15. FAILURES ARE COMMON Given a large number machines, failures are common – Large warehouses may see machine failures weekly or even daily • Hadoop is designed to cope with node failures – Data is replicated – Tasks are retried
- 16. ABSTRACT COMPLEXITIES Hadoop abstracts many complexities in distributed and concurrent applications – Defines small number of components – Provides simple and well defined interfaces of interactions between these components • Frees developer from worrying about systemlevel challenges – race conditions, data starvation – processing pipelines, data partitioning, code distribution – etc. • Allows developers to focus on application development and business logi
- 17. HISTORY OF HADOOP Started as a sub-project of Apache Nutch – Nutch’s job is to index the web and expose it for searching – Open Source alternative to Google – Started by Doug Cutting • In 2004 Google publishes Google File System (GFS) and MapReduce framework papers • Doug Cutting and Nutch team implemented Google’s frameworks in Nutch • In 2006 Yahoo! hires Doug Cutting to work on Hadoop with a dedicated team • In 2008 Hadoop became Apache Top Level Project – http://hadoop.apache.org
- 18. NAMING CONVENTIONS Doug Cutting drew inspiration from his family – Lucene: Doug’s wife’s middle name – Nutch: A word for "meal" that his son used as a toddler – Hadoop: Yellow stuffed elephant named by his son
- 19. COMPARISON TO RDBMS Until recently many applications utilized Relational Database Management Systems (RDBMS) for batch processing – Oracle, Sybase, MySQL, Microsoft SQL Server, etc. – Hadoop doesn’t fully replace relational products; many architectures would benefit from both Hadoop and a Relational product(s) • Scale-Out vs. Scale-Up – RDBMS products scale up • Expensive to scale for larger installations • Hits a ceiling when storage reaches 100s of terabytes – Hadoop clusters can scale-out to 100s of machines and to petabytes of storage
- 20. COMPARISON TO RDBMS Structured Relational vs. Semi-Structured vs. Unstructured – RDBMS works well for structured data - tables that conform to a predefined schema – Hadoop works best on Semi-structured and Unstructured data • Semi-structured may have a schema that is loosely followed • Unstructured data has no structure whatsoever and is usually just blocks of text (or for example images) • At processing time types for key and values are chosen by the implementer – Certain types of input data will not easily fit into Relational Schema such as images, JSON, XML, etc...
- 21. END Thank you.