Big Data Overview Part 1
Download as PPTX, PDF0 likes892 views
Big Data has become the new buzzword like “Agile” and “Cloud”. Like those two others, it’s a transformative technology. We’ll be discussing: •What is it? •Technology key words •HDFS •Hadoop •MapReduce This will be part 1 of 2 (at least). This first talk will not be overly technical. We’ll go over the concepts and terms you’ll encounter when considering a big data solution.
![Solution - Reduce
• Data is merged
• Merged into Key/Values:
{21, [2,000, 23,000]}
{23, [3,000]}
{2, [1,000]}
• You process each row](https://image.slidesharecdn.com/bigdatapart1-141031082523-conversion-gate01/85/Big-Data-Overview-Part-1-20-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (17)
Ad
Similar to Big Data Overview Part 1 (20)
Ad
Recently uploaded (20)
Big Data Overview Part 1
- 1. Big Data Overview – Part 1 Wm. Barrett Simms [email protected] @wbsimms
- 2. Opening remarks • Sponsors • Pluralsight • Free month gift card give away. Enter your name in the pot! • DevExpress • $250 in developer JustCode tools. • O’Reilly • Book give away. Enter your name in the pot! • Boston Code Camp 22 (November 22nd) • http://www.bostoncodecamp.com/ • Thanks to 3thought for the space
- 3. About Me Software Developer Agile Team Member Team Lead Agile Advocate SDLC Implementer
- 4. SDLC
- 5. Big Data “Big data is an all-encompassing term for any collection of data sets so large and complex that it becomes difficult to process using traditional data processing applications.” - Wikipedia
- 6. The 3 Vs • Volume • A few Gigabytes -> Petabyte • Velocity • Arrives quickly • Variety • Multiple Sources
- 7. Volume • Traditional SQL architectures don’t scale to very large • Maybe this isn’t so true …but the MMP systems are expensive
- 8. An example problem (Volume) • You own a chain of stores • … with 25,000 stores and 100,000 POS systems • Need information on inventory changes • By region • By store
- 9. Velocity • Traditional solutions don’t handle fast inbound data • Maybe this isn’t so true …but you lose data.
- 10. Another example (Velocity) • You host a website • … on 10,000 servers • Monitor logs for errors
- 11. Variety • Most traditional solutions don’t handle a variety of data types well • Maybe this isn’t so true …But you need to write a custom importer for every type.
- 12. A final example (Variety) • You own a business • With a sales and marketing teams • … in different regions around the world • Correlate sales numbers against marketing expenses
- 13. The First Problem : Computing Power First Second Third First Second Third First Second Third First Second Third First Second Third Limited by cores (Scaling up)
- 14. Solution: Scale out (not up!) Server 1 Server 2 Coordinator Server 3 Server 4
- 15. Coordination Job Coordinator Runner Runner Runner
- 16. MapReduce • A programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. – Wikipedia WHAT?
- 17. Map and Reduce • Map • Process data returning key value pairs • Reduce • Aggregate/Filter key value pairs into result Map Map Data Data Reduce Result
- 18. Mapping • Easy example • Store Sales • Find most sales per store in 2010 Year Month Store Id SalesTotal 2010 1 13 1,000 2010 3 43 12,000 2010 3 21 21,000 2010 4 13 3,000 2010 2 56 4,000 2010 6 32 12,000 2010 7 1 4,000 2010 2 23 2,000
- 19. Solution – Map 1. Mapper feeds document rows to your program 2. You return key value pairs StoreId Sales 21 2,000 23 3,000 2 1,000 21 23,000
- 20. Solution - Reduce • Data is merged • Merged into Key/Values: {21, [2,000, 23,000]} {23, [3,000]} {2, [1,000]} • You process each row
- 21. Data Access • Each process needs access to data Typical Desired
- 22. HDFS • Hadoop File System • Open-source implementation of the Google File System (GFS) Hard drives last about 1,000 days. So, if you have 1K hard drives, you’ll lose one per day.
- 23. The ecosystem • Hive • SQL-like query language • Define and enforce schema • Pig • SQL-like query language • Sqoop • SQL/Hadoop integration • Oozie • Scheduling • Mahout • Machine Learning interface • Storm • Stream-based MapReduce … and Many Others
- 24. Vendors • Hortonworks • Single click install of Sandbox • Cloudera • Downloadable VM • Syncfusion • Single click install of Syncfusion Big Data • Amazon AWS • Elastic MapReduce • Microsoft Azure • HDInsight
- 25. Contact Me Barrett Simms [email protected] http://wbsimms.com Twitter: @wbsimms Phone: 781.405.4686