Benchmarking
1 like•1,471 views
The document discusses benchmarking Hadoop performance and discusses several challenges: estimating Hadoop performance on hardware and clusters, designing Hadoop-ready servers and clusters, and optimizing networks for Hadoop. It also discusses a customer request to benchmark Hadoop Sort on a 100GB dataset and considers other benchmark tests like PageRank, RAM and CPU intensive tests, and tests that perform seeks in files. Finally, it discusses measuring network traffic related to specific Hadoop jobs, using small test datasets to predict performance on large datasets, and recommendations for Hadoop-ready server, rack, and container hardware designs.
1 of 10
Downloaded 87 times
More Related Content
PPTX
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...DataWorks Summit/Hadoop SummitNear Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...DataWorks Summit/Hadoop Summit
What's hot (20)
Ad
Viewers also liked (19)
Ad
Similar to Benchmarking (20)
PPTX
Performance evaluation of cloud-based log file analysis with Apache Hadoop an...Kishor Datta GuptaPDF
BDM37: Hadoop in production – the war stories by Nikolaï Grigoriev, Principal...Big Data Montreal PPTX
Introduction To Big Data with Hadoop and Spark - For Batch and Real Time Proc...Agile Testing AlliancePPTX
A modern, flexible approach to Hadoop implementation incorporating innovation...DataWorks SummitIntroduction To Big Data with Hadoop and Spark - For Batch and Real Time Proc...Agile Testing Alliance
More from Steve Loughran (20)
Recently uploaded (20)
DOCX
India's Emerging Global Leadership in Sustainable Energy Production The Rise ...Insolation EnergyBenchmarking
- 1. Benchmarking Steve Loughran Julio Guijarro © 2009 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
- 2. Benchmarks
- 3. Some Problems • Estimating Hadoop performance of hardware • Estimating Hadoop performance of a cluster • Designing Hadoop-ready servers • Designing Hadoop-ready clusters • Optimising the network for Hadoop • Optimising Hadoop/HDFS for specific applications
- 5. Recent customer request "They want data for Hadoop Sort for 100GB."
- 6. Terasort: what else? • PageRank: CPU intensive, small (static) input dataset • Something that stresses RAM and CPU • Something that seeks in the files?
- 7. Test Datasets • Wikipedia: 5-10 TB of XML data with changes; user relationships have to be inferred • SpamAssassin: 70+ GB of SPAM • Physics? Something Small?
- 8. Network Measurement What to add to Hadoop/Avro/Thrift to monitor network traffic -and relate to specific jobs?
- 9. Predicting performance Can an MR job on small datasets predict performance on full size datasets? What extra instrumentation can help?
- 10. Hardware Q What should a Hadoop-ready server look like? What about a rack? Or a container?