Ad
More Related Content
What's hot (20)
Viewers also liked (14)
Ad
Similar to Introduction to apache spark (20)
Ad
Recently uploaded (20)
Introduction to apache spark
- 1. Introduction to Spark Eric Eijkelenboom - UserReport - userreport.com
- 2. • What is Spark and why should I care? • Architecture and programming model • Examples • Mini demo • Related projects
- 4. RTFM • A general-purpose computation framework that leverages distributed memory • More flexible than MapReduce (it supports general execution graphs) • Linear scalability and fault tolerance • It supports a rich set of higher-level tools including • Shark (Hive on Spark) and Spark SQL • MLlib for machine learning • GraphX for graph processing • Spark Streaming
- 5. Who cares?
- 6. ! ! ! ! • Slow due to serialisation & replication • Inefficient for iterative computing & interactive querying Limitations of MapReduce Input iter. 1 iter. 2 . . . HDFS read HDFS write HDFS read HDFS write Map Map Map Reduce Reduce Input Output
- 7. Leveraging memory iter. 1 iter. 2 . . . Input HDFS read HDFS write HDFS read HDFS write
- 8. Leveraging memory iter. 1 iter. 2 . . . Input iter. 1 iter. 2 . . . Input HDFS read HDFS write HDFS read HDFS write
- 9. Leveraging memory iter. 1 iter. 2 . . . Input iter. 1 iter. 2 . . . Input HDFS read HDFS write HDFS read HDFS write Not tied to 2-stage MapReduce paradigm 1. Extract a working set 2. Cache it 3. Query it repeatedly
- 10. So, Spark is… • In-memory analytics, many times faster than Hadoop/Hive • Designed for running iterative algorithms & interactive querying • Highly compatible with Hadoop’s Storage APIs • Can run on your existing Hadoop Cluster Setup • Programming in Scala, Python or Java
- 11. Spark stack
- 12. Architecture HDFS Datanode Datanode Datanode.... Spark Worker Spark Worker Spark Worker .... Cache Cache Cache Block Block Block Cluster Manager Spark Driver (Master)
- 13. Architecture HDFS Datanode Datanode Datanode.... Spark Worker Spark Worker Spark Worker .... Cache Cache Cache Block Block Block Cluster Manager Spark Driver (Master) • YARN • Mesos • Standalone
- 14. Programming model • Resilient Distributed Datasets (RDDs) are basic building blocks • Distributed collection of objects, cached in-memory across cluster nodes • Automatically rebuilt on failure • RDD operations • Transformations: create new RDDs from existing ones • Actions: return a value to the master node after running a computation on the dataset
- 15. As you know… • … Hadoop is a distributed system for counting words • Here is how it’s done is Spark
- 16. As you know… • … Hadoop is a distributed system for counting words • Here is how it’s done is Spark Blue code: Spark operations Red code: functions (closures) that get passed to the cluster automatically
- 17. Text search
- 18. Text search In memory text search: ! ! caches the RDD in memory for faster reuse
- 19. Logistic regression ! • 100 GB of data on a 100 node cluster
- 21. Spark shell
- 22. Mini demo
- 24. Hive on Spark = Shark • A large scale data warehouse system just like Hive • Highly compatible with Hive (HQL, metastore, serialization formats, and UDFs) • Built on top of Spark (thus a faster execution engine) • Provision of creating in-memory materialized tables (Cached Tables) • And cached tables utilise columnar storage instead of raw storage
- 25. Shark Shark uses the existing Hive client and metastore
- 26. MLlib • Machine learning library based on Spark ! ! • Supports a range of machine learning algorithms, including classification, regression, clustering, collaborative filtering, dimensionality reduction, and more
- 27. Spark Streaming • Write streaming applications in the same way as batch applications • Reuse code between batch processing and streaming • Write more than analytics applications: • Join streams against historical data • Run ad-hoc queries on stream state
- 28. Spark Streaming • Count tweets on a sliding window ! ! • Find words with higher frequency than historic data