![Let’s see some of the frequently used RDD Transformations.
Map(func)
It returns a new distributed dataset formed by passing each element of the source
through a function func.
flatMap(func)
Here, each input item can be mapped to zero or more output items, so func should
return a sequence rather than a single item.
groupByKey([numPartitions])
It returns a dataset of (K, Iterable) pairs when called on a dataset of (K, V) pairs.
reduceByKey(func, [numPartitions])
When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the
values for each key are aggregated using the given reduce function func, which must be
of type (V,V) => V.](https://image.slidesharecdn.com/apachespark-220604085644-b6ab093a/85/Apache-Spark-12-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Apache Spark. (20)
Ad
Recently uploaded (20)
Ad
Apache Spark.
- 2. Spark 💜 Spark it’s used for data processing. 💜Apache Spark has been designed for quick computation by a simple cluster technology. 💜Apache Spark is another open source cluster computing framework for data analytics. However, Spark supports in-memory cluster computing, Parallel processing and promises to be Faster than Hadoop. 💜 Spark supports various high-level tools for data analysis. 💜 Spark provides APIs for Scala, Java and Python Language.
- 3. Spark Unifies 🍒 Batch processing 🍒 Real-time processing 🍒 Stream Analytics 🍒 Machine Learning 🍒 Interactive SQL 🍒 Graph processing
- 4. How does Apache Spark work?
- 6. Spark Core Spark Core provides distributed task dispatching, scheduling, basic I/O etc. All these functionalities are exposed through an application programming interface (for Java, Python, Scala, and R) called driver program.
- 8. Driver Program 🎈The Driver Program is a process that runs the main() function of the application and creates the SparkContext object. The purpose of SparkContext is to coordinate the spark applications, running as independent sets of processes on a cluster 🎈To run on a cluster, the SparkContext connects to a different type of cluster managers and then perform the following tasks: - 🎈It acquires executors on nodes in the cluster. 🎈Then, it sends your application code to the executors. Here, the application code can be defined by JAR or Python files passed to the SparkContext. 🎈At last, the SparkContext sends tasks to the executors to run.
- 9. Cluster Manager 💜The role of the cluster manager is to allocate resources across applications. The Spark is capable enough of running on a large number of clusters. 💜It consists of various types of cluster managers such as Hadoop YARN, Apache Mesos and Standalone Scheduler. 💜Here, the Standalone Scheduler is a standalone spark cluster manager that facilitates to install Spark on an empty set of machines. Worker node 💜The worker node is a slave node 💜Its role is to run the application code in the cluster.
- 10. Executor 👉An executor is a process launched for an application on a worker node. 👉It runs tasks and keeps data in memory or disk storage across them. 👉It read and write data to the external sources. 👉Every application contains its executor. Task 👉A unit of work that will be sent to one executor.
- 11. Resilient Distributed Datasets… ⭐ Spark Core is embedded with RDDs(Resilient Distributed Datasets), an immutable fault-tolerant, distributed collection of objects that can be operated on in parallel.
- 12. Let’s see some of the frequently used RDD Transformations. Map(func) It returns a new distributed dataset formed by passing each element of the source through a function func. flatMap(func) Here, each input item can be mapped to zero or more output items, so func should return a sequence rather than a single item. groupByKey([numPartitions]) It returns a dataset of (K, Iterable) pairs when called on a dataset of (K, V) pairs. reduceByKey(func, [numPartitions]) When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function func, which must be of type (V,V) => V.
- 13. Let’s see some of the frequently used RDD Actions. Reduce(func) It aggregate the elements of the dataset using a function func (which takes two arguments and returns one). The function should be commutative and associative so that it can be computed correctly in parallel.
- 14. Thank you..