Scribd
Upload
39 views

Pyspark Cashing & Persisting - Complete Guide

Uploaded by

Richard Smith
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
39 views

Pyspark Cashing & Persisting - Complete Guide

Uploaded by

Richard Smith
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

In PySpark, both caching and persisting are strategies to improve the performance of your Spark jobs by

storing intermediate results in memory or disk. Understanding the difference between caching and persisting
is important for optimizing the performance of applications that involve heavy data transformations and
iterative computations.

1. Caching in PySpark #
Caching is a way to store a DataFrame (or RDD) in memory for future operations. By default, when you
cache a DataFrame or RDD, Spark stores it in the memory of executors. If the dataset cannot fit in memory,
Spark recomputes the remaining partitions from the original source when required.

Characteristics of Caching: #

Stores the data only in memory by default (MEMORY_ONLY).


Data stored in memory can be retrieved much faster, improving job performance for iterative
algorithms.
Suitable for smaller datasets or computations that involve multiple transformations on the same
DataFrame.

How to Use Caching: #

from pyspark.sql import SparkSession

# Initialize Spark session


spark = SparkSession.builder \
.appName("Caching Example") \
.getOrCreate()

# Sample data
data = [(1, "Alice"), (2, "Bob"), (3, "Charlie"), (4, "David")]

# Create a DataFrame
df = spark.createDataFrame(data, ["id", "name"])

# Perform transformations
df_transformed = df.withColumn("id_squared", df["id"] ** 2)

# Cache the DataFrame


df_transformed.cache()

# Action to trigger caching


df_transformed.show()

2. Persisting in PySpark #

Persisting is more flexible than caching because it allows you to store data in various storage levels,
including both memory and disk. Unlike caching, where data is stored in memory only by default, persisting
lets you specify different storage levels, such as:

MEMORY_ONLY: Stores the RDD/DataFrame in memory only.


MEMORY_AND_DISK: Stores data in memory, but spills it to disk if memory is insufficient.
DISK_ONLY: Stores data only on disk.
MEMORY_ONLY_SER: Similar to MEMORY_ONLY, but serialized (reduces memory usage but increases
CPU overhead).
MEMORY_AND_DISK_SER: Serialized format, stores in memory, spills to disk if necessary.

Characteristics of Persisting: #

You can control the storage level more granularly compared to caching.
Suitable for larger datasets or situations where memory might be limited.
Provides fault tolerance by spilling data to disk when memory is insufficient.

How to Use Persisting: #

from pyspark import StorageLevel

# Persist the DataFrame with MEMORY_AND_DISK storage level


df_transformed.persist(StorageLevel.MEMORY_AND_DISK)

# Action to trigger persisting


df_transformed.show()

3. Comparing Caching and Persisting #

Feature Caching Persisting


Stores data in memory No default, user can choose storage
Default Behavior
(MEMORY_ONLY) level
Less flexible, only memory by More flexible (memory, disk,
Storage Flexibility
default serialization)
Usage Recommended for smaller datasets Recommended for large datasets
Performance Slightly slower if disk or serialization
Fastest when data fits in memory
Impact is used
Limited (recomputation for spilled Provides fault tolerance when using
Fault Tolerance
data) disk

4. Storage Levels in Detail #


Here are the storage levels available for persisting:

1. MEMORY_ONLY:
Stores data in memory. If it does not fit, recomputes the remaining partitions.
Use Case: Suitable for small datasets that can fit into memory.
pythonCopy codedf_transformed.persist(StorageLevel.MEMORY_ONLY)
2. MEMORY_AND_DISK:
Stores data in memory but spills it to disk if there is insufficient memory.
Use Case: Ideal for datasets that may not entirely fit in memory.
pythonCopy codedf_transformed.persist(StorageLevel.MEMORY_AND_DISK)
3. DISK_ONLY:
Stores data on disk only. This storage level is slower but useful when memory is a constraint.
Use Case: Suitable for large datasets where memory is limited.
pythonCopy codedf_transformed.persist(StorageLevel.DISK_ONLY)
4. MEMORY_ONLY_SER:
Stores the data in memory in serialized form, reducing memory consumption at the cost of
additional CPU usage.
Use Case: Good for memory-limited scenarios where the overhead of serialization is acceptable.
pythonCopy codedf_transformed.persist(StorageLevel.MEMORY_ONLY_SER)
5. MEMORY_AND_DISK_SER:
Similar to MEMORY_AND_DISK, but stores data in serialized format to save memory.
Use Case: Suitable for datasets that are large and may not fit in memory in their raw form.
pythonCopy codedf_transformed.persist(StorageLevel.MEMORY_AND_DISK_SER)

5. Code Example: Caching vs Persisting #


from pyspark import StorageLevel
from pyspark.sql import SparkSession
# Initialize Spark session
spark = SparkSession.builder \
.appName("Caching vs Persisting Example") \
.getOrCreate()

# Sample data
data = [(1, "Alice"), (2, "Bob"), (3, "Charlie"), (4, "David")]

# Create a DataFrame
df = spark.createDataFrame(data, ["id", "name"])

# Perform a transformation
df_transformed = df.withColumn("id_squared", df["id"] ** 2)

# Cache the DataFrame


df_transformed.cache()

# Persist the DataFrame with MEMORY_AND_DISK storage level


df_transformed.persist(StorageLevel.MEMORY_AND_DISK)

# Action to trigger caching or persisting


df_transformed.show()

# Count the number of rows to further trigger cached/persisted data


print(f"Row count: {df_transformed.count()}")

# Unpersist the DataFrame when done to free resources


df_transformed.unpersist()

6. When to Use Caching vs Persisting #


Caching is ideal when:
Your dataset is small enough to fit entirely in memory.
You need quick access to the data and want to avoid recomputing transformations.
Your application involves iterative algorithms like machine learning or graph processing.
Persisting is ideal when:
Your dataset is large and cannot fit in memory.
You need more control over how data is stored (e.g., disk, memory, or a combination).
You want to ensure fault tolerance, especially in long-running jobs.

7. Best Practices for Caching and Persisting #

Monitor Memory Usage: Use Spark’s web UI to monitor how much memory your job is using and
adjust caching/persisting accordingly.
Unpersist Data: Always unpersist cached or persisted DataFrames once you’re done with them to free
up resources.pythonCopy codedf_transformed.unpersist()
Use Serialization with Large Datasets: If you are working with large datasets, consider using
MEMORY_ONLY_SER or MEMORY_AND_DISK_SER to reduce memory usage.
Use Caching for Iterative Workloads: Caching is a good choice when you perform multiple actions
on the same DataFrame, as it avoids recomputing transformations repeatedly.

Conclusion #
Caching and persisting are two important strategies for optimizing Spark applications, especially when
dealing with large datasets and repeated transformations. While caching is simpler and faster, persisting
provides more flexibility and fault tolerance. Choosing between them depends on the size of your data,
memory constraints, and the specific needs of your application.

You might also like