Understanding Open Table Formats

Introduction
Imagine running a coffee shop that serves thousands of customers daily. You need an efficient system to
track sales, manage inventory, and analyze customer preferences. Now, think of a data lake as a massive
warehouse storing all this data, but with no proper labeling, organization, or real-time tracking. That’s
where Open Table Formats (OTFs) come in—they bring order to this chaos by making data lakes more
structured, efficient, and reliable.

What are Open Table Formats?

Open Table Formats (OTFs) are storage frameworks that add database-like features (transactions,
schema evolution, versioning) to data lakes. They help organizations manage large datasets efficiently
while ensuring consistency and reliability.

Think of it Like Google Drive vs. a Database

● A raw data lake is like a Google Drive folder—anyone can dump files, but searching for a specific
version or rolling back changes is tough.
● An Open Table Format is like a well-managed database—it keeps track of updates, ensures no
conflicts, and optimizes how data is stored and retrieved.

Why Do We Need Open Table Formats?

Problems with Traditional Data Lakes

Companies store vast amounts of data in data lakes, but Data Engineers (DEs) face major challenges
when providing this data to Data Scientists (DSs) and Data Analysts (DAs):

● No ACID Transactions → Imagine placing an online food order, but halfway through, your cart
resets. That’s what happens in a data lake when multiple users update data at the same time—
data inconsistencies arise.
● No Indexing → Searching data is like looking for a needle in a haystack—queries take forever.
● No Data Versioning → No way to track who changed what, leading to messy data.
● No Time Travel (Rollback) → Accidentally deleted records? Sorry, no way to restore them.
 ● No Schema Management → Different teams dumping data in different formats breaks existing
reports.

How Open Table Formats Fix This

OTFs solve these issues by bringing database-like features to data lakes:

ACID Transactions → No data corruption when multiple users update data.

Data Versioning → View previous versions (time travel feature).

Schema Evolution → Adjust schema without breaking queries.

Faster Queries → Built-in indexing speeds up data retrieval.

Secure Data Masking → Hide sensitive data for compliance.

Multi-Engine Compatibility → Works with Spark, Flink, Trino, etc.

Real-World Scenarios: Where Open Table Formats Shine

1. E-Commerce Transactions (Delta Lake)

Imagine Amazon tracking millions of orders daily. Customers might modify orders, return items, or
apply discounts. If one update fails, it shouldn't affect other transactions. Solution: Delta Lake
ensures ACID transactions, meaning partial updates won’t corrupt the data.

2. Financial Reporting & Compliance (Apache Iceberg)

Banks generate huge financial records that require tracking past transactions accurately. They also
need to manage schema changes without rewriting historical data. Solution: Apache Iceberg
supports time travel and schema evolution seamlessly.

3. Real-Time Fraud Detection (Apache Hudi)

A fintech app monitors millions of transactions per second for fraud. It needs real-time updates and
incremental processing to catch fraudsters instantly. Solution: Apache Hudi supports incremental
data ingestion, making real-time analytics possible.
Implementation of Open Table Formats

1. Delta Lake (Best for ACID Transactions & Query Performance)

from delta import *

deltaTable = DeltaTable.forPath(spark, "s3://data-lake/delta-table")

deltaTable.toDF().show()

Use when you need strong data consistency (e.g., financial transactions, e-commerce order
tracking).

2. Apache Iceberg (Best for Schema Evolution & Time Travel)

from iceberg.spark import *

df = spark.read.format("iceberg").load("s3://data-lake/iceberg-table")

df.show()

Use when you need historical data tracking (e.g., bank statements, audit logs).

3. Apache Hudi (Best for Real-Time & CDC Processing)

from pyspark.sql import SparkSession

hudiOptions = {

"hoodie.table.name": "hudi_table",

"hoodie.datasource.write.recordkey.field": "id",

"hoodie.datasource.write.precombine.field": "ts"

df.write.format("hudi").options(**hudiOptions).mode("append").save("s3://data-lake/hudi-table")

Use when you need real-time updates (e.g., fraud detection, social media analytics).
Choosing the Right Open Table Format
Feature Delta Lake Apache Iceberg Apache Hudi

ACID Transactions

Schema Evolution

Time Travel

Change Data Capture

Hidden Partitioning

Optimized for Streaming

When to use:

● Choose Delta Lake if you need strong data consistency & fast queries.
● Choose Apache Iceberg for historical tracking & complex analytics.
● Choose Apache Hudi for real-time updates & incremental processing.

At the end:
Open Table Formats revolutionize data lakes by making them structured, efficient, and reliable.
Choosing between Delta Lake, Apache Iceberg, and Apache Hudi depends on your use case:

● Need strong ACID transactions? → Use Delta Lake.

● Handling historical & evolving schemas? → Use Apache Iceberg.
● Working with real-time data? → Use Apache Hudi.

By leveraging these technologies, businesses can build scalable, high-performance data architectures
that drive insights and decision-making.
 Delta Lake - Detailed Explanation with Real-
World Scenarios
What is Delta Lake?
Imagine you run an online store like Amazon. Every second, thousands of customers are placing orders,
updating their profiles, or canceling purchases. What happens if multiple users update the same order at
the same time? Or if a system failure causes partial updates?

Delta Lake solves these problems by bringing ACID transactions, schema enforcement, and versioning
to data lakes. It ensures that your data remains consistent, reliable, and always recoverable, just like an
advanced order management system in an e-commerce platform.

Reading CSV Data into Spark

Before working with Delta Lake, let’s load sales data from a CSV file into a Spark DataFrame.

Example: Sales Data

Imagine you own a restaurant chain and collect daily sales records from multiple locations. These
records are stored in CSV files.

Code Example
ad.format('csv')

dt = spark.read.

option("header", "True").

load('datjes/File Store/opentable Format/rawdata (sales.csv')

display()
Real-World Scenario

Your restaurant receives thousands of orders daily. You store transaction details in a CSV file. But CSV
files are prone to errors (like missing values or incorrect formats). When loading this into Spark, Delta
Lake ensures data consistency and schema enforcement so you don’t end up with corrupted records.

Converting Data to Delta Format

Once data is loaded, we convert it into Delta format to make it more structured and reliable.

Code Example
df.write.format('delta')

.option("path", "Filestore/OpenTable format/Sink data/sales_data")

.save()

Real-World Scenario

Your restaurant’s sales data is growing daily. If you keep storing everything in raw CSV files, queries will
slow down over time. Converting to Delta format ensures your data is optimized for fast retrieval,
updates, and future scalability.

Delta Transaction Log

A Delta table maintains a transaction log that records all changes.

Example Files:

● _delta_log/000000.json (First transaction log)

● _delta_log/000001.json (Second transaction log)

Code Example
%fs head filestore/OpenTable Format/Sink data/sales_data/_delta_log/000000.json
Real-World Scenario

Imagine a banking system where customers withdraw and deposit money. If a system crash happens,
transactions shouldn’t be lost. Delta Lake’s transaction log acts like a banking ledger, ensuring every
update is tracked and can be recovered if needed.

Creating a Delta Table (Bronze Layer)

We can manually create a Delta table to store structured data.

Code Example
CREATE TABLE bronze.my_delta_table (

id INT,

name STRING,

salary DOUBLE

) USING DELTA

LOCATION '/filestore/OpenTableFormat/Sink data/bronze_my_delta_table';

Real-World Scenario

A hospital stores patient records (ID, name, medical history). Using Delta tables, data integrity is
enforced, preventing accidental overwrites or data corruption.

Schema Enforcement & Evolution

Schema enforcement prevents wrong data types from being inserted, while schema evolution allows
adding new columns dynamically.
Real-World Scenario

Imagine a social media app. Initially, users store their name and age, but later, you introduce a profile
picture feature. Schema evolution allows the new field to be added seamlessly without breaking
existing data.

Code Example
INSERT INTO bronze.my_delta_table VALUES (1, 'AA', 100), (2, 'BB', 200);

SELECT * FROM bronze.my_delta_table;

Optimization Features in Delta Lake

Checkpointing

● Stores transaction logs in compact form to improve query speed.

Deletion Vectors

● Instead of deleting data physically, Delta Lake marks it as deleted.

Real-World Scenario

Imagine a ride-sharing app (Uber, Lyft). When a driver deletes their profile, the app doesn’t remove it
immediately (in case of reactivation). Instead, Delta Lake marks it as deleted but keeps history for
analytics.

Code Example
ALTER TABLE bronze.my_delta_table SET TBLPROPERTIES ('delta.enableDeleteVectors' = false);

DELETE FROM bronze.my_delta_table WHERE id = 3;

Time Travel in Delta Lake

Time travel allows querying previous versions of data.
Real-World Scenario

Imagine you are a stock market analyst tracking daily stock prices. If you want to see prices from last
Monday, you can use time travel to query past data versions.

Code Example
SELECT * FROM bronze.my_delta_table VERSION AS OF 4;

Vacuuming Old Data

Removes old Parquet files to save storage.

Real-World Scenario

A university keeps student records but wants to delete graduated students' data after 5 years to free
up storage.

Code Example
VACUUM bronze.my_delta_table;

Schema Modifications
Delta Lake allows adding or dropping columns.

Real-World Scenario

A retail store introduces a loyalty points system. Schema modification helps add a new column without
breaking existing customer records.

Code Example
ALTER TABLE bronze.my_delta_table ADD COLUMN flag INT;
Optimization Techniques
Optimizing a Delta table improves query performance and storage efficiency.

Real-World Scenario

A video streaming platform (Netflix, YouTube) needs optimized queries to recommend personalized
content instantly.

Code Example
OPTIMIZE bronze.my_delta_table ZORDER BY (id);

Structured Streaming in Delta Lake

Delta Lake supports incremental data processing.

Real-World Scenario

A real-time fraud detection system in banks monitors transactions continuously. Structured streaming
ensures instant fraud alerts.

Code Example
df = spark.readStream.format('delta').load('/filestore/OpenTableFormat/Sink
data/bronze_my_delta_table')

display(df)