Building Robust ETL

Pipelines with Apache Spark

Xiao Li
Spark Summit | SF | Jun 2017
About Databricks
TEAM
Started Spark project (now Apache Spark) at UC Berkeley in 2009

MISSION
Making Big Data Simple

PRODUCT
Unified Analytics Platform

222
About Me
• Apache Spark Committer
• Software Engineer at Databricks
• Ph.D. in University of Florida
• Previously, IBM Master Inventor, QRep, GDPS A/A and STC
• Spark SQL, Database Replication, Information Integration
• Github: gatorsmile

3
Overview
1. What’s an ETL Pipeline?
2. Using Spark SQL for ETL
- Extract: Dealing with Dirty Data (Bad Records or Files)
- Extract: Multi-line JSON/CSV Support
- Transformation: High-order functions in SQL
- Load: Unified write paths and interfaces
3. New Features in Spark 2.3
- Performance (Data Source API v2, Python UDF)
4
What is a Data Pipeline?
1. Sequence of transformations on data
2. Source data is typically semi-structured/unstructured
(JSON, CSV etc.) and structured (JDBC, Parquet, ORC, the
other Hive-serde tables)
3. Output data is integrated, structured and curated.
– Ready for further data processing, analysis and reporting

5
Example of a Data Pipeline
Kafka, Log
Ad-hoc Queries

Cloud
Kafka, Log Warehouse Database

Aggregate Reporting
Applications
ML
Model
6
ETL is the First Step in a Data Pipeline
1. ETL stands for EXTRACT, TRANSFORM and LOAD
2. Goal is to clean or curate the data
- Retrieve data from sources (EXTRACT)
- Transform data into a consumable format (TRANSFORM)
- Transmit data to downstream consumers (LOAD)

7
An ETL Query in Apache Spark
spark.read.json("/source/path") EXTRACT
.filter(...)
.agg(...) TRANSFORM

.write.mode("append")
.parquet("/output/path") LOAD

8
An ETL Query in Apache Spark
val csvTable = spark.read.csv("/source/path")
val jdbcTable = spark.read.format("jdbc") Extract
.option("url", "jdbc:postgresql:...") EXTRACT
.option("dbtable", "TEST.PEOPLE")
.load()
csvTable
.join(jdbcTable, Seq("name"), "outer")
TRANSFORM
.filter("id <= 2999")
.write
.mode("overwrite")
.format("parquet")
.saveAsTable("outputTableName") LOAD
9
What’s so hard about ETL
Queries?

10
Why is ETL Hard?
1. Various sources/formats 1. Too complex
2. Schema mismatch 2. Error-prone
3. Different representation 3. Too slow
4. Corrupted files and data 4. Too expensive
5. Scalability
6. Schema evolution
7. Continuous ETL

11
This is why ETL is important
Consumers of this data don’t want to deal with this
messiness and complexity

12
Using Spark SQL for ETL

13
Spark SQL's flexible APIs,
support for a wide
variety of datasources, Structured

build-in support for Streaming

structured streaming,
state of art catalyst
optimizer and tungsten
execution engine make it
a great framework for
building end-to-end ETL
pipelines.
14
Data Source Supports
1. Built-in connectors in Spark:
– JSON, CSV, Text, Hive, Parquet, ORC, JDBC
2. Third-party data source connectors:
– https://spark-packages.org
3. Define your own data source connectors by
Data Source APIs
– Ref link: https://youtu.be/uxuLRiNoDio
15
Schema Inference – semi-structured files
{"a":1, "b":2, "c":3}
{"e":2, "c":3, "b":5}
{"a":5, "d":7}

spark.read
.json("/source/path”)
.printSchema()

16
Schema Inference – semi-structured files
{"a":1, "b":2, "c":3.1}
{"e":2, "c":3, "b":5}
{"a":"5", "d":7}

spark.read
.json("/source/path”)
.printSchema()

17
User-specified Schema
{"a":1, "b":2, "c":3} val schema = new StructType()
{"e":2, "c":3, "b":5} .add("a", "int")
{"a":5, "d":7}
.add("b", "int")

spark.read
.json("/source/path")
.schema(schema)
.show()

18
User-specified DDL-format Schema
{"a":1, "b":2, "c":3} spark.read
{"e":2, "c":3, "b":5} .json("/source/path")
{"a":5, "d":7}
.schema("a INT, b INT")
.show()

Availability: Apache Spark 2.2

19
Dealing with Bad Data: Skip Corrupt Files
java.io.IOException. For example, java.io.EOFException: Unexpected end of input
stream at org.apache.hadoop.io.compress.DecompressorStream.decompress

java.lang.RuntimeException: file:/temp/path/c000.json is not a Parquet file (too

small)
[SPARK-17850] If true, the Spark jobs will
continue to run even when it encounters
Corrupt
corrupt files. The contents that have
Files
been read will still be returned.

spark.sql.files.ignoreCorruptFiles = true
20
Dealing with Bad Data: Skip Corrupt Records
[SPARK-12833][SPARK-
13764] TextFile formats
(JSON and CSV) support
3 different ParseModes Missing or
Corrupt
while reading data:
Records
1. PERMISSIVE
2. DROPMALFORMED
3. FAILFAST
21
Json: Dealing with Corrupt Records
{"a":1, "b":2, "c":3}
{"a":{, b:3}
{"a":5, "b":6, "c":7}

spark.read
.option("mode", "PERMISSIVE")
.option("columnNameOfCorruptRecord", "_corrupt_record")
.json(corruptRecords)
.show() The default can be configured via
spark.sql.columnNameOfCorruptRecord
22
Json: Dealing with Corrupt Records
{"a":1, "b":2, "c":3}
{"a":{, b:3}
{"a":5, "b":6, "c":7}

spark.read
.option("mode", "DROPMALFORMED")
.json(corruptRecords)
.show()

23
Json: Dealing with Corrupt Records
{"a":1, "b":2, "c":3}
{"a":{, b:3}
{"a":5, "b":6, "c":7}

spark.read org.apache.spark.sql.catalyst.json
.SparkSQLJsonProcessingException:
.option("mode", "FAILFAST")
Malformed line in FAILFAST mode:
.json(corruptRecords) {"a":{, b:3}
.show()

24
CSV: Dealing with Corrupt Records
year,make,model,comment,blank
"2012","Tesla","S","No comment",
1997,Ford,E350,"Go get one now they",
2015,Chevy,Volt

spark.read java.lang.RuntimeException:
.option("mode", "FAILFAST") Malformed line in FAILFAST mode:
.csv(corruptRecords) 2015,Chevy,Volt
.show()

25
CSV: Dealing with Corrupt Records
year,make,model,comment,blank
"2012","Tesla","S","No comment",
1997,Ford,E350,"Go get one now they",
2015,Chevy,Volt

spark.read.
.option("mode", "PERMISSIVE")
.csv(corruptRecords)
.show()

26
CSV: Dealing with Corrupt Records
year,make,model,comment,blank
"2012","Tesla","S","No comment",
1997,Ford,E350,"Go get one now they",
2015,Chevy,Volt

spark.read
.option("header", true)
.option("mode", "PERMISSIVE")
.csv(corruptRecords)
.show()
27
CSV: Dealing with Corrupt Records
val schema = "col1 INT, col2 STRING, col3 STRING, col4 STRING, " +
"col5 STRING, __corrupted_column_name STRING"
spark.read
.option("header", true)
.option("mode", "PERMISSIVE")
.csv(corruptRecords)
.show()

28
CSV: Dealing with Corrupt Records
year,make,model,comment,blank
"2012","Tesla","S","No comment",
1997,Ford,E350,"Go get one now they",
2015,Chevy,Volt

spark.read
.option("mode", ”DROPMALFORMED")
.csv(corruptRecords)
.show()

29
Functionality: Better Corruption Handling
badRecordsPath: a user-specified path to store exception files for
recording the information about bad records/files.
- A unified interface for both corrupt records and files
- Enabling multi-phase data cleaning
- DROPMALFORMED + Exception files
- No need an extra column for corrupt records
- Recording the exception data, reasons and time.

Availability: Databricks Runtime 3.0

30
Functionality: Better JSON and CSV Support

[SPARK-18352] [SPARK-19610] Multi-line JSON and CSV Support

- Spark SQL currently reads JSON/CSV one line at a time
- Before 2.2, it requires custom ETL

spark.read spark.read
.option(”multiLine",true) .option(”multiLine",true)
.json(path) .json(path)

Availability: Apache Spark 2.2

31
Transformation: Higher-order Function in SQL
Transformation on complex objects like arrays, maps and
structures inside of columns.
tbl_nested
|-- key: long (nullable = false)
|-- values: array (nullable = false)
| |-- element: long (containsNull = false)

UDF ? Expensive data serialization

32
Transformation: Higher order function in SQL
Transformation on complex objects like arrays, maps and
structures inside of columns.
1) Check for element existence tbl_nested
SELECT EXISTS(values, e -> e > 30) AS v |-- key: long (nullable = false)
FROM tbl_nested; |-- values: array (nullable = false)
| |-- element: long (containsNull = false)
2) Transform an array
SELECT TRANSFORM(values, e -> e * e) AS v
FROM tbl_nested;
33
Transformation: Higher order function in SQL
3) Filter an array tbl_nested
|-- key: long (nullable = false)
SELECT FILTER(values, e -> e > 30) AS v
FROM tbl_nested; |-- values: array (nullable = false)
| |-- element: long (containsNull = false)
4) Aggregate an array
SELECT REDUCE(values, 0, (value, acc) -> value + acc) AS sum
FROM tbl_nested;

Ref Databricks Blog: http://dbricks.co/2rUKQ1A

More cool features available in DB Runtime 3.0: http://dbricks.co/2rhPM4c

Availability: Databricks Runtime 3.0

34
New Format in DataframeWriter API
Users can create Hive-serde tables using
DataframeWriter APIs
df.write.format("hive") df.write.format("parquet")
.option("fileFormat", "avro") .saveAsTable("tab")
.saveAsTable("tab")

CREATE Hive-serde tables CREATE data source tables

Availability: Apache Spark 2.2

35
Unified CREATE TABLE [AS SELECT]
CREATE TABLE t1(a INT, b INT)
STORED AS ORC

CREATE TABLE t1(a INT, b INT) CREATE TABLE t1(a INT, b INT)
USING hive USING ORC
OPTIONS(fileFormat 'ORC')

CREATE Hive-serde tables CREATE data source tables

Availability: Apache Spark 2.2

36
Unified CREATE TABLE [AS SELECT]
Apache Spark preferred syntax
CREATE [TEMPORARY] TABLE [IF NOT EXISTS]
[db_name.]table_name
USING table_provider
[OPTIONS table_property_list]
[PARTITIONED BY (col_name, col_name, ...)]
[CLUSTERED BY (col_name, col_name, ...)
[SORTED BY (col_name [ASC|DESC], ...)]
INTO num_buckets BUCKETS]
[LOCATION path]
[COMMENT table_comment]
[AS select_statement];
Availability: Apache Spark 2.2
37
Apache Spark 2.3+
Massive focus on building ETL-friendly pipelines

38
[SPARK-15689] Data Source API v2
1. [SPARK-20960] An efficient column batch interface for data
exchanges between Spark and external systems.
o Cost for conversion to and from RDD[Row]
o Cost for serialization/deserialization
o Publish the columnar binary formats
2. Filter pushdown and column pruning
3. Additional pushdown: limit, sampling and so on.

Target: Apache Spark 2.3

39
Performance: Python UDFs
1. Python is the most popular language for ETL
2. Python UDFs are often used to express elaborate data
conversions/transformations
3. Any improvements to python UDF processing will ultimately
improve ETL.
4. Improve data exchange between Python and JVM
5. Block-level UDFs
o Block-level arguments and return types
Target: Apache Spark 2.3
40
Recap
1. What’s an ETL Pipeline?
2. Using Spark SQL for ETL
- Extract: Dealing with Dirty Data (Bad Records or Files)
- Extract: Multi-line JSON/CSV Support
- Transformation: High-order functions in SQL
- Load: Unified write paths and interfaces
3. New Features in Spark 2.3
- Performance (Data Source API v2, Python UDF)
41
Try Apache Spark in Databricks!
UNIFIED ANALYTICS PLATFORM
• Collaborative cloud environment
• Free version (community edition)
Try for free today.
DATABRICKS RUNTIME 3.0 databricks.com
• Apache Spark - optimized for the cloud
• Caching and optimization layer - DBIO
• Enterprise security - DBES

42
42
42
Questions?
Xiao Li ([email protected])