Accelerating Data Ingestion with Databricks Autoloader
2 likes2,186 views
Tracking which incoming files have been processed has always required thought and design when implementing an ETL framework. The Autoloader feature of Databricks looks to simplify this, taking away the pain of file watching and queue management. However, there can also be a lot of nuance and complexity in setting up Autoloader and managing the process of ingesting data using it. After implementing an automated data loading process in a major US CPMG, Simon has some lessons to share from the experience. This session will run through the initial setup and configuration of Autoloader in a Microsoft Azure environment, looking at the components used and what is created behind the scenes. We’ll then look at some of the limitations of the feature, before walking through the process of overcoming these limitations. We will build out a practical example that tackles evolving schemas, applying transformations to your stream, extracting telemetry from the process and finally, how to merge the incoming data into a Delta table. After this session you will be better equipped to use Autoloader in a data ingestion platform, simplifying your production workloads and accelerating the time to realise value in your data!
![Schema Metastore
_schemas
{“ID”:1,
“ProductName”:“Belt”}
{
"type": "struct",
"fields": [
{
"name": "ID",
"type": “string",
"nullable": true,
"metadata": {}
},
{
"name": "ProductName",
"type": “string",
"nullable": true,
"metadata": {}
}
]
}
0
On First Read](https://image.slidesharecdn.com/1060simonwhiteley-210614173625/85/Accelerating-Data-Ingestion-with-Databricks-Autoloader-30-320.jpg)
![Schema Metastore – DataType Inference
_schemas
{“ID”:1,
“ProductName”:“Belt”}
{
"type": "struct",
"fields": [
{
"name": "ID",
"type": “int",
"nullable": true,
"metadata": {}
},
{
"name": "ProductName",
"type": “string",
"nullable": true,
"metadata": {}
}
]
}
0
On First Read
.option(“cloudFiles.inferColumnTypes”,”True”)](https://image.slidesharecdn.com/1060simonwhiteley-210614173625/85/Accelerating-Data-Ingestion-with-Databricks-Autoloader-31-320.jpg)
![Schema Metastore – Schema Hints
_schemas
{“ID”:1,
“ProductName”:“Belt”}
{
"type": "struct",
"fields": [
{
"name": "ID",
"type": “long",
"nullable": true,
"metadata": {}
},
{
"name": "ProductName",
"type": “string",
"nullable": true,
"metadata": {}
}
]
}
0
On First Read
.option(“cloudFiles.schemaHints”,”ID long”)](https://image.slidesharecdn.com/1060simonwhiteley-210614173625/85/Accelerating-Data-Ingestion-with-Databricks-Autoloader-32-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Accelerating Data Ingestion with Databricks Autoloader (20)
![[DanNotes] XPages - Beyound the Basics](https://cdn.slidesharecdn.com/ss_thumbnails/dannotes052012-beyondthebasics-120430052622-phpapp01-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Databricks (20)
Ad
Recently uploaded (20)
![PRE-NATAL GRnnnmnnnnmmOWTH seminar[1].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/pre-natalgrowthseminar1-250427093235-de04befc-thumbnail.jpg?width=560&fit=bounds)
Accelerating Data Ingestion with Databricks Autoloader
- 1. Accelerating Data Ingestion with Databricks Autoloader Simon Whiteley Director of Engineering, Advancing Analytics
- 2. Agenda ▪ Why Incremental is Hard ▪ Autoloader Components ▪ Implementation ▪ Evolution ▪ Lessons
- 3. Why Incremental is Hard
- 5. Incremental Ingestion ▪ Only Read New Files ▪ Don’t Miss Files ▪ Trigger Immediately ▪ Repeatable Pattern ▪ Fast over large directories ?
- 6. Existing Patterns – 1) ETL Metadata etl batch read {“lastRead”:”2021/05/26”} Contents: • /2021/05/24/file 1 • /2021/05/25/file 2 • /2021/05/26/file 3 • /2021/05/27/file 4 .load(f“/{loadDate}/”
- 7. Existing Patterns – 2) Spark File Streaming file stream read Contents: • File 1 • File 2 • File 3 • File 4 Checkpoint: • File 1 • File 2 • File 3
- 8. Existing Patterns – 3) DIY triggered batch read Blob File Trigger Logic App Azure Function Databricks Job API
- 9. Incremental Ingestion Approaches Approach Good At Bad At Metadata ETL Repeatable Not immediate, requires polling File Streaming Repeatable Immediate Slows down over large directories DIY Architecture Immediate Triggering Not Repeatable
- 11. Prakash Chockalingam Databricks Engineering Blog Auto Loader is an optimized cloud file source for Apache Spark that loads data continuously and efficiently from cloud storage as new data arrives.
- 12. What is Autoloader? Essentially, Autoloader combines our three approaches of: • Storing Metadata about what has been read • Using Structured Streaming for immediate processing • Utilising Cloud-Native Components to optimise identifying arriving files There are two parts to the Autoloader job: • CloudFiles DataReader • CloudNotification Services (optional)
- 13. Cloudfiles Reader Blob Storage Blob Storage Queue {“fileAdded”:”/landing/file 4.json” • File 1.json • File 2.json • File 3.json • File 4.json Dataframe Check Files in Queue Read specific files from source
- 14. CloudFiles DataReader df = ( spark .readStream .format(“cloudfiles”) .option(“cloudfiles.format”,”json”) .option(“cloudfiles.useNotifications”,”true”) .schema(mySchema) .load(“/mnt/landing/”) ) Tells Spark to use Autoloader Tells Autoloader to expect JSON files Should Autoloader use the Notification Queue
- 15. Cloud Notification Services - Azure Blob Storage Event Grid Topic Event Grid Subscription Blob Storage Queue Event Grid Subscription Blob Storage Queue Event Grid Subscription Blob Storage Queue
- 16. Cloud Notification Services - Azure Blob Storage New File Arrives, Triggers Event Topic Subscription checks message filters, inserts into queue {fileAdded:“/file 4/”}
- 17. NotificationServices Config cloudFiles .useNotifications – Directory Listing VS Notification Queue .queueName – Use an Existing Queue .connectionString – Queue Storage Connection .subscriptionId .resourceGroup .tenantId .clientId .clientSecret Service Principal for Queue Creation
- 18. Implementing Autoloader ▪ Setup Steps ▪ Reading New Files ▪ A Basic ETL Setup
- 21. Low Frequency Streams Autoloader One File Per Day 24/7 Cluster
- 22. Low Frequency Streams Autoloader One File Per Day 1/7 Cluster df .writeStream .trigger(once=True) .save(path) Autoloader can be combined with trigger.Once – each run finds only files not processed since last run
- 24. Delta Merge Autoloader df .writeStream .foreachBatch(runThis) .save(path) def runThis(df, batchId): (df .write .save(path) )
- 25. Delta Implementation ▪ Batch ETL Pattern ▪ Merge Statements ▪ Logging State
- 26. Evolving Schemas New Features since Databricks Runtime 8.2
- 27. What is Schema Evolution? {“ID”:1,“ProductName”:“Belt”} {“ID”:2,“ProductName”:“T-Shirt”,”Size”:”XL”} {“ID”:3,“ProductName”:“Shirt”,“Size”:“14”, “Care”:{ “DryClean”: “Yes”, “MachineWash”:“Don’t you dare” } }
- 28. How do we handle Evolution? 1. Fail the Stream 2. Manually Intervene 3. Automatically Evolve In order to manage schema evolution, we need to know: • What the schema is expected to be • What the schema is now • How we want to handle any changes in schema
- 29. Schema Inference In Databricks 8.2 Onwards – simply don’t provide a Schema to enable Schema Inference. This infers the schema once when the stream is started and stores it as metadata. cloudfiles .schemaLocation – where to store the schema .inferColumnTypes – sample data to infer types .schemaHints – manually specify data types for certain columns
- 30. Schema Metastore _schemas {“ID”:1, “ProductName”:“Belt”} { "type": "struct", "fields": [ { "name": "ID", "type": “string", "nullable": true, "metadata": {} }, { "name": "ProductName", "type": “string", "nullable": true, "metadata": {} } ] } 0 On First Read
- 31. Schema Metastore – DataType Inference _schemas {“ID”:1, “ProductName”:“Belt”} { "type": "struct", "fields": [ { "name": "ID", "type": “int", "nullable": true, "metadata": {} }, { "name": "ProductName", "type": “string", "nullable": true, "metadata": {} } ] } 0 On First Read .option(“cloudFiles.inferColumnTypes”,”True”)
- 32. Schema Metastore – Schema Hints _schemas {“ID”:1, “ProductName”:“Belt”} { "type": "struct", "fields": [ { "name": "ID", "type": “long", "nullable": true, "metadata": {} }, { "name": "ProductName", "type": “string", "nullable": true, "metadata": {} } ] } 0 On First Read .option(“cloudFiles.schemaHints”,”ID long”)
- 33. Schema Evolution cloudFiles.schemaEvolutionMode • addNewColumns – Fail the job, update the schema metastore • failOnNewColumns – Fail the job, no updates made • rescue – Do not fail, pull all unexpected data into _rescued_data To allow for schema evolution, we can include a schema evolution mode option:
- 34. Evolution Reminder {“ID”:1,“ProductName”:“Belt”} {“ID”:2,“ProductName”:“T-Shirt”,”Size”:”XL”} {“ID”:3,“ProductName”:“Shirt”,“Size”:“14”, “Care”:{ “DryClean”: “Yes”, “MachineWash”:“Don’t you dare” } } 1 2 3
- 35. Schema Evolution - Rescue 1 2 3 ID Product Name _rescued_data 1 Belt ID Product Name _rescued_data 2 T-Shirt {“Size”:”XL”} ID Product Name _rescued_data 3 Shirt {“Size”:”14”,”Care”:{“DryC…
- 36. Schema Evolution – Add New Columns _schemas {“ID”:2, “ProductName”:“T-Shirt”, “Size”:”XL”} 0 On Arrival 2 1 { "type": "struct", "fields": [ { "name": "ID", "type": “string", }, { "name": "ProductName", "type": “string", } , { "name": “Size", "type": “string", }…
- 37. Schema Evolution ▪ Inference & The Schema Metastore ▪ Schema Hints ▪ Schema Evolution
- 38. Lessons from an Autoloader Life
- 39. Autoloader Lessons ▪ EventGrid Quotas & Settings ▪ Streaming Best Practices ▪ Batching Best Practices
- 41. EventGrid Quota Lessons • You can have 500 files from a single storage account using the system topic • Deleting checkpoint will reset the stream ID and create a new Subscription/Queue, leaving an orphan set • Use the CloudNotification Libraries to manage this more closely with custom topics
- 42. Streaming Optimisation • MaxBytesPerTrigger / MaxFilesPerTrigger Manages the size of the streaming microbatch • FetchParallelism Manages the workload on your queue
- 43. Batch Lessons – Look for Lost Messages Default 7 days!
- 44. Databricks Autoloader ▪ Reduces complexity of ingesting files ▪ Has some quirks in implementing ETL processes ▪ Growing number of schema evolution features
- 46. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.