DevOps for Databricks
1 like1,500 views
The document discusses DevOps practices in the context of Databricks, outlining essential concepts such as CI/CD, Infrastructure as Code (IaC), and the use of build agents. It provides real-world examples and live demos for leveraging Python scripts and the Databricks REST API in Azure DevOps to automate tasks like cluster management and notebook uploads. The document emphasizes that DevOps is crucial for maintaining efficient code deployment and offers various tooling options to facilitate these processes.
DevOps for Databricks
- 1. DevOps for Databricks Anna-Maria Wykes Data Engineering Consultant
- 2. Agenda § What is DevOps § CI/CD (Continuous Integration/Continuous Deployment) § IAC (Infrastructure as Code) § Build Agents § Databricks Rest API § Real World Example § Other Tooling Examples
- 4. BI Developer Data Scientist Software Engineer Data Engineer “I want to get my dashboard published on the website” “I want to productionize my models and have them automatically update” “I want to update the website with the latest dashboard” “I want to push the latest ETL pipelines to production” DevOps
- 5. DevOps
- 6. DevOps Pipelines Development Test Production
- 7. DevOps Tools Continuous Integration/Continuous Deployment (CI/CD) Infrastructure as Code (IAC) ARM (Azure Resource Manager) Templates Azure Bicep
- 8. Continuous Integration & Continuous Deployment (CI/CD)
- 9. CI/CD • Continues Improvements • Feature releases • Fast bug fixes • Ability to quickly rollback • Testing • Unit/Integration/End to End testing • Linting (check code is formatted correctly)
- 10. Infrastructure as Code (IAC)
- 11. IAC: The Blueprint of your Solution
- 12. Build Agents
- 13. What is a Build Agent? • It is the compute under your DevOps Pipelines • “Out of the box” available Agents in Azure DevOps • Custom VM Agent • Custom Docker Container Agent Pipeline (yml) triggered Agent found/located Pipeline executed on Agent
- 14. Why a Custom Build Agent? • You can decide specifically what you want your code you run on (what Linux/Windows version/docker image) • Make sure all the tools you need are installed on your Agent • Keep state • Run within a VNet (Virtual Network)
- 16. Why the Databricks REST API? • Can use your existing knowledge of REST • Can incorporate into our language of choice (Python) • Cross Platform https://docs.databricks.com/dev-tools/api/latest/index.html
- 18. What are we going to do? • Use Python Scripts and Databricks Rest API to: • Create a Databricks Cluster • Check Cluster Status • Upload Notebooks to Databricks Workspace • Run some tests against our Python code • Build and upload a Python Wheel to Databricks • Install/uninstall/update Python Wheel in Databricks • Use Azure DevOps to run our scripts • YML Pipelines • Custom DevOps Agent
- 19. Create a Databricks Cluster Live demo using VSCode and Databricks in Azure
- 20. Check Cluster Status Live demo using VSCode and Databricks in Azure
- 21. Live demo using VSCode and Databricks in Azure Upload Notebooks to Databricks Workspace
- 22. Run some tests against our Python code Live demo using VSCode and Databricks in Azure
- 23. Live demo using VSCode and Databricks in Azure Build and upload a Python Wheel to Databricks
- 24. Live demo using VSCode and Databricks in Azure Install/uninstall/update Python Wheel in Databricks
- 25. Introduction to Azure DevOps • How to Create a Pipeline • How to Run a Pipeline • How to use a Custom Agent
- 26. How to Create and Run a DevOps Pipeline Live demo using Azure DevOps
- 27. Adding our Python Scripts to a Pipeline Live demo using Azure DevOps and Databricks in Azure
- 28. Using a Custom DevOps Agent Live demo using Azure DevOps and Databricks in Azure
- 29. Examples of other DevOps IAC tools
- 30. Azure ARM templates Azure Bicep Other IAC Tools
- 31. What is Terraform? Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services. Terraform codifies cloud APIs into declarative configuration files. Write Plan Apply
- 34. Build, deploy, and manage modern cloud applications and infrastructure using familiar languages, tools, and engineering practices. https://github.com/pulumi/pulumi-azure What is Pulumi? https://www.pulumi.com/docs/reference/pkg/azure/databricks/ Cloud Engineering for Everyone
- 35. Pulumi azure.databricks Module Based on the azurerm Terraform Provider. Creating a Workspace Resource using Pulumi
- 36. What is Bicep? Write Apply • Project Bicep – Next Generation ARM Templates • ARM Templates can get complex • Bicep is a cleaner more readable language, that gets compiled into ARM to deploy (a language around ARM) Write and Compile Bicep Language ARM Templates Azure Resource Manager Deployed Solution
- 37. Summary
- 38. Summary • DevOps is for Everyone • CI/CD keeps your code in check and the latest features/changes in production as soon as possible • IAC is the blueprint of your solution • Lots of tooling options • Databricks Rest API can be used in conjunction with Python and Azure DevOps to create effective fault tolerant pipelines
- 39. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.