[DSC DACH 24] Cost efficient alternative to databricks lock-in - Georg Heiler

Editor's Notes

• #1: Mention talk modality interactive ask questions during the talk
• #2: Supply Chain, Text analytics & data architecture & pipelines, graphs, spatial time series
• #4: Physical goods Software 400TiB Commoncrawl, AIS, Satellite, OSINT, …),
• #5: Easy GPU/accelerator prototyping AWS is migrating to daft https://www.getdaft.io/ https://aws.amazon.com/de/blogs/opensource/amazons-exabyte-scale-migration-from-apache-spark-to-ray-on-amazon-ec2/ as we are in control of how the individual steps in the pipeline relate to each other we can relatively easily switch out the compute framework (like AWS did it)
• #9: PaaS solutions offer big benefits. Easy scalability Single centerpiece of the data engineering strategy not just an implementation detail Runaway expenses due to usage-based pricing High CI costs for spinning up resources Prioritizes simplicity over best practices All-notebook environments: Limited code reuse Limited testability Limited VCS integration Developer productivity hampered by VM spin-up times Single central platform dependence
• #10: Pass as implementation detail Containerization, CI/CD, testability flexibility
• #11: Img https://intellipaat.com/blog/tutorial/spark-tutorial/spark-architecture/
• #13: Single pane of glass for operative monitoring of opaque saas tools. Allows us to get standard software best practices like testing, modularity, DRY, ... and maintain these in the SaaS tools. It even allows us to abstract SaaS vendors (Databricks vs. EMR) and substitute one against the other one to save money for large-scale pipelines where we just need compute without wanting to pay for all the enterprisey extra features. 1) boostrap of the remote execution environments 2) centralized logging 3) single/simple start/stop of all pipeline in dagster 3) integration of upstream/downstream pipeline steps in one place PaaS solutions offer big benefits. Easy scalability Single centerpiece of the data engineering strategy not just an implementation detail Runaway expenses due to usage-based pricing High CI costs for spinning up resources Prioritizes simplicity over best practices All-notebook environments: Limited code reuse Limited testability Limited VCS integration Developer productivity hampered by VM spin-up times Single central platform dependence
• #20: The volume of trial runs required to achieve stability on EMR is high. It shows the complex setup and optimization demands of these platforms. Yet, once set up now, it proves hugely beneficial for us.
• #21: EMR was labor-intensive. This is shown by more failed and successful trials. They happened before the product was ready. In fact we required almost twice as many trial runs for EMR as for Databricks. The increase was mainly due to the complexity of setting up EMR systems. They had to handle large datasets well and safely. This setup needs lots of customization and tuning. Databricks provided these features out of the box. EMR demanded more frequent code changes. They were sometimes extensive. This reflected a steeper learning curve and higher complexity. But, it was in exchange for lower costs.
• #24: Teams need to learn the specifics of each platform. This includes API differences, setup quirks, and best practices. This learning curve can delay initial deployment and require additional training.
• #25: For small-ish to medium workloads EMR works fine out of the box => we can save money (cheaper compute) For very large workloads, we use EMR with fine-tuning. It lets us save money with cheaper compute. For special workloads, Photon is very profitable. We use Databricks to save money and time again.
• #26: https://aws.amazon.com/de/blogs/opensource/amazons-exabyte-scale-migration-from-apache-spark-to-ray-on-amazon-ec2/
• #27: Mention talk modality interactive ask questions during the talk