Orchestrating workflows Apache Airflow on GCP & AWS
- 1. Orchestrating workflows Apache Airflow on GCP & AWS Derrick Qin - Cloud Data Architect @ DoiT International Multi-Cloud Engineering Meetup Australia Tuesday 23 March 2021
- 2. About the speaker Derrick Qin Cloud Data Architect, DoiT International Data engineering on GCP and AWS
- 3. Agenda ● Typical workflows ● Introducing Airflow ● Why Airflow is popular ● Airflow Concepts ● Demo ● How to run Airflow ● Google Cloud Composer ● Amazon Managed Workflows for Apache Airflow (MWAA) ● Airflow Best Practices (my view) ● Q&A
- 4. Typical workflow ● Daily - load batch files from different databases to a reporting database ● Daily/Weekly/Monthly - generate and deliver reports to stakeholders ● Daily - re-train machine learning models with fresh data ● Hourly - back up database ● Hourly - generate and send recommended products to customers based on customers activities - think spamsemails you get from eBay ● On-demand - send registration emails to newly registered customers Every 5 minutes - run your price/discount watchdog - automatic price check on retail websites or OZBargain
- 5. Introducing Airflow ● Airflow is an orchestration platform to programatically schedule and monitor workflows ● Started in late 2014 @Airbnb, open sourced in mid-2015 ● Governed under Apache Foundation ● Over 20K stars on Github ● Used by lots of well-known organizations
- 6. Why Airflow is popular ● Workflows are defined as Python code ○ More flexible - because of Python programming language ○ Workflow as code is more testable ○ Reuse ● Battery included platform ○ popular database: mysql, postgres, mongodb, oracle, SQL server, Snowflake, BigQuery ○ services: Databricks, Datadog, ElasticSearch, Jenkins, Salesforce, SendGrid, Slack, Zendesk ○ public cloud platform: AWS, GCP, Azure ● Informational and feature-rich UI to visualize workflows' status, monitor progress, troubleshoot issues, trigger and re-trigger workflows and tasks in them
- 10. Airflow concepts
- 11. Airflow concepts ● DAG: directed acyclic graph is a directed graph with no directed cycles - workflows ● Operator: they define what should be executed. Example: Bash command, read a file, call an API, load data to a table, etc ● Task: instance of an operator, it is a node in a DAG/Workflow ● Sensor: a special operator which runs repeatedly until the predefined condition is fulfilled. Example: a file sensor can wait until the file lands, then continue the workflow ● Hook: an interface to external platform or system. Example: S3Hook wraps AWS S3 API to provide easy access to S3 bucket ● DAG run: when a DAG is triggered, it is called a DAG run. It represents the instance of the workflow
- 12. Airflow architecture ● Web UI/webserver ● Scheduler ● Worker ● Metadata database Executors ● SequentialExecutor ● LocalExecutor ● CeleryExecutor
- 13. How to run Airflow locally ● Local setup with Python Virtualenv ○ http://airflow.apache.org/docs/apache-airflow/stable/start/local.html ● Local setup with Docker ○ http://airflow.apache.org/docs/apache-airflow/stable/start/docker.html
- 14. Demo - local docker setup
- 15. Any hosted Airflow solutions? ● GCP Cloud Composer ● Amazon Managed Workflows for Apache Airflow (MWAA) ● astronomer.io
- 16. GCP Cloud Composer ● Deployment via Console, gcloud, API, Terraform ● Run on GKE with *auto-scaling support ○ https://medium.com/tra veloka-engineering/enabl ing-autoscaling-in-google -cloud-composer-ac84d3 ddd60 ● Running the scheduler and workers on GKE enables you to use the KubernetesPodOperator to run any container workload.
- 17. GCP Cloud Composer ● DAGs and plugins are deployed and managed on Google Cloud Storage(GCS) buckets ● DAGs can be triggered via ○ Composer API ○ Composer CLI - wrapped on top of Airflow CLI ● Plugin management can be tricky ○ Managed by CloudBuild - troubleshooting from CloudBuild Logs
- 18. GCP Cloud Composer ● Airflow containers can be accessed via Kubectl ○ kubectl -n composer-1-14-4-airflow-example-namespace exec -it airflow-worker-1a2b3c-x0yz -c airflow-worker -- /bin/bash ● Airflow Data can only be accessed from worker - ssh to worker, then use SQLAlchemy ● Security ○ Airflow permission binds with service account ○ Google Cloud Secret Manager - used as Airflow connection/variable/jinja template ○ Overwrite default service account using a new connection
- 19. Demo - Cloud Composer
- 20. AWS MWAA ● Deployment via Console, AWS CLI, SDK, Cloudformation ● Run on AWS Fargate and AWS SQS ○ Workers can be auto-scaled based on load ● Security ○ Data encrypted using AWS KMS ○ Use AWS Secret Manager to manage secret, connection/variables
- 21. AWS MWAA ● DAGs and plugins are deployed and managed on AWS S3 buckets ● DAGs can be triggered via ○ AWS SDK/API ○ AWS CLI Airflow container access?
- 22. Demo - AWS MWAA
- 23. Airflow Best Practices ● Try to balance between DAG readability and code abstraction ● Limit local compute ● Use built-in libraries if possible ● Generate custom dashboard for non-technical stakeholders ● One DAG per data source ● Testing DAG and custom Plugins ○ Unit testing and end-to-end testing ■ Will be covered in next meetup talk
- 24. Is Airflow my only option? ● Crontab ● Jenkins ● GCP CloudBuild ● Argo ● AWS Step Functions ● ...