![Start Airflow ( From install to UI )
š AIRFLOW_GPL_UNICODE=yes pip install “apache-airflow[celery, crypto, postgres, hive, rabbitmq, redis]”
š Airflow initdb
š Airflow upgraded
š Ls
š cd airflow
š grep dags_folder airflow.cfg
š mkdir –p /home/airflow/airflow/dags
š Ls
š Vim airflow.cfg ( Configuration File )
š Load.example ( false )
š Airflow resetdb
š Airflow scheduler
š Airflow webserver](https://image.slidesharecdn.com/airflowtutorialshandson1617980067-210409153541/85/Airflow-tutorials-hands_on-7-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Airflow tutorials hands_on (20)
Ad
More from pko89403 (11)
Ad
Recently uploaded (20)
Airflow tutorials hands_on
- 1. Airflow The Compele Hands-On Course 강석우 [email protected]
- 2. Why
- 3. AIRFLOW
- 4. What is airflow ? š 프로그램적으로 데이터 파이프라인을 author, schedule, monitor š 컴포넌트 : Web Server , Scheduler, Executor, Worker, Metadatabase š 키 컨셉 : DAG, Operator, Task, TaskInstance, Workflow
- 5. Airflow Architecture š Airflow Webserver : Serves the UI Dashboard over http š Airflow Scheduler : A daemon š Airflow Worker : working wrapper š Metadata Database : Stores information regarding state of tasks š Executor : Message queuing process that is bound to the scheduler and determines the worker processes that executes scheduled tasks Airflow Webserver Airflow Scheduler Worker Worker Worker Meta DB Logs Dags
- 6. How airflow works ? š 1. The scheduler reads the DAG folder š 2. Your Dag is parsed by a process to create a DagRun based on the scheduling parameters of your DAG. š 3. A TaskInstance is instantiated for each Task that needs to be executed and flagged to “Scheduled” in the metadata database š 4. The Scheduler gets all TaskInstances flagged “Scheduled” from the metadata database, changes the state to “Queued” and sends them to the executors to be executed. š 5. Executors pull out Tasks from the queue ( depending on your execution setup ), change the state from “Queued” to “Running” and Workers start executing the TaskInstances. š 6. When a Task is finished, the Executor changes the state of that task to its final state( success, failed, etc ) in the database and the DAGRun is updated by the scheduler with the state “Success” or “Failed”. Of course, the web server periodically fetch data from metadatabae to update UI.
- 7. Start Airflow ( From install to UI ) š AIRFLOW_GPL_UNICODE=yes pip install “apache-airflow[celery, crypto, postgres, hive, rabbitmq, redis]” š Airflow initdb š Airflow upgraded š Ls š cd airflow š grep dags_folder airflow.cfg š mkdir –p /home/airflow/airflow/dags š Ls š Vim airflow.cfg ( Configuration File ) š Load.example ( false ) š Airflow resetdb š Airflow scheduler š Airflow webserver
- 8. QuickTour of Airflow š airflow list_dags š airflow list_tasks {dag name} –tree š airflow test {dag name} python_task {execution date} š airflow –h
- 9. What is DAG ? š Finte directed graph with no directed cycles. No cycle š Dag represents a collection of tasks to run, organized in a way that represent their dependencies and relations š Each node is a Task š Each edge is Dependency š 어떻게 워크플로우를 실행시킬건가?
- 10. DAG’s important properties š Defined in Python files placed into Airflow’s DAG_FOLDER ( usually ~/airflow/dags) š Dag_id š Description š Start_date š Schedule_interval š Dependent_on_past : run the next DAGRun if the Previous one completed successfully š Default_args : constructor keyword parameter when initializing opeators
- 11. What is Operator? š Determines what actually gets done. š Operators are usually (but now always) atomic, meaning they can stand on their own and don’t need to share resources with any other operators. š Definition of single task š Should be idempotent ( 항상 같은 결과를 출력 ) š Task is created by instantiating an Operator class š An operator defines the nature of this task and how should it be executed š Operator is instantiated, this task becomes a node in your DAG.
- 12. Many Operators š Bash Operator š Python Operator š EmailOperator ( sends an email ) š SqlOperator ( Executes a SQL command š All Operators inherit from BaseOperator š 3 types of operators š Action operators that perform action (BashOperator, PythonOperator, EmailOperator … ) š Transfer operators that move data from one system to another ( sqlOperator, sftpOperator) š Sensor operators waiting for data to arrive at defined location.
- 13. Operator ++ š Transfer Operators š Move data from one system to another š Pulled out from the source, staged on the machine where the executor is running, and then transferred to the target system. š Don’t use if you are dealing with a large amount of data š Sensor Operators š Inherit of BaseSensorOperator š They are useful for monitoring external processes like waiting for files to be uploaded in HDFS or a partition appearing in Hive š Basically long running task š Sensor operator has a poke method called repeatedly until it returns True ( method used for monitoring the external process)
- 14. Make Dependencies in python š set_upstream() š set_downstream() š << ( = set_upstream ) š >> ( = set_downstream ) A B C D š B depends of A š C depends of A š D depends of B and C ( Example ) A.set_downstream(B) A >> B A >> { B, C } >> D
- 15. How the Scheduler Works š DagRun š A Dag consists of Tasks and need those tasks to run š When the Scheduler parses a Dag, it automatically creates a DagRun which is an instantiation of a DAG in time according to start_date and schedule š Backfill and Catchup š Scheduler Interval š None š @once š @hourly š @daily š @weekly š @monthly š @yearly š Cron time string format can be used : ( * * * * * - Minute(0-59) Hour(0-23) Day of the month(1-31) Month(1-12) Day of the week(0-7)
- 16. Concurrency vs Parallelism š Concurrent – If it can support two or more actions in progress at the same time š Parallel – If it can support two or more actions executing simultaneously š In concurrent systems, multiple actions can be in progress (may not be executed) at the same time š In parallel systems, multiple actions are simultaneously executed
- 17. Database and Executor š Sequential Executor ( Default executor, SQLlite ) š Default executor you get when you run Apache Airflow š Only run one task at time (Sequential), useful for debugging š It is the only executor that can be used with SQLite since SQLlite donesn’t support multiple writers š Local Executor ( PostgreSQL ) š It can run multiple tasks at a time š Multiprocessing python library and queues to parallelize the execution of tasks š Run tasks by spawning processes in a controlled fashion in different modes on the same machine š Can tune the number of processes to spawn by using the parallelism parameter
- 18. Database and Executor š Celery Executor š Celery == Python Task-Queue System š Task-Queue System handle distribution of tasks on workers across threads or network nodes š Tasks need to be pushed into a broker( RabbitMQ ) š celery workers will pop them and schedule task executions š Recommend for production use of Airflow š Allows distributing the execution of task instances to multiple worker node(Computer) š ++ Dask, Mesos, Kubernetes … etc
- 19. Celery Executor, PostgreSQL and RabbitMQ Structure
- 20. Executor Architecture Meta DB Web Server Scheduler + Worker Local Executor ( Single Machine ) Meta DB Web Server Scheduler + Worker Worker Worker Celery Celery Executor
- 21. Advanced Concept š SubDAG š Minimising repetitive patterns š Main DAG mangages all the subDAGs as normal taks š SubDAGs must be scheduled the same as their parent DAG š Hooks š Interfaces to interact with your external sources such as (PostgreSQL, Spark, SFTP … )
- 22. XCOM š Tasks communicate ( cross-communication , allows multiple tasks to exchange messages ) š Principally defined by a key, value and a timestamp š XCOMs data can be “pushed” or “pulled” š X_com_push() š If a task returns a value, a XCOM containing that value is automatically pushed š X_com_pull() š Task gets the message based on parameters such as “key”, “task_ids” and “dag_id” š Keys that are automatically given to XCOMs when they are pushed by being returned from
- 23. Branching š Allowing DAG to choose between different paths according to the result of a specific task š Use BranchPythonOperator š When using branch, do not use property depends on past+
- 24. Service Level Agreement ( SLAs ) š SLA is a contract between a service provider and the end user that defines the level of service expected from the service provider š Define what the end user will received ( Must be received ) š Time, relative to the execution_date of tast not the start time(more than 30 min from exec ) š Different from ‘execution_timeout’ parameter << It makes task stopped and marks failed