Apex as yarn application
- 1. Apache Apex as YARN Application Chinmay Kolhatkar ([email protected]) Mar 22, 2016 Apache Apex Meetup
- 2. Agenda • Directed Acyclic Graph • Apex as a YARN Application • Application Components of Apex • Lifecycle of Apex as a YARN Application Apache Apex Meetup
- 3. Directed Acyclic Graph (DAG) • Defines compute stages of streaming application • Defines tuple flow across Operators via Stream Compute 1 Apache Apex Meetup Compute 3 Compute 2 Compute 4
- 4. DAG Components • Tuple ● Atomic data that flows over a stream • Operator ● Basic compute unit per tuple • Stream ● Connector abstraction between operators ● Tuples flow over this Operator 1 Operator 2 Apache Apex Meetup Stream tuple 3 tuple 1 tuple 2
- 5. DAG Types O1 O2 O3 O4 Physical DAG Apache Apex Meetup O5 Logical DAG • Logical Plan ● Logical representation of computation ● Defines operators, streams and dataflow • Physical Plan ● Deployable plan on cluster ● Contains partition information of operators ● Has ready-to-deploy serialized operator instances O1 P1 O1 P2 O1 P3 O2 P1 O2 P2 O2 P3 U O3 O4 O5
- 6. Apex as YARN application Node ResourceManager (AsM + Scheduler) NM Node NM Node NM YarnClient AppMaster YarnContainer YarnContainer YarnContainer StrAM (AppMaster) YarnContainer StrAMChild O1 O2 YarnContainer StrAMChild O3 DTCLI StrAMClient YarnClient Apache Apex Meetup ClientRM Protocol AMRM Protocol ContainerManager Protocol ContainerManager Protocol ClientRM Protocol AMRM Protocol ContainerManager Protocol
- 7. Application Components of Apex - StrAMClient • Part of dtcli client interface • Invoked by “launch” command of dtcli • Tasks: ● Copy required the application package files into HDFS ● Validate Logical Plan ● Serialize Logical plan to HDFS ● Launch Application Master i.e. StrAM Apache Apex Meetup
- 8. Application Components of Apex - StrAM • Streaming Application Master • Started by StrAMClient on a YarnContainer • Tasks: ● Convert logical plan to physical plan ● Serialize operators to HDFS ● Request for resources to ResourceManager ● Start StrAMChild in YarnContainer(s) ● Monitor StrAMChild using ContainerManager protocol ● Generate Application statistics ● Host results on WebService (dtManage) ● Fault Tolerance ● Checkpointing/Committing Application States ● Support Security ● Shutdown Application Apache Apex Meetup
- 9. Application Components of Apex - StrAMChild • Deployed on YarnContainer • Started by NodeManager as instructed by StrAM • Instance of StreamingContainer • Contains Operators (compute-related) • Contains BufferServer (stream-related) • Tasks: ● Regularly send heartbeat to StrAM ● Execute commands from StrAM ● Shutdown or Kill self if instructed ● Manage lifecycle of an Operator ● Network communication using BufferServer Apache Apex Meetup
- 10. Lifecycle of Apex/YARN Application - Start Node ResourceManager (AsM + Scheduler) NM Node NM Node NM DTCLI/ StrAMClient (YarnClient) 1) Access cluster information HDFS 3) Submit Application to RM StrAM (AppMaster) 4) StrAM Registers with RM 5) StrAM sends heartbeats regularly 6) StrAM request containers with specifications 7) StrAMChild reads serialized operator from HDFS 8) StrAMChild starts operator lifecycle Apache Apex Meetup 2) Copies files from HDFS ClientRMProtocol AMRMProtocol YarnContainer StrAMChild O2 O1 YarnContainer StrAMChild O3 YarnContainer StrAMChild O4ContainerManager Protocol ContainerManager Protocol
- 11. Lifecycle of Apex/YARN Application - Running Node ResourceManager (AsM + Scheduler) NM Node NM Node NM DTCLI/ StrAMClient (YarnClient) HDFS StrAM (AppMaster) Apache Apex Meetup ClientRMProtocol AMRMProtocol YarnContainer StrAMChild O2 O1 YarnContainer StrAMChild O3 YarnContainer StrAMChild O4ContainerManager Protocol ContainerManager Protocol 1) StrAMChild sends heartbeats 2) StrAMChild sends operator data 3) StrAM send regular heartbeats to RM 4) Query status of application
- 12. Lifecycle of Apex/YARN Application - Shutdown Node ResourceManager (AsM + Scheduler) NM Node NM Node NM DTCLI/ StrAMClient (YarnClient) HDFS StrAM (AppMaster) Apache Apex Meetup ClientRMProtocol AMRMProtocol YarnContainer StrAMChild O2 O1 YarnContainer StrAMChild O3 YarnContainer StrAMChild O4ContainerManager Protocol ContainerManager Protocol 1) Connect on WebService REST API 3) Send shutdown signal to StrAMChild 4) StrAMChild finishes operator lifecycle 5) Check if all containers are freed 6) StrAM unregisters itself 7) StrAM exits 8) Check if application has shutdown 2) Send command to StrAM
- 13. Lifecycle of Apex/YARN Application - Kill Node ResourceManager (AsM + Scheduler) NM Node NM Node NM DTCLI/ StrAMClient (YarnClient) HDFS StrAM (AppMaster) Apache Apex Meetup ClientRMProtocol AMRMProtocol YarnContainer StrAMChild O2 O1 YarnContainer StrAMChild O3 YarnContainer StrAMChild O4ContainerManager Protocol ContainerManager Protocol 1) Send kill-app command to YARN 2) RM kills all containers
- 14. Summary – Apex platform • Enables YARN to be used for Streaming Applications • Takes care of YARN specific work • User can focus on business logic defined in Operators Apache Apex Meetup
- 16. Resources Apache Apex Meetup • Apache Apex website - http://apex.incubator.apache.org/ • Subscribe - http://apex.incubator.apache.org/community.html • Download - http://apex.incubator.apache.org/downloads.html • Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex • Facebook - https://www.facebook.com/ApacheApex/ • Meetup - http://www.meetup.com/topics/apache-apex • Startup Program – Free Enterprise License for startups, Universities, Non-Profits
- 17. Upcoming events... Apache Apex Meetup • March 24th 9am PST - Fault Tolerance and Processing Semantics with Apache Apex • March 28th 6pm PST - Low-latency ingestion and analytics with Apache Kafka and Apache Apex (Hadoop) • ...