Apache Airflow Architecture
9 likes2,386 views
Presentation given at Coolblue B.V. demonstrating Apache Airflow (incubating), what we learned from the underlying design principles and how an implementation of these principles reduce the amount of ETL effort. Why choose Airflow? Because it makes your engineering life easier, more people can contribute to how data flows through the organization, so that you can spend more time applying your brain to more difficult problems like Machine Learning, Deep Learning and higher level analysis.
1 of 34
![Follow important ETL principles
● Processes are idempotent
● Reusable, parameterisable components
● Data at rest between operations
s3://[bucket]/[dag-id]/[task-id]/2017/10/13](https://image.slidesharecdn.com/btsairflowarchitecture-171026124553/85/Apache-Airflow-Architecture-8-320.jpg)
![Follow important ETL principles
● Processes are idempotent
● Reusable, parameterisable components
● Data at rest between operations
s3://[my-bucket-name]/[dag-id]/[task-id]/2017/10/13
● Testing strategy](https://image.slidesharecdn.com/btsairflowarchitecture-171026124553/85/Apache-Airflow-Architecture-9-320.jpg)
Ad
Recommended
Introducing Apache Airflow and how we are using it
Introducing Apache Airflow and how we are using itBruno Faria This presentation is a brief introduction about Apache Airflow and how we implemented it. I explain how it can be used and some pros and cons.
Apache Airflow
Apache AirflowSumit Maheshwari Introductory talk on Apache Airflow (Incubator) by Sumit Maheshwari at recent Bangalore Big Data Meetup.
Introduction to Apache Airflow
Introduction to Apache Airflowmutt_data Introduction to Apache Airflow, it's main concepts and features and an example of a DAG. Afterwards some lessons and best practices learned by from the 3 years I have been using Airflow to power workflows in production.
Airflow 101
Airflow 101SaarBergerbest The document provides an overview of Apache Airflow, an open-source workflow management platform for data pipelines. It describes how Airflow allows users to programmatically author, schedule and monitor workflows or data pipelines via a GUI. It also outlines key Airflow concepts like DAGs (directed acyclic graphs), tasks, operators, sensors, XComs (cross-communication), connections, variables and executors that allow parallel task execution.
Orchestrating workflows Apache Airflow on GCP & AWS
Orchestrating workflows Apache Airflow on GCP & AWSDerrick Qin Working in a cloud or on-premises environment, we all somehow move data from A to B on-demand or on schedule. It is essential to have a tool that can automate recurring workflows. This can be anything from an ETL(Extract, Transform, and Load) job for a regular analytics report all the way to automatically re-training a machine learning model.
In this talk, we will introduce Apache Airflow and how it can help orchestrate your workflows. We will cover key concepts, features, and use cases of Apache Airflow, as well as how you can enjoy Apache Airflow on GCP and AWS by demo-ing a few practical workflows.
Apache airflow
Apache airflowPurna Chander This document provides an overview of building data pipelines using Apache Airflow. It discusses what a data pipeline is, common components of data pipelines like data ingestion and processing, and issues with traditional data flows. It then introduces Apache Airflow, describing its features like being fault tolerant and supporting Python code. The core components of Airflow including the web server, scheduler, executor, and worker processes are explained. Key concepts like DAGs, operators, tasks, and workflows are defined. Finally, it demonstrates Airflow through an example DAG that extracts and cleanses tweets.
Apache Airflow overview
Apache Airflow overviewNikolayGrishchenkov Apache Airflow is an open-source workflow management platform developed by Airbnb and now an Apache Software Foundation project. It allows users to define and manage data pipelines as directed acyclic graphs (DAGs) of tasks. The tasks can be operators to perform actions, move data between systems, and use sensors to monitor external systems. Airflow provides a rich web UI, CLI and integrations with databases, Hadoop, AWS and others. It is scalable, supports dynamic task generation and templates, alerting, retries, and distributed execution across clusters.
Airflow presentation
Airflow presentationIlias Okacha Airflow is a workflow management system for authoring, scheduling and monitoring workflows or directed acyclic graphs (DAGs) of tasks. It has features like DAGs to define tasks and their relationships, operators to describe tasks, sensors to monitor external systems, hooks to connect to external APIs and databases, and a user interface for visualizing pipelines and monitoring runs. Airflow uses a variety of executors like SequentialExecutor, CeleryExecutor and MesosExecutor to run tasks on schedulers like Celery or Kubernetes. It provides security features like authentication, authorization and impersonation to manage access.
Airflow - a data flow engine
Airflow - a data flow engineWalter Liu Airflow is a platform created by Airbnb to automate and schedule workflows. It uses a Directed Acyclic Graph (DAG) structure to define dependencies between tasks, and allows scheduling tasks on a timetable or triggering them manually. Some key features include monitoring task status, resuming failed tasks, backfilling historical data, and a web-based user interface. While additional databases are required for high availability, Airflow provides a flexible way to model complex data workflows as code.
Apache Airflow
Apache AirflowKnoldus Inc. In the session, we discussed the End-to-end working of Apache Airflow that mainly focused on "Why What and How" factors. It includes the DAG creation/implementation, Architecture, pros & cons. It also includes how the DAG is created for scheduling the Job and what all steps are required to create the DAG using python script & finally with the working demo.
Apache Airflow Introduction
Apache Airflow IntroductionLiangjun Jiang Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs). It allows defining and monitoring cron jobs, automating DevOps tasks, moving data periodically, and building machine learning pipelines. Many large companies use Airflow for tasks like data ingestion, analytics automation, and machine learning workflows. The author proposes using Airflow to manage data movement and automate tasks for their organization to benefit business units. Instructions are provided on installing Airflow using pip, Docker, or Helm along with developing sample DAGs connecting to Azure services like Blob Storage, Cosmos DB, and Databricks.
Airflow introduction
Airflow introductionChandler Huang This document provides an overview of Apache Airflow, an open-source workflow management system. It describes Airflow's key features like workflow definition using directed acyclic graphs (DAGs), rich UI, scheduler, operators for tasks like databases and web services, and use of Jinja templating. The document also discusses Airflow's architecture with parallel execution, UI, command line operations like backfilling, and security features. Airflow is used by over 200 companies for workflows like ETL, analytics, and machine learning pipelines.
Building a Data Pipeline using Apache Airflow (on AWS / GCP)
Building a Data Pipeline using Apache Airflow (on AWS / GCP)Yohei Onishi This is the slide I presented at PyCon SG 2019. I talked about overview of Airflow and how we can use Airflow and the other data engineering services on AWS and GCP to build data pipelines.
Airflow presentation
Airflow presentationAnant Corporation Slide deck for the fourth data engineering lunch, presented by guest speaker Will Angel. It covered the topic of using Airflow for data engineering. Airflow is a scheduling tool for managing data pipelines.
Apache Airflow
Apache AirflowKnoldus Inc. In the session, we discussed the End-to-end working of Apache Airflow that mainly focused on "Why What and How" factors. It includes the DAG creation/implementation, Architecture, pros & cons. It also includes how the DAG is created for scheduling the Job and what all steps are required to create the DAG using python script & finally with the working demo.
Building Better Data Pipelines using Apache Airflow
Building Better Data Pipelines using Apache AirflowSid Anand Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs). It allows users to programmatically author DAGs in Python without needing to bundle many XML files. The UI provides a tree view to see DAG runs over time and Gantt charts to see performance trends. Airflow is useful for ETL pipelines, machine learning workflows, and general job scheduling. It handles task dependencies and failures, monitors performance, and enforces service level agreements. Behind the scenes, the scheduler distributes tasks from the metadata database to Celery workers via RabbitMQ.
Airflow Intro-1.pdf
Airflow Intro-1.pdfBagustTriCahyo1 Airflow is a platform for authoring, scheduling, and monitoring workflows or data pipelines. It uses a directed acyclic graph (DAG) to define dependencies between tasks and schedule their execution. The UI provides dashboards to monitor task status and view workflow histories. Hands-on exercises demonstrate installing Airflow and creating sample DAGs.
Building an analytics workflow using Apache Airflow
Building an analytics workflow using Apache AirflowYohei Onishi This document discusses using Apache Airflow to build an analytics workflow. It begins with an overview of Airflow and how it can be used to author workflows through Python code. Examples are shown of using Airflow to copy files between S3 buckets. The document then covers setting up a highly available Airflow cluster, implementing continuous integration/deployment, and monitoring workflows. It emphasizes that Google Cloud Composer can simplify deploying and managing Airflow clusters on Google Kubernetes Engine and integrating with other Google Cloud services.
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow management
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow managementBurasakorn Sabyeying This document discusses Apache Airflow, an open-source workflow management platform for authoring, scheduling, and monitoring workflows or pipelines. It provides an overview of Airflow's key features and components, including Directed Acyclic Graphs (DAGs) for defining workflows as Python code, various operators for building tasks, and its rich web UI. The document compares Airflow to traditional cron jobs, noting Airflow can handle task dependencies and failures better than cron. It also outlines how to set up an Airflow cluster on multiple nodes for scaling workflows.
Introduction to Apache Airflow - Data Day Seattle 2016
Introduction to Apache Airflow - Data Day Seattle 2016Sid Anand Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs) of tasks. It includes a DAG scheduler, web UI, and CLI. Airflow allows users to author DAGs in Python without needing to bundle many XML files. The UI provides tree and Gantt chart views to monitor DAG runs over time. Airflow was accepted into the Apache Incubator in 2016 and has over 300 users from 40+ companies. Agari uses Airflow to orchestrate message scoring pipelines across AWS services like S3, Spark, SQS, and databases to enforce SLAs on correctness and timeliness. Areas for further improvement include security, APIs, execution scaling, and on
Airflow for Beginners
Airflow for BeginnersVarya Karpenko A presentation about Apache Airflow at PyCon & PyData Berlin 2019.
https://github.com/karpenkovarya/airflow_for_beginners
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeFlink Forward Flink Forward San Francisco 2022.
Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo.
by
Robert Metzger
Airflow tutorials hands_on
Airflow tutorials hands_onpko89403 This document provides an overview of Airflow, an open-source workflow management platform for authoring, scheduling and monitoring data pipelines. It describes Airflow's key components including the web server, scheduler, workers and metadata database. It explains how Airflow works by parsing DAGs, instantiating tasks and changing their state as they are scheduled, queued, run and monitored. The document also covers concepts like DAGs, operators, dependencies, concurrency vs parallelism and advanced topics such as subDAGs, hooks, XCOM and branching workflows.
Apache Airflow in Production
Apache Airflow in ProductionRobert Sanders We will introduce Airflow, an Apache Project for scheduling and workflow orchestration. We will discuss use cases, applicability and how best to use Airflow, mainly in the context of building data engineering pipelines. We have been running Airflow in production for about 2 years, we will also go over some learnings, best practices and some tools we have built around it.
Speakers: Robert Sanders, Shekhar Vemuri
Apache airflow
Apache airflowPavel Alexeev Apache Airflow is a platform to author, schedule and monitor workflows as directed acyclic graphs (DAGs) of tasks. It allows workflows to be defined as code making them more maintainable, versionable and collaborative. The rich user interface makes it easy to visualize pipelines and monitor progress. Key concepts include DAGs, operators, hooks, pools and xcoms. Alternatives include Azkaban from LinkedIn and Oozie for Hadoop workflows.
Where is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in FlinkFlink Forward Flinkn Forward San Francisco 2022.
In this talk, we will cover various topics around performance issues that can arise when running a Flink job and how to troubleshoot them. We’ll start with the basics, like understanding what the job is doing and what backpressure is. Next, we will see how to identify bottlenecks and which tools or metrics can be helpful in the process. Finally, we will also discuss potential performance issues during the checkpointing or recovery process, as well as and some tips and Flink features that can speed up checkpointing and recovery times.
by
Piotr Nowojski
How I learned to time travel, or, data pipelining and scheduling with Airflow
How I learned to time travel, or, data pipelining and scheduling with AirflowPyData This document discusses how the author learned to use Airflow for data pipelining and scheduling tasks. It describes some early tools like Cron and Luigi that were used for scheduling. It then evaluates options like Drake, Pydoit, Pinball, Luigi, and AWS Data Pipeline before settling on Airflow due to its sophistication in handling complex dependencies, built-in scheduling and monitoring, and flexibility. The author also develops a plugin called smart-airflow to add file-based checkpointing capabilities to Airflow to track intermediate data transformations.
Airflow at WePay
Airflow at WePayChris Riccomini A 20 minute talk about how WePay runs airflow. Discusses usage and operations. Also covers running Airflow in Google cloud.
Video of the talk is available here:
https://wepayinc.box.com/s/hf1chwmthuet29ux2a83f5quc8o5q18k
Airflow techtonic template
Airflow techtonic templateSampath Kumar This document provides an overview of Apache Airflow, a workflow automation and scheduling system for data engineering. It discusses key Airflow concepts like DAGs, tasks, hooks and variables. It demonstrates a sample data engineering pipeline using an S3 sensor task and loading data into Elasticsearch. The document also covers Airflow architecture, features, challenges and recommendations for use. It aims to explain how Airflow can be used to build scalable and cost-efficient data pipelines.
Buliding Reliable Data Apps
Buliding Reliable Data AppsGleb Mezhanskiy Presented on Data Quality Meetup by Max Gasner, Co-author of Dagster – data orchestration framework for ETL
https://datafold.com/
Ad
More Related Content
What's hot (20)
Airflow - a data flow engine
Airflow - a data flow engineWalter Liu Airflow is a platform created by Airbnb to automate and schedule workflows. It uses a Directed Acyclic Graph (DAG) structure to define dependencies between tasks, and allows scheduling tasks on a timetable or triggering them manually. Some key features include monitoring task status, resuming failed tasks, backfilling historical data, and a web-based user interface. While additional databases are required for high availability, Airflow provides a flexible way to model complex data workflows as code.
Apache Airflow
Apache AirflowKnoldus Inc. In the session, we discussed the End-to-end working of Apache Airflow that mainly focused on "Why What and How" factors. It includes the DAG creation/implementation, Architecture, pros & cons. It also includes how the DAG is created for scheduling the Job and what all steps are required to create the DAG using python script & finally with the working demo.
Apache Airflow Introduction
Apache Airflow IntroductionLiangjun Jiang Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs). It allows defining and monitoring cron jobs, automating DevOps tasks, moving data periodically, and building machine learning pipelines. Many large companies use Airflow for tasks like data ingestion, analytics automation, and machine learning workflows. The author proposes using Airflow to manage data movement and automate tasks for their organization to benefit business units. Instructions are provided on installing Airflow using pip, Docker, or Helm along with developing sample DAGs connecting to Azure services like Blob Storage, Cosmos DB, and Databricks.
Airflow introduction
Airflow introductionChandler Huang This document provides an overview of Apache Airflow, an open-source workflow management system. It describes Airflow's key features like workflow definition using directed acyclic graphs (DAGs), rich UI, scheduler, operators for tasks like databases and web services, and use of Jinja templating. The document also discusses Airflow's architecture with parallel execution, UI, command line operations like backfilling, and security features. Airflow is used by over 200 companies for workflows like ETL, analytics, and machine learning pipelines.
Building a Data Pipeline using Apache Airflow (on AWS / GCP)
Building a Data Pipeline using Apache Airflow (on AWS / GCP)Yohei Onishi This is the slide I presented at PyCon SG 2019. I talked about overview of Airflow and how we can use Airflow and the other data engineering services on AWS and GCP to build data pipelines.
Airflow presentation
Airflow presentationAnant Corporation Slide deck for the fourth data engineering lunch, presented by guest speaker Will Angel. It covered the topic of using Airflow for data engineering. Airflow is a scheduling tool for managing data pipelines.
Apache Airflow
Apache AirflowKnoldus Inc. In the session, we discussed the End-to-end working of Apache Airflow that mainly focused on "Why What and How" factors. It includes the DAG creation/implementation, Architecture, pros & cons. It also includes how the DAG is created for scheduling the Job and what all steps are required to create the DAG using python script & finally with the working demo.
Building Better Data Pipelines using Apache Airflow
Building Better Data Pipelines using Apache AirflowSid Anand Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs). It allows users to programmatically author DAGs in Python without needing to bundle many XML files. The UI provides a tree view to see DAG runs over time and Gantt charts to see performance trends. Airflow is useful for ETL pipelines, machine learning workflows, and general job scheduling. It handles task dependencies and failures, monitors performance, and enforces service level agreements. Behind the scenes, the scheduler distributes tasks from the metadata database to Celery workers via RabbitMQ.
Airflow Intro-1.pdf
Airflow Intro-1.pdfBagustTriCahyo1 Airflow is a platform for authoring, scheduling, and monitoring workflows or data pipelines. It uses a directed acyclic graph (DAG) to define dependencies between tasks and schedule their execution. The UI provides dashboards to monitor task status and view workflow histories. Hands-on exercises demonstrate installing Airflow and creating sample DAGs.
Building an analytics workflow using Apache Airflow
Building an analytics workflow using Apache AirflowYohei Onishi This document discusses using Apache Airflow to build an analytics workflow. It begins with an overview of Airflow and how it can be used to author workflows through Python code. Examples are shown of using Airflow to copy files between S3 buckets. The document then covers setting up a highly available Airflow cluster, implementing continuous integration/deployment, and monitoring workflows. It emphasizes that Google Cloud Composer can simplify deploying and managing Airflow clusters on Google Kubernetes Engine and integrating with other Google Cloud services.
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow management
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow managementBurasakorn Sabyeying This document discusses Apache Airflow, an open-source workflow management platform for authoring, scheduling, and monitoring workflows or pipelines. It provides an overview of Airflow's key features and components, including Directed Acyclic Graphs (DAGs) for defining workflows as Python code, various operators for building tasks, and its rich web UI. The document compares Airflow to traditional cron jobs, noting Airflow can handle task dependencies and failures better than cron. It also outlines how to set up an Airflow cluster on multiple nodes for scaling workflows.
Introduction to Apache Airflow - Data Day Seattle 2016
Introduction to Apache Airflow - Data Day Seattle 2016Sid Anand Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs) of tasks. It includes a DAG scheduler, web UI, and CLI. Airflow allows users to author DAGs in Python without needing to bundle many XML files. The UI provides tree and Gantt chart views to monitor DAG runs over time. Airflow was accepted into the Apache Incubator in 2016 and has over 300 users from 40+ companies. Agari uses Airflow to orchestrate message scoring pipelines across AWS services like S3, Spark, SQS, and databases to enforce SLAs on correctness and timeliness. Areas for further improvement include security, APIs, execution scaling, and on
Airflow for Beginners
Airflow for BeginnersVarya Karpenko A presentation about Apache Airflow at PyCon & PyData Berlin 2019.
https://github.com/karpenkovarya/airflow_for_beginners
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeFlink Forward Flink Forward San Francisco 2022.
Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo.
by
Robert Metzger
Airflow tutorials hands_on
Airflow tutorials hands_onpko89403 This document provides an overview of Airflow, an open-source workflow management platform for authoring, scheduling and monitoring data pipelines. It describes Airflow's key components including the web server, scheduler, workers and metadata database. It explains how Airflow works by parsing DAGs, instantiating tasks and changing their state as they are scheduled, queued, run and monitored. The document also covers concepts like DAGs, operators, dependencies, concurrency vs parallelism and advanced topics such as subDAGs, hooks, XCOM and branching workflows.
Apache Airflow in Production
Apache Airflow in ProductionRobert Sanders We will introduce Airflow, an Apache Project for scheduling and workflow orchestration. We will discuss use cases, applicability and how best to use Airflow, mainly in the context of building data engineering pipelines. We have been running Airflow in production for about 2 years, we will also go over some learnings, best practices and some tools we have built around it.
Speakers: Robert Sanders, Shekhar Vemuri
Apache airflow
Apache airflowPavel Alexeev Apache Airflow is a platform to author, schedule and monitor workflows as directed acyclic graphs (DAGs) of tasks. It allows workflows to be defined as code making them more maintainable, versionable and collaborative. The rich user interface makes it easy to visualize pipelines and monitor progress. Key concepts include DAGs, operators, hooks, pools and xcoms. Alternatives include Azkaban from LinkedIn and Oozie for Hadoop workflows.
Where is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in FlinkFlink Forward Flinkn Forward San Francisco 2022.
In this talk, we will cover various topics around performance issues that can arise when running a Flink job and how to troubleshoot them. We’ll start with the basics, like understanding what the job is doing and what backpressure is. Next, we will see how to identify bottlenecks and which tools or metrics can be helpful in the process. Finally, we will also discuss potential performance issues during the checkpointing or recovery process, as well as and some tips and Flink features that can speed up checkpointing and recovery times.
by
Piotr Nowojski
How I learned to time travel, or, data pipelining and scheduling with Airflow
How I learned to time travel, or, data pipelining and scheduling with AirflowPyData This document discusses how the author learned to use Airflow for data pipelining and scheduling tasks. It describes some early tools like Cron and Luigi that were used for scheduling. It then evaluates options like Drake, Pydoit, Pinball, Luigi, and AWS Data Pipeline before settling on Airflow due to its sophistication in handling complex dependencies, built-in scheduling and monitoring, and flexibility. The author also develops a plugin called smart-airflow to add file-based checkpointing capabilities to Airflow to track intermediate data transformations.
Airflow at WePay
Airflow at WePayChris Riccomini A 20 minute talk about how WePay runs airflow. Discusses usage and operations. Also covers running Airflow in Google cloud.
Video of the talk is available here:
https://wepayinc.box.com/s/hf1chwmthuet29ux2a83f5quc8o5q18k
Similar to Apache Airflow Architecture (20)
Airflow techtonic template
Airflow techtonic templateSampath Kumar This document provides an overview of Apache Airflow, a workflow automation and scheduling system for data engineering. It discusses key Airflow concepts like DAGs, tasks, hooks and variables. It demonstrates a sample data engineering pipeline using an S3 sensor task and loading data into Elasticsearch. The document also covers Airflow architecture, features, challenges and recommendations for use. It aims to explain how Airflow can be used to build scalable and cost-efficient data pipelines.
Buliding Reliable Data Apps
Buliding Reliable Data AppsGleb Mezhanskiy Presented on Data Quality Meetup by Max Gasner, Co-author of Dagster – data orchestration framework for ETL
https://datafold.com/
Gowthami_Resume
Gowthami_ResumeGowthami Subramaniam Gowthami S is a software developer and designer with over 2 years of experience in data warehousing using databases like Teradata and Oracle. She has extensive experience with ETL tools like Informatica and data loading utilities for Teradata. She has worked on full data warehouse development lifecycles including requirements, design, implementation and maintenance. Currently working as a software engineer at Tech Mahindra, her projects include developing ETL processes and Teradata SQL queries to load and transform data from various sources into a Cisco enterprise data warehouse supporting business intelligence reporting and analytics.
Resume
Resumerajeswari p This document is a curriculum vitae for Rajeswari Pothala. It outlines her professional experience working for Tata Consultancy Services for over 6 years leading teams of up to 8 members on data warehousing and ETL development projects. It also lists her educational qualifications including a B.Tech in Electronics and Communication Engineering. Key projects outlined include work on the TCS Trimatrix EDW project and several projects for Aviva involving data integration, mappings development, and module lead responsibilities.
SAP Teched 2012 Session Tec3438 Automate IaaS SAP deployments
SAP Teched 2012 Session Tec3438 Automate IaaS SAP deploymentsChris Kernaghan This document summarizes automation of infrastructure as a service deployments and monitoring. It discusses Infrastructure as a Service (IaaS) and how IaaS environments allow for scalable, on-demand provisioning of computing resources. It also discusses SAP's support for AWS and how Capgemini UK uses AWS for SAP deployments. The document advocates for automating infrastructure tasks to improve consistency, auditability and repeatability. It provides examples of automation for build processes, configuration management, change management, exception monitoring, and other areas. Overall, the document promotes automating infrastructure processes in IaaS environments to improve agility, reduce costs, and ensure compliance.
Ashish_Maheshwari_Data_Analyst
Ashish_Maheshwari_Data_AnalystAshish Maheshwari Ashish Maheshwari has over 8 years of experience in data modeling, integration, analytics, mining, ETL processes, BI applications, and data warehousing. He currently works as a Technical Lead at Ness Technologies, Bangalore and has experience working on projects for clients such as FGL, Marks, and Target. His skills include Oracle SQL, PL/SQL, Datastage, data warehousing concepts, R programming, and Agile development processes.
Microsoft Azure Big Data Analytics
Microsoft Azure Big Data AnalyticsMark Kromer This is a run-through at a 200 level of the Microsoft Azure Big Data Analytics for the Cloud data platform based on the Cortana Intelligence Suite offerings.
Sudeshna Ghosh Dastidar-Resume
Sudeshna Ghosh Dastidar-ResumeSudeshna Ghosh Dastidar Sudeshna Ghosh Dastidar has over 6 years of experience as an IT Analyst at TCS. She has expertise in design, development, testing, implementation, and reporting using tools like Crystal Reports, SAP BO, and SQL. She is proficient in Oracle, SQL, PL/SQL, and has experience in data warehousing and ETL using Informatica. She has worked on projects in domains like banking, financial services, insurance, and telecom.
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...Daniel Zivkovic Two #ModernDataStack talks and one DevOps talk: https://youtu.be/4R--iLnjCmU
1. "From Data-driven Business to Business-driven Data: Hands-on #DataModelling exercise" by Jacob Frackson of Montreal Analytics
2. "Trends in the #DataEngineering Consulting Landscape" by Nadji Bessa of Infostrux Solutions
3. "Building Secure #Serverless Delivery Pipelines on #GCP" by Ugo Udokporo of Google Cloud Canada
We ran out of time for the 4th presenter, so the event will CONTINUE in March... stay tuned! Compliments of #ServerlessTO.
Amit_Kumar_CV
Amit_Kumar_CVAmit Kumar This document contains a summary of Amit Kumar's professional experience and qualifications. He has over 9 years of IT experience, including 8 years of data warehouse and business intelligence experience. Currently he works as a data architect at Capgemini, leading a team of 12 on an oil and gas equipment install base project. He has extensive experience designing and developing data integration solutions using tools like Informatica, Hadoop, and SQL.
ADV Slides: The Evolution of the Data Platform and What It Means to Enterpris...
ADV Slides: The Evolution of the Data Platform and What It Means to Enterpris...DATAVERSITY Thirty years is a long time for a technology foundation to be as active as relational databases. Are their replacements here?
In this webinar, we look at this foundational technology for modern Data Management and show how it evolved to meet the workloads of today, as well as when other platforms make sense for enterprise data.
Infrastructure Agnostic Machine Learning Workload Deployment
Infrastructure Agnostic Machine Learning Workload DeploymentDatabricks When it comes to Large Scale data processing and Machine Learning, Apache Spark is no doubt one of the top battle-tested frameworks out there for handling batched or streaming workloads. The ease of use, built-in Machine Learning modules, and multi-language support makes it a very attractive choice for data wonks. However bootstrapping and getting off the ground could be difficult for most teams without leveraging a Spark cluster that is already pre-provisioned and provided as a managed service in the Cloud, while this is a very attractive choice to get going, in the long run, it could be a very expensive option if it’s not well managed.
As an alternative to this approach, our team has been exploring and working a lot with running Spark and all our Machine Learning workloads and pipelines as containerized Docker packages on Kubernetes. This provides an infrastructure-agnostic abstraction layer for us, and as a result, it improves our operational efficiency and reduces our overall compute cost. Most importantly, we can easily target our Spark workload deployment to run on any major Cloud or On-prem infrastructure (with Kubernetes as the common denominator) by just modifying a few configurations.
In this talk, we will walk you through the process our team follows to make it easy for us to run a production deployment of our Machine Learning workloads and pipelines on Kubernetes which seamlessly allows us to port our implementation from a local Kubernetes set up on the laptop during development to either an On-prem or Cloud Kubernetes environment
Postgres survey podcast
Postgres survey podcastinside-BigData.com In this slidecast, Ed Boyajian from EnterpriseDB presents: Postgres - A Tipping Point?
"PostgreSQL is the #1 enterprise-class open source database with a feature set comparable to the major proprietary RDBMS vendors and a customer list that spans every industry. EnterpriseDB’s Postgres Plus solutions let you confidently develop and deploy PostgreSQL-backed applications that scale all the way from embedded solutions to massive OLTP and data warehouse systems that serve thousands of users."
Learn more at:
http://enterprisedb.com
Watch the presentation:
http://inside-bigdata.com/slidecast-postgres-a-tipping-point/
Automating Infrastructure as a Service Deployments and monitoring – TEC213
Automating Infrastructure as a Service Deployments and monitoring – TEC213Chris Kernaghan The document discusses automating infrastructure as a service deployments and monitoring. It covers several topics:
- IaaS environments allow for scalable cloud computing resources billed based on usage. SAP has been working with Amazon Web Services since 2008.
- Automation can schedule repetitive tasks, enable consistent processes, and provide auditable records. DevOps focuses on collaboration, automation, measurement, and sharing to create flexible infrastructure.
- Automating infrastructure provisioning, configuration management, change management, and exception monitoring can improve speed, reduce costs, and ensure compliance. Cloud security also needs automation to ensure data protection with the cloud's flexibility.
Alteryx Desktop Designer Overview
Alteryx Desktop Designer OverviewTridant The Alteryx Designer solves this by delivering an intuitive workflow for data blending and advanced analytics that leads to deeper insights in hours, not the weeks typical of traditional approaches! The Alteryx Designer empowers data analysts by combining data blending, predictive analytics, spatial analytics, and reporting, visualization and analytic apps into one workflow.
Triple 20 IT – How to reduce costs on target while increasing speed and quality
Triple 20 IT – How to reduce costs on target while increasing speed and qualityStormForge .io 1) The webinar will discuss how to reduce IT costs while increasing speed and quality using a method called "Triple 20 IT" with presenters Benedikt Stemmildt and Niels Roetert.
2) Niels will discuss using machine learning for cost efficiency in Kubernetes to optimize resource allocation.
3) There will be a live demo and question/answer session.
Ajith_kumar_4.3 Years_Informatica_ETL
Ajith_kumar_4.3 Years_Informatica_ETLAjith Kumar Pampatti The candidate has over 4 years of experience as an ETL developer using Informatica. They have extensive experience developing mappings using transformations like aggregator, lookup, filter and joiner. They have worked on multiple projects in the telecommunications domain for clients like British Telecom extracting data from various sources and loading to data warehouses. Their skills include Informatica, Oracle, SQL and they have experience in requirements analysis, mapping development, testing and support.
GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...
GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...James Anderson Do you know The Cloud Girl? She makes the cloud come alive with pictures and storytelling.
The Cloud Girl, Priyanka Vergadia, Chief Content Officer @Google, joins us to tell us about Scaleable Data Analytics in Google Cloud.
Maybe, with her explanation, we'll finally understand it!
Priyanka is a technical storyteller and content creator who has created over 300 videos, articles, podcasts, courses and tutorials which help developers learn Google Cloud fundamentals, solve their business challenges and pass certifications! Checkout her content on Google Cloud Tech Youtube channel.
Priyanka enjoys drawing and painting which she tries to bring to her advocacy.
Check out her website The Cloud Girl: https://thecloudgirl.dev/ and her new book: https://www.amazon.com/Visualizing-Google-Cloud-Illustrated-References/dp/1119816327
Choosing the right IDP Solution
Choosing the right IDP SolutionProvectus Looking to make your document processing operations more effective and cost-efficient with AI/ML? Learn from the experts of Provectus and Amazon Web Services (AWS) how to choose the right solution for your company! We will look into the management and engineering perspectives of AI document processing, from industry use cases and the solution map to our unique methodology for assessing available document processing solutions to Provectus IDP. Whether you are looking for a ready-made solution or you plan to build a custom solution of your own, this webinar will help you find the best option for your business.
Agenda
- Introductions
- Industry use cases
- Intelligent Document Processing (IDP) overview
- IDP Solutions map
- AWS IDP Solution
- Provectus IDP Platform
- Q&A
Intended Audience
Technology executives and decision makers, including such roles as CIO, CCO, COO, and CDO; digital transformation managers; data and ML engineers.
Presenters
Almir Davletov, IDP Subject Matter Expert, Provectus
Yaroslav Tarasyuk, Business Development, Provectus
Sonali Sahu, Sr. Solutions Architect, AWS
Interested? Learn more about Provectus Intelligent Document Processing Solution: https://provectus.com/document-processing-solution/
Analyzing Business Requirements in a Visible Enterprise
Analyzing Business Requirements in a Visible EnterpriseDafna Levy The document discusses business requirements analysis and process modeling using Metastorm Provision BPA. It describes how BPA can be used to close gaps in business processes by providing a common language between business and IT. BPA captures requirements as software models to gain consensus and jointly analyze and design processes.
Ad
Recently uploaded (20)
Automation Dreamin': Capture User Feedback From Anywhere
Automation Dreamin': Capture User Feedback From AnywhereLynda Kane Slide Deck from Automation Dreamin' 2022 presentation Capture User Feedback from Anywhere
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp This is the keynote of the Into the Box conference, highlighting the release of the BoxLang JVM language, its key enhancements, and its vision for the future.
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsInData Labs In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy.
This infographic contains:
-AI and data privacy: Key findings
-Statistics on AI data privacy in the today’s world
-Tips on how to overcome data privacy challenges
-Benefits of AI data security investments.
Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.
Drupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove How to measure web front-end energy consumption using Firefox Profiler. Presented in DrupalCamp Finland on April 25th, 2025.
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john Analyze the growth of meme coins from mere online jokes to potential assets in the digital economy. Explore the community, culture, and utility as they elevate themselves to a new era in cryptocurrency.
Hands On: Create a Lightning Aura Component with force:RecordData
Hands On: Create a Lightning Aura Component with force:RecordDataLynda Kane Slide Deck from the 3/26/2020 virtual meeting of the Cleveland Developer Group presentation on creating a Lightning Aura Component using force:RecordData.
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBrian McKeiver May 2nd, 2025 talk at StirTrek 2025 Conference.
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your UsersLynda Kane Slide Deck from Automation Dreamin'2022 presentation Sharing Some Gratitude with Your Users on creating a Flow to present a random statement of Gratitude to a User in Salesforce.
Role of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo From predictive maintenance to robotic automation, AI is driving the future of manufacturing. But without high-quality annotated data, even the smartest models fall short.
Discover how data annotation services are powering accuracy, safety, and efficiency in AI-driven manufacturing systems.
Precision in data labeling = Precision on the production floor.
Semantic Cultivators : The Critical Future Role to Enable AI
Semantic Cultivators : The Critical Future Role to Enable AIartmondano By 2026, AI agents will consume 10x more enterprise data than humans, but with none of the contextual understanding that prevents catastrophic misinterpretations.
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptx
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock Building 10x Organizations with Modern Productivity Metrics
10x developers may be a myth, but 10x organizations are very real, as proven by the influential study performed in the 1980s, ‘The Coding War Games.’
Right now, here in early 2025, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience. It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them.
But which of these approaches actually work? DORA? SPACE? DevEx? What should we invest in and create urgency behind today, so that we don’t find ourselves having the same discussion again in a decade?
Network Security. Different aspects of Network Security.
Network Security. Different aspects of Network Security.gregtap1 Network Security. Different aspects of Network Security.
Big Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan This is a Quick Research Guide (QRG).
QRGs include the following:
- A brief, high-level overview of the QRG topic.
- A milestone timeline for the QRG topic.
- Links to various free online resource materials to provide a deeper dive into the QRG topic.
- Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic.
QRGs planned for the series:
- Artificial Intelligence QRG
- Quantum Computing QRG
- Big Data Analytics QRG
- Spacecraft Guidance, Navigation & Control QRG (coming 2026)
- UK Home Computing & The Birth of ARM QRG (coming 2027)
Any questions or comments?
- Please contact Arthur Morgan at [email protected].
100% human made.
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts!
📕 Agenda
Welcome & Introductions
Orchestrator API Overview
Exploring the Swagger Interface
Test Manager API Highlights
Streamlining Automation & Testing with APIs (Demo)
Q&A and Open Discussion
Perfect for developers, testers, and automation enthusiasts!
👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/
This session streamed live on April 29, 2025, 18:00 CET.
Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3 The latest updates on Manifest's pre-seed stage progress.
Learn the Basics of Agile Development: Your Step-by-Step Guide
Learn the Basics of Agile Development: Your Step-by-Step GuideMarcel David New to Agile? This step-by-step guide is your perfect starting point. "Learn the Basics of Agile Development" simplifies complex concepts, providing you with a clear understanding of how Agile can improve software development and project management. Discover the benefits of iterative work, team collaboration, and flexible planning.
Linux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru Introduction to LPIC-1 Exam - overview, exam details, price and job opportunities
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat
Ad
Apache Airflow Architecture
- 1. Gerard Toonstra | Sr. Data Platform Engineer | [email protected] | 08-10-2017
- 3. “Seek more understanding through data analytics about business processes.”
- 5. Where is time spent?
- 6. Follow important ETL principles ● Processes are idempotent
- 7. Follow important ETL principles ● Processes are idempotent ● Reusable, parameterisable components
- 8. Follow important ETL principles ● Processes are idempotent ● Reusable, parameterisable components ● Data at rest between operations s3://[bucket]/[dag-id]/[task-id]/2017/10/13
- 9. Follow important ETL principles ● Processes are idempotent ● Reusable, parameterisable components ● Data at rest between operations s3://[my-bucket-name]/[dag-id]/[task-id]/2017/10/13 ● Testing strategy
- 10. “Airflow is a platform to programmatically author, schedule and monitor workflows.”
- 11. What is a scheduler?
- 12. What is a DAG?
- 14. Operator?
- 27. extract_customer = PostgresToPostgresOperator( src_postgres_conn_id='postgres_oltp', dest_postgress_conn_id='postgres_dwh', sql='select_customer.sql', pg_table='staging.customer', parameters={"window_start_date":"{{ ds }}", "window_end_date":"{{ tomorrow_ds }}"}, pool='postgres_dwh')
- 28. extract_customer = PostgresToPostgresOperator( src_postgres_conn_id='postgres_oltp', dest_postgress_conn_id='postgres_dwh', sql='select_customer.sql', pg_table='staging.customer', parameters={"window_start_date":"{{ ds }}", "window_end_date":"{{ tomorrow_ds }}"}, pool='postgres_dwh')
- 29. extract_customer = PostgresToPostgresOperator( src_postgres_conn_id='postgres_oltp', dest_postgress_conn_id='postgres_dwh', sql='select_customer.sql', pg_table='staging.customer', parameters={"window_start_date":"{{ ds }}", "window_end_date":"{{ tomorrow_ds }}"}, pool='postgres_dwh')
- 30. extract_customer = PostgresToPostgresOperator( src_postgres_conn_id='postgres_oltp', dest_postgress_conn_id='postgres_dwh', sql='select_customer.sql', pg_table='staging.customer', parameters={"window_start_date":"{{ ds }}", "window_end_date":"{{ tomorrow_ds }}"}, pool='postgres_dwh')
- 31. Where is time spent?
- 32. Follow important ETL principles ● Idempotency ● Reusable, parameterizable components ● Data from rest to rest ● Testing strategy
- 34. Gerard Toonstra | Sr. Data Platform Engineer | [email protected]