Airflow - a data flow engine

Feb 17, 2016Download as PPTX, PDF33 likes11,178 views

Airflow is a platform created by Airbnb to automate and schedule workflows. It uses a Directed Acyclic Graph (DAG) structure to define dependencies between tasks, and allows scheduling tasks on a timetable or triggering them manually. Some key features include monitoring task status, resuming failed tasks, backfilling historical data, and a web-based user interface. While additional databases are required for high availability, Airflow provides a flexible way to model complex data workflows as code.

AIRFLOW - DATA FLOW
ENGINE FROM AIRBNB
By Walter Liu 2016/01/28

Who am I?
 Archer Architect of TrendMicro Coretech
backend
 A new user in Airflow

Data relationships
 Data availability
 if the data is not there, trigger the process to
generate the data.
 Data dependency
 Some data relies on other data to generate.

Operability
 Job failed and resume
 Job monitor
 Backfill

Demo
 python tutorial.py
 airflow list_dags
 airflow list_tasks tutorial
 airflow list_tasks tutorial --tree
 airflow test tutorial print_date 2015-06-01
 airflow test tutorial sleep 2015-06-01
 airflow run tutorial templated 2015-06-01
 Backfill: airflow backfill tutorial -s 2015-06-07 -e 2015-06-10
 Run again
 Run another date range
Site: http://localhost:8080/admin/

Airflow - Pros
 Dynamic generating path
 Have both Time scheduler and Command line trigger
 Has Master/Worker model (automatically distribute tasks)
 Scale if you have many tasks in a chain.
 But not be so useful to most of our tasks. Maybe useful for full dump.
 Fancy UI
 Dependencies of tasks
 Task success/failure
 Scheduled tasks status
 Has utility lib to wait_data for S3, Hadoop …

Airflow - Cons
 Additional DB/Redis or Rabbitmq for Celery
 HA design: Use RDBMS/redis-cache in AWS
 Require python 2.7 and many other libraries.
 Not dependent on data. Just task dependency.
(Not big cons)
 Write check data file code.

Recap
 Solving DAG job management problem
 Features to improve daily operation
 Monitor/Visualization
 Job failure and resume
 Backfill

Other Selections?
 Oozie (on Hadoop)
 Luigi (by Spotify)
 Mario (Luigi in Scala)
 Ruigi (Luigi in R)
 Airflow (by Airbnb)
 Mistral (by OpenStack)
 Pinball (by Pinterest)
 …

Github statistics
Start Date Star Commits in
recent 2
months
Airflow 2014/10/05 1516 362
Luigi 2011/11/13 3777 105
Pinball 2015/05/01 476 0
Oozie 2011/08/28 167 14

Oozie
 Pros
 Mature
 Native support of Hadoop ECO system
 Python is not required.
 Cons
 Big and complex XML to define the flow and config.
 Control flow is somehow restrictive
 Hadoop ECO system is required.
 Unix box, Java, Hadoop, Pig, ExtJS library

Luigi - Pros
 Makefile like, dependencies on data (Upward instead of downward)
 Command line trigger
 Not only Hadoop
 Support target: HDFS/S3/RDBMS/…. (not limited)
 Write your own code to support Casandra, etc.
 Dependencies are decentralized, no big XML.
 UI
 Dynamic dependency/task is supported. (Don’t know it is better than
airflow or not)
 [util] Date algebra

Luigi - Cons
 No built-in triggering
 No crontab like things (could borrow Chronos)
 Use cron to run tasks on specific nodes.
(normally)

Luigi - Notes
 Local execution
 Pros: Easy to debug
 Cons:
 need access to other system. (Well, other system didn’t achieve this either.)
 No scalability if you have many tasks in a chain.
 Centralized scheduler servers (optional, recommended for
production)
 Make sure two instances of the same task are not running
simultaneously
 Provide visualization of everything that’s going on.
 HA option
 Run 2+ tasks on 2+ machines at the same time

References
 http://erikbern.com/2015/07/02/more-luigi-
alternatives/

This document provides an overview of Apache Airflow, an open-source workflow management system. It describes Airflow's key features like workflow definition using directed acyclic graphs (DAGs), rich UI, scheduler, operators for tasks like databases and web services, and use of Jinja templating. The document also discusses Airflow's architecture with parallel execution, UI, command line operations like backfilling, and security features. Airflow is used by over 200 companies for workflows like ETL, analytics, and machine learning pipelines.

Introducing Apache Airflow and how we are using itBruno Faria

Airflow presentationAnant Corporation

Apache AirflowKnoldus Inc.

Apache airflowPurna Chander

This document provides an overview of building data pipelines using Apache Airflow. It discusses what a data pipeline is, common components of data pipelines like data ingestion and processing, and issues with traditional data flows. It then introduces Apache Airflow, describing its features like being fault tolerant and supporting Python code. The core components of Airflow including the web server, scheduler, executor, and worker processes are explained. Key concepts like DAGs, operators, tasks, and workflows are defined. Finally, it demonstrates Airflow through an example DAG that extracts and cleanses tweets.

Intro to Airflow: Goodbye Cron, Welcome scheduled workflow managementBurasakorn Sabyeying

This document discusses Apache Airflow, an open-source workflow management platform for authoring, scheduling, and monitoring workflows or pipelines. It provides an overview of Airflow's key features and components, including Directed Acyclic Graphs (DAGs) for defining workflows as Python code, various operators for building tasks, and its rich web UI. The document compares Airflow to traditional cron jobs, noting Airflow can handle task dependencies and failures better than cron. It also outlines how to set up an Airflow cluster on multiple nodes for scaling workflows.

Airflow for BeginnersVarya Karpenko

Building an analytics workflow using Apache AirflowYohei Onishi

This document discusses using Apache Airflow to build an analytics workflow. It begins with an overview of Airflow and how it can be used to author workflows through Python code. Examples are shown of using Airflow to copy files between S3 buckets. The document then covers setting up a highly available Airflow cluster, implementing continuous integration/deployment, and monitoring workflows. It emphasizes that Google Cloud Composer can simplify deploying and managing Airflow clusters on Google Kubernetes Engine and integrating with other Google Cloud services.

Airflow presentationIlias Okacha

Airflow is a workflow management system for authoring, scheduling and monitoring workflows or directed acyclic graphs (DAGs) of tasks. It has features like DAGs to define tasks and their relationships, operators to describe tasks, sensors to monitor external systems, hooks to connect to external APIs and databases, and a user interface for visualizing pipelines and monitoring runs. Airflow uses a variety of executors like SequentialExecutor, CeleryExecutor and MesosExecutor to run tasks on schedulers like Celery or Kubernetes. It provides security features like authentication, authorization and impersonation to manage access.

Apache Airflow overviewNikolayGrishchenkov

Apache Airflow is an open-source workflow management platform developed by Airbnb and now an Apache Software Foundation project. It allows users to define and manage data pipelines as directed acyclic graphs (DAGs) of tasks. The tasks can be operators to perform actions, move data between systems, and use sensors to monitor external systems. Airflow provides a rich web UI, CLI and integrations with databases, Hadoop, AWS and others. It is scalable, supports dynamic task generation and templates, alerting, retries, and distributed execution across clusters.

Introduction to Apache Airflowmutt_data

Apache Airflow ArchitectureGerard Toonstra

Presentation given at Coolblue B.V. demonstrating Apache Airflow (incubating), what we learned from the underlying design principles and how an implementation of these principles reduce the amount of ETL effort. Why choose Airflow? Because it makes your engineering life easier, more people can contribute to how data flows through the organization, so that you can spend more time applying your brain to more difficult problems like Machine Learning, Deep Learning and higher level analysis.

Building a Data Pipeline using Apache Airflow (on AWS / GCP)Yohei Onishi

Airflow 101SaarBergerbest

The document provides an overview of Apache Airflow, an open-source workflow management platform for data pipelines. It describes how Airflow allows users to programmatically author, schedule and monitor workflows or data pipelines via a GUI. It also outlines key Airflow concepts like DAGs (directed acyclic graphs), tasks, operators, sensors, XComs (cross-communication), connections, variables and executors that allow parallel task execution.

Apache airflowPavel Alexeev

Apache Airflow is a platform to author, schedule and monitor workflows as directed acyclic graphs (DAGs) of tasks. It allows workflows to be defined as code making them more maintainable, versionable and collaborative. The rich user interface makes it easy to visualize pipelines and monitor progress. Key concepts include DAGs, operators, hooks, pools and xcoms. Alternatives include Azkaban from LinkedIn and Oozie for Hadoop workflows.

Airflow Intro-1.pdfBagustTriCahyo1

How I learned to time travel, or, data pipelining and scheduling with AirflowPyData

This document discusses how the author learned to use Airflow for data pipelining and scheduling tasks. It describes some early tools like Cron and Luigi that were used for scheduling. It then evaluates options like Drake, Pydoit, Pinball, Luigi, and AWS Data Pipeline before settling on Airflow due to its sophistication in handling complex dependencies, built-in scheduling and monitoring, and flexibility. The author also develops a plugin called smart-airflow to add file-based checkpointing capabilities to Airflow to track intermediate data transformations.

Airflow Best Practises & Roadmap to Airflow 2.0Kaxil Naik

This document provides an overview of new features in Airflow 1.10.8/1.10.9 and best practices for writing DAGs and configuring Airflow for production. It also outlines the roadmap for Airflow 2.0, including dag serialization, a revamped real-time UI, developing a production-grade modern API, releasing official Docker/Helm support, and improving the scheduler. The document aims to help users understand recent Airflow updates and plan their migration to version 2.0.

Apache Airflow IntroductionLiangjun Jiang

Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs). It allows defining and monitoring cron jobs, automating DevOps tasks, moving data periodically, and building machine learning pipelines. Many large companies use Airflow for tasks like data ingestion, analytics automation, and machine learning workflows. The author proposes using Airflow to manage data movement and automate tasks for their organization to benefit business units. Instructions are provided on installing Airflow using pip, Docker, or Helm along with developing sample DAGs connecting to Azure services like Blob Storage, Cosmos DB, and Databricks.

Apache Airflow in ProductionRobert Sanders

We will introduce Airflow, an Apache Project for scheduling and workflow orchestration. We will discuss use cases, applicability and how best to use Airflow, mainly in the context of building data engineering pipelines. We have been running Airflow in production for about 2 years, we will also go over some learnings, best practices and some tools we have built around it. Speakers: Robert Sanders, Shekhar Vemuri

Building Better Data Pipelines using Apache AirflowSid Anand

Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs). It allows users to programmatically author DAGs in Python without needing to bundle many XML files. The UI provides a tree view to see DAG runs over time and Gantt charts to see performance trends. Airflow is useful for ETL pipelines, machine learning workflows, and general job scheduling. It handles task dependencies and failures, monitors performance, and enforces service level agreements. Behind the scenes, the scheduler distributes tasks from the metadata database to Celery workers via RabbitMQ.

Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Kaxil Naik

Apache Airflow allows users to programmatically author, schedule, and monitor workflows or directed acyclic graphs (DAGs) using Python. It is an open-source workflow management platform developed by Airbnb that is used to orchestrate data pipelines. The document provides an overview of Airflow including what it is, its architecture, and concepts like DAGs, tasks, and operators. It also includes instructions on setting up Airflow and running tutorials on basic and dynamic workflows.

Observability: Beyond the Three Pillars with SpringVMware Tanzu

In this presentation, we’ll explore the basics of the three pillars and what Spring has to offer to implement them for logging (SLF4J), metrics (Micrometer), and distributed tracing (Spring Cloud Sleuth, Zipkin/Brave, OpenTelemetry). I’ll also talk about how to take your system to the next level, and what else you can find in Spring and related technologies to look under the hood of your running system (Spring Boot Actuator, Logbook, Eureka, Spring Boot Admin, Swagger, Spring HATEOAS) and what our future plans are.

Prometheus - basicsJuraj Hantak

Airflow tutorials hands_onpko89403

This document provides an overview of Airflow, an open-source workflow management platform for authoring, scheduling and monitoring data pipelines. It describes Airflow's key components including the web server, scheduler, workers and metadata database. It explains how Airflow works by parsing DAGs, instantiating tasks and changing their state as they are scheduled, queued, run and monitored. The document also covers concepts like DAGs, operators, dependencies, concurrency vs parallelism and advanced topics such as subDAGs, hooks, XCOM and branching workflows.

Prometheus (Prometheus London, 2016)Brian Brazil

Brian Brazil is an engineer passionate about reliable software operations. He worked at Google SRE for 7 years and is the founder of Prometheus, an open source time series database designed for monitoring system and service metrics. Prometheus supports metric labeling, unified alerting and graphing, and is efficient, decentralized, reliable, and opinionated in how it encourages good monitoring practices.

Rounds analytics pipelineAviv Laufer

Capital onehadoopintroDoug Chang

This document provides an overview of Capital One's plans to introduce Hadoop and discusses several proof of concepts (POCs) that could be developed. It summarizes the history and practices of using Hadoop at other companies like LinkedIn, Netflix, and Yahoo. It then outlines possible POCs for Hadoop distributions, ETL/analytics frameworks, performance testing, and developing a scaling layer. The goal is to contribute open source code and help with Capital One's transition to using Hadoop in production.

More Related Content

What's hot (20)

Airflow for BeginnersVarya Karpenko

Building an analytics workflow using Apache AirflowYohei Onishi

Airflow presentationIlias Okacha

Apache Airflow overviewNikolayGrishchenkov

Introduction to Apache Airflowmutt_data

Apache Airflow ArchitectureGerard Toonstra

Building a Data Pipeline using Apache Airflow (on AWS / GCP)Yohei Onishi

Airflow 101SaarBergerbest

Apache airflowPavel Alexeev

Airflow Intro-1.pdfBagustTriCahyo1

How I learned to time travel, or, data pipelining and scheduling with AirflowPyData

Airflow Best Practises & Roadmap to Airflow 2.0Kaxil Naik

Apache Airflow IntroductionLiangjun Jiang

Apache Airflow in ProductionRobert Sanders

Building Better Data Pipelines using Apache AirflowSid Anand

Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Kaxil Naik

Observability: Beyond the Three Pillars with SpringVMware Tanzu

Prometheus - basicsJuraj Hantak

Airflow tutorials hands_onpko89403

Prometheus (Prometheus London, 2016)Brian Brazil

Airflow for BeginnersVarya Karpenko

Building an analytics workflow using Apache AirflowYohei Onishi

Airflow presentationIlias Okacha

Apache Airflow overviewNikolayGrishchenkov

Introduction to Apache Airflowmutt_data

Apache Airflow ArchitectureGerard Toonstra

Building a Data Pipeline using Apache Airflow (on AWS / GCP)Yohei Onishi

Airflow 101SaarBergerbest

Apache airflowPavel Alexeev

Airflow Intro-1.pdfBagustTriCahyo1

How I learned to time travel, or, data pipelining and scheduling with AirflowPyData

Airflow Best Practises & Roadmap to Airflow 2.0Kaxil Naik

Apache Airflow IntroductionLiangjun Jiang

Apache Airflow in ProductionRobert Sanders

Building Better Data Pipelines using Apache AirflowSid Anand

Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Kaxil Naik

Observability: Beyond the Three Pillars with SpringVMware Tanzu

Prometheus - basicsJuraj Hantak

Airflow tutorials hands_onpko89403

Prometheus (Prometheus London, 2016)Brian Brazil

Similar to Airflow - a data flow engine (20)

Rounds analytics pipelineAviv Laufer

Capital onehadoopintroDoug Chang

Building data pipelinesJonathan Holloway

This document provides an overview of data pipelines and various technologies that can be used to build them. It begins with a brief history of pipelines and their origins in UNIX. It then discusses common pipeline concepts like decoupling of tasks, encapsulation of processing, and reuse of tasks. Several examples of graphical and programmatic pipeline solutions are presented, including Luigi, Piecepipe, Spring Batch, and workflow engines. Big data pipelines using Hadoop and technologies like Pig and Oozie are also covered. Finally, cloud-based pipeline technologies from AWS like Kinesis, Data Pipeline, Lambda, and EMR are described. Throughout the document, examples are provided to illustrate how different technologies can be used to specify and run data processing pipelines.

Building an MLOps Stack for Companies at Reasonable ScaleMerelda

Spring boot 3gvasya10

This document summarizes a presentation about SpringBoot, a framework for creating Spring applications. It discusses how SpringBoot uses Groovy and removes the need for XML configuration. It provides an overview of Texas NIC where the presenter works and some of their applications. It then discusses why SpringBoot was created, how it can be used with Groovy, and demonstrates some sample applications. It also covers some good aspects of SpringBoot, things that can be confusing, and unexpected behaviors. It concludes with some tips for using SpringBoot with IntelliJ IDEA.

Google Cloud Platform for Data Science teamsBarton Rhodes

The 90-Day Startup with Google AppEngine for JavaDavid Chandler

The document discusses Google App Engine, a platform for developing and hosting web applications on Google's infrastructure. It provides an overview of App Engine and how to get started, discusses some limitations and tradeoffs compared to traditional web hosting, and recommends frameworks and techniques for building scalable applications on App Engine, including Objectify, Guice, and gwt-dispatch. It also notes that while App Engine is still relatively new, it has significant potential for developing scalable applications with minimal upfront costs.

Big data or big dealeduarderwee

The document discusses Hadoop and big data technologies. It begins with an introduction to big data concepts and the various Hadoop components like HDFS, MapReduce, YARN, Hive, Pig and Mahout. It then explains how big data is different from traditional data warehousing through the concept of schema-on-read. Finally, it provides recommendations on tools for working with big data technologies locally and in the cloud, as well as sources of inspiration like sandbox environments, Apache projects and GitHub.

Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...Cloudera, Inc.

Michael Sun presented on CBS Interactive's use of Hadoop for web analytics processing. Some key points: - CBS Interactive processes over 1 billion web logs daily from hundreds of websites on a Hadoop cluster with over 1PB of storage. - They developed an ETL framework called Lumberjack in Python for extracting, transforming, and loading data from web logs into Hadoop and databases. - Lumberjack uses streaming, filters, and schemas to parse, clean, lookup dimensions, and sessionize web logs before loading into a data warehouse for reporting and analytics. - Migrating to Hadoop provided significant benefits including reduced processing time, fault tolerance, scalability, and cost effectiveness compared to their

Yaetos Tech Overviewprevota

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Bhupesh Bansal

Hadoop and Voldemort @ LinkedInHadoop User Group

Zeppelin at TwitterPrasad Wagle

Application Architecture For The CloudSteve Loughran

The document discusses considerations for building application architectures for cloud computing. It outlines some benefits and drawbacks of moving applications to the cloud. It also discusses components needed for cloud applications, such as web UIs, agile scaling, live upgrades, and using services like Amazon S3, SimpleDB, and EC2. It proposes an "Apache Cloud Computing Edition" with technologies like Hadoop and MapReduce and addresses needs like configuration, resource management, and testing in the cloud.

Using and Extending Memory Analyzer into Uncharted WatersVladimir Pavlov

Apache Spark Introduction @ University College LondonVitthal Gogate

Building Automated Data Pipelines with Airflow.pdfabhaykm804

Zeppelin at twitter (sf data science meetup, july 2016)Prasad Wagle

Zeppelin is a data analysis notebook tool that Twitter adopted in late 2015. It is now used widely at Twitter with over 850 users creating 860 notebooks containing 4000 paragraphs. Zeppelin integrates with Twitter's various data platforms and has custom authentication and security features. Twitter engineers have contributed improvements to Zeppelin's stability, operations, and interpreters. Ongoing work focuses on further enhancing security, usability, and integration with Twitter systems.

Discovery & Consumption of Analytics Data @TwitterKamran Munshi

Endofday: A Container Workflow Engine for Scalable, Reproducible ComputationEnis Afgan

Rounds analytics pipelineAviv Laufer

Capital onehadoopintroDoug Chang

Building data pipelinesJonathan Holloway

Building an MLOps Stack for Companies at Reasonable ScaleMerelda

Spring boot 3gvasya10

Google Cloud Platform for Data Science teamsBarton Rhodes

The 90-Day Startup with Google AppEngine for JavaDavid Chandler

Big data or big dealeduarderwee

Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...Cloudera, Inc.

Yaetos Tech Overviewprevota

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Bhupesh Bansal

Hadoop and Voldemort @ LinkedInHadoop User Group

Zeppelin at TwitterPrasad Wagle

Application Architecture For The CloudSteve Loughran

Using and Extending Memory Analyzer into Uncharted WatersVladimir Pavlov

Apache Spark Introduction @ University College LondonVitthal Gogate

Building Automated Data Pipelines with Airflow.pdfabhaykm804

Zeppelin at twitter (sf data science meetup, july 2016)Prasad Wagle

Discovery & Consumption of Analytics Data @TwitterKamran Munshi

Endofday: A Container Workflow Engine for Scalable, Reproducible ComputationEnis Afgan

More from Walter Liu (18)

Generative AI 在手機遊戲發行上的應用介紹.pdfWalter Liu

Infrastructure as code using KubernetesWalter Liu

手機遊戲數位行銷工具淺談Walter Liu

關於第三方追蹤廣告再行銷的那些事兒Walter Liu

Kubernetes WorkshopWalter Liu

This document summarizes Walter's Kubernetes workshop. The workshop will cover why containers and Kubernetes are useful, the basic Kubernetes components like pods and deployments, and how to complete some hands-on labs. It will demonstrate how to create pods and deployments, expose services, and configure additional Kubernetes objects like statefulsets, secrets and configmaps. Troubleshooting techniques like describing and logging pods will also be discussed.

Using Kubernetes to deploy Django in GCPWalter Liu

Walter discusses using Kubernetes on Google Cloud Platform to deploy a Django application. He describes how Kubernetes provides infrastructure as code to automate and scale the application. Key Kubernetes components used include pods, deployments, statefulsets, secrets and configmaps. Services are used for internal and external load balancing, with an ingress controller providing a global load balancer. The document also touches on cluster creation steps, load balancing options, and templating with Python Jinja.

Game DDOS PreventionWalter Liu

This document discusses solutions for preventing distributed denial-of-service (DDoS) attacks on game servers at different levels including DNS, network, and application levels. It recommends purchasing anti-DDoS services, using content delivery networks, web application firewalls, blacklisting abnormal IP addresses, and implementing packet marking and filtering techniques. The document also provides references to several commercial anti-DDoS service providers and their pricing.

Super Fast Gevent IntroductionWalter Liu

This document discusses gevent, a library for asynchronous programming that uses greenlets and monkey patching. It provides a simple example of gevent, describes how gevent works by monkey patching functions like sockets and threading to be cooperative instead of preemptive. It also discusses how gevent provides a fast event loop and WSGI server using libevent under the hood, and notes that monkey patching makes gevent incompatible with some third party libraries.

HTTP/2 to web devWalter Liu

HTTP/2 IntroductionWalter Liu

HTTP/2 aims to address issues with HTTP/1.x such as head-of-line blocking and wasted bandwidth through duplicate requests. It uses a binary format for multiplexing requests, server push, header compression, stream prioritization and flow control. Major browsers now support HTTP/2 over TLS, though server implementations are still in development. While preserving the HTTP/1.1 API, HTTP/2 provides advantages like cheaper requests and more efficient use of network resources and server capacity.

Consul - service discovery and othersWalter Liu

Docker introductionWalter Liu

This document provides an introduction and overview of Docker. It discusses what Docker is and how it differs from virtual machines by using layers, containers, and shared resources. Docker aims to solve the problem of inconsistent environments between development, testing, and production by allowing applications and their dependencies to be packaged into standardized units called containers that can run on any Docker host. The document provides instructions on installing Docker and using basic commands like running, publishing ports, and mounting volumes for containers. It also covers some tips, tricks, pitfalls, and references for further reading.

WRS GIT Branching Model － draftWalter Liu

Salty OPS – Saltstack IntroductionWalter Liu

This document provides an introduction and overview of Saltstack, including: - The basic components of Saltstack including the salt-master, salt-minion, and salt-syndic - Common Saltstack commands like salt, salt-key, and salt-call - How to configure Saltstack including basic configuration files and directories - Examples of using Saltstack to install packages, copy files, run commands, and update systems to the newest state - Additional Saltstack features like grains, pillars, batch operations, testing commands, and troubleshooting - How to set up GitFS and collaborate using Saltstack Formulas and States repositories

Django deployment and rpm+yumWalter Liu

This document discusses deploying Django applications using Apache, WSGI, RPM, and YUM. It begins with an overview of using Apache and WSGI to run Django applications stability. It then covers deployment notes for Django like setting DEBUG to False. Next, it discusses using RPM to package and deploy Django applications with features like file management and installation scripting. RPMs can then be hosted and updated using YUM for remote deployment and version management across multiple servers.

Game Localization in PythonWalter Liu

This document discusses using Python for game localization. It describes failures in localizing text, XML, and script files across different files. It then outlines a localization workflow using Python tools to extract text differences, merge translations, and verify outputs. Key benefits of Python for localization are its Unicode library, fast string processing, and ease of development. The workflow helped reduce bugs from over 2000 to 200 and localization time from 9 months to 3 months for one expansion.

Service production from d3 pitfall viewpointWalter Liu

This document discusses lessons learned from the Diablo 3 launch, including mistakes made around quantity estimation, scalability, and customer service. It emphasizes the importance of planning for high traffic, having horizontally scalable architectures, and developing fast and responsive processes for incident management and communication. The document also provides examples of how Trend Micro's web reputation and parental control services can help protect users from online threats.

Celery in the DjangoWalter Liu

This document discusses Celery, an asynchronous task queue/job queue based on distributed message passing. It describes Celery as a distributed, asynchronous task queue. It provides examples of using Celery for tasks like sending thousands of emails or handling computationally heavy queries. The document outlines some benefits of Celery like making tasks distributed and allowing configuration of queues, routing, and task utilities. It also discusses some pitfalls encountered with Celery and complaints about it.

Generative AI 在手機遊戲發行上的應用介紹.pdfWalter Liu

Infrastructure as code using KubernetesWalter Liu

手機遊戲數位行銷工具淺談Walter Liu

關於第三方追蹤廣告再行銷的那些事兒Walter Liu

Kubernetes WorkshopWalter Liu

Using Kubernetes to deploy Django in GCPWalter Liu

Game DDOS PreventionWalter Liu

Super Fast Gevent IntroductionWalter Liu

HTTP/2 to web devWalter Liu

HTTP/2 IntroductionWalter Liu

Consul - service discovery and othersWalter Liu

Docker introductionWalter Liu

WRS GIT Branching Model － draftWalter Liu

Salty OPS – Saltstack IntroductionWalter Liu

Django deployment and rpm+yumWalter Liu

Game Localization in PythonWalter Liu

Service production from d3 pitfall viewpointWalter Liu

Celery in the DjangoWalter Liu

Recently uploaded (20)

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

tecnologias de las primeras civilizaciones.pdffjgm517

TrsLabs - Fintech Product & Business ConsultingTrs Labs

Hybrid Growth Mandate Model with TrsLabs Strategic Investments, Inorganic Growth, Business Model Pivoting are critical activities that business don't do/change everyday. In cases like this, it may benefit your business to choose a temporary external consultant. An unbiased plan driven by clearcut deliverables, market dynamics and without the influence of your internal office equations empower business leaders to make right choices. Getting things done within a budget within a timeframe is key to Growing Business - No matter whether you are a start-up or a big company Talk to us & Unlock the competitive advantage

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

Talk at the final event of Data Fusion Dynamics: A Collaborative UK-Saudi Initiative in Cybersecurity and Artificial Intelligence funded by the British Council UK-Saudi Challenge Fund 2024, Cardiff Metropolitan University, 29th April 2025 https://alandix.com/academic/talks/CMet2025-AI-Changes-Everything/ Is AI just another technology, or does it fundamentally change the way we live and think? Every technology has a direct impact with micro-ethical consequences, some good, some bad. However more profound are the ways in which some technologies reshape the very fabric of society with macro-ethical impacts. The invention of the stirrup revolutionised mounted combat, but as a side effect gave rise to the feudal system, which still shapes politics today. The internal combustion engine offers personal freedom and creates pollution, but has also transformed the nature of urban planning and international trade. When we look at AI the micro-ethical issues, such as bias, are most obvious, but the macro-ethical challenges may be greater. At a micro-ethical level AI has the potential to deepen social, ethnic and gender bias, issues I have warned about since the early 1990s! It is also being used increasingly on the battlefield. However, it also offers amazing opportunities in health and educations, as the recent Nobel prizes for the developers of AlphaFold illustrate. More radically, the need to encode ethics acts as a mirror to surface essential ethical problems and conflicts. At the macro-ethical level, by the early 2000s digital technology had already begun to undermine sovereignty (e.g. gambling), market economics (through network effects and emergent monopolies), and the very meaning of money. Modern AI is the child of big data, big computation and ultimately big business, intensifying the inherent tendency of digital technology to concentrate power. AI is already unravelling the fundamentals of the social, political and economic world around us, but this is a world that needs radical reimagining to overcome the global environmental and human challenges that confront us. Our challenge is whether to let the threads fall as they may, or to use them to weave a better future.

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including: -Artificial Intelligence Market Overview -Strategies for AI Adoption in 2025 -Anticipated drivers of AI adoption and transformative technologies -Benefits of AI and Big data for your business -Tips on how to prepare your business for innovation -AI and data privacy: Strategies for securing data privacy in AI models, etc. Download your free copy nowand implement the key findings to improve your business.

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/ HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client. Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience. In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including - Accessing the console - Locating and interpreting log files - Accessing the data folder within the browser’s cache (using OPFS) - Understand the difference between single- and multi-user scenarios - Utilizing Client Clocking

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, presentation slides, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Splunk Security Update | Public Sector Summit Germany 2025Splunk

How Can I use the AI Hype in my Business Context?Daniel Lehner

𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨? Everyone’s talking about AI but is anyone really using it to create real value? Most companies want to leverage AI. Few know 𝗵𝗼𝘄. ✅ What exactly should you ask to find real AI opportunities? ✅ Which AI techniques actually fit your business? ✅ Is your data even ready for AI? If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency. This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data. Attendees will learn: - Consumer awareness around data brokers and what consumers are doing to limit data collection - How businesses assess third-party vendors and their consent management operations - Where business preparedness needs improvement - What these trends mean for the future of privacy governance and public trust This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.

How analogue intelligence complements AIPaul Rowe

Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results. News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?