So we're running Apache ZooKeeper. Now What? By Camille Fournier

Oct 30, 2013Download as PPTX, PDF27 likes16,512 views

The ZooKeeper framework was originally built at Yahoo! to make it easy for the company’s applications to access configuration information in a robust and easy-to-understand way, but it has since grown to offer a lot of features that help coordinate work across distributed clusters. Apache Zookeeper became a de-facto standard for coordination service and used by Storm, Hadoop, HBase, ElasticSearch and other distributed computing frameworks.

So we’re running

ZooKeeper. Now What?
Camille Fournier, Rent the Runway
@skamille

• Outages
• Coordination
• Operational Complexity

Common Challenges

• Consistency guarantees

Common Deficiency

Storm uses Zookeeper for coordinating the cluster.
Zookeeper is not used for message passing, so the
load Storm places on Zookeeper is quite low

ZooKeeper in Storm

A centralized service for maintaining
configuration information, naming,
providing distributed synchronization,
and providing group services
•
•
•
•

Distributed, Consistent Data Store
Highly Available
High performance
Strictly ordered access

ZooKeeper

• Tolerates the loss of a minority ((n/2) – 1) of ensemble
members and still function

Highly Available

• All data is stored in memory
• Performance measured around 50,000

operations/second
• Particularly fast for read performance, built for readdominant workloads

High Performance

• Atomic Writes
• In the order you sent them
• Changes always seen in the order they occurred

• Reliable, no writes acked will be dropped

Strictly Ordered Access

leader
cli
cli

follower

follower

cli
cli

cli
cli

Basics: Cluster
Interactions

leader
cli
cli

follower

cli
cli

cli
cli

Basics: Cluster
Interactions

cli

/a/b/myNode

/a/b
/a/b/d

/a
/a/c

Data Structure

/a/c/e000001

• Nodes can contain data, have children, or both
• Ephemeral nodes are associated with the session that
created them
• They cannot have children, and disappear when that
session ends
• Sequential nodes have an ever-increasing number
attached to them

Basics: Data Structure

client

leader

getData “/foo” true
client

client

Watches

return “mydata”

follower

follower

client

leader

client

follower

setData “/foo” “bar”
client

Watches

follower

client

leader

NOTIFICATION
client

follower

client

follower

Watches

client

leader

getData “/foo” true
client

client

Watches

return “bar”

follower

follower

• Set against data or path changes
• Ordered with respect to other events, other watches, and
asynchronous replies.
• A client will see a watch event for a node it is watching
before seeing the new data that corresponds to that node.
• The order of watch events corresponds to the order of the
updates as seen by the ZooKeeper service
• One time notifications; must be reset, changes can be
missed between notification and reset of the watch

Basics: Watches

• create
• delete
• setData

Basics: Creation API

• exists
• getData
• getChildren

Basics: Get/Watch API

• multi * new in 3.4
• sync

Basics: API

Service Management
Distributed Locking

Common Uses

• In Storm, ZooKeeper is the source of
communication between Nimbus and
Supervisors
• Nimbus finds Supervisors via ZooKeeper

Coordination

Find servers doing job “Products”
Encode as path in ZooKeeper:
/servers/products
Servers register as ephemeral nodes under this path
with details about location, other connection info

Discovery (Naming)

Read config from nodes
Watch nodes for config changes

Configuration

Shared Locks
Barriers and Latches
Leader Election
Two-Phase Commit

Locking

• Curator (Java)
• Kazoo (Python)
• Twitter Commons for Discovery

Recommended Clients

ZooKeeper Owns Your Availability

(maybe)

Be Aware

•
•
•
•
•
•

Thank you to @zaa for the format of the slide on watches
Tweet me! @skamille
Email me! camille@apache.org
Kazoo: http://kazoo.readthedocs.org/en/latest/
Curator: http://curator.incubator.apache.org/
Twitter commons: http://twitter.github.io/commons/

Credits and Contact

Big Data with Hadoop & Spark Training: http://bit.ly/2kvXlPd This CloudxLab Introduction to Apache ZooKeeper tutorial helps you to understand ZooKeeper in detail. Below are the topics covered in this tutorial: 1) Data Model 2) Znode Types 3) Persistent Znode 4) Sequential Znode 5) Architecture 6) Election & Majority Demo 7) Why Do We Need Majority? 8) Guarantees - Sequential consistency, Atomicity, Single system image, Durability, Timeliness 9) ZooKeeper APIs 10) Watches & Triggers 11) ACLs - Access Control Lists 12) Usecases 13) When Not to Use ZooKeeper

Introduction to apache zoo keeper Omid Vahdaty

Zookeeper Introducejhao niu

ZooKeeper is a distributed coordination service that allows distributed applications to synchronize data and configuration information. It uses a data model of directories and files, called znodes, that can contain small amounts of structured data. ZooKeeper maintains data consistency through a leader election process and quorum-based consensus algorithm called Paxos. It provides applications with synchronization primitives and configuration maintenance in a highly-available and reliable way.

DDRPayal Dwivedi

DDR memory is a type of RAM that allows for increased performance over single data rate memory by facilitating two data transactions per clock cycle without doubling the clock speed. It consists of over 130 signals and uses mode and extended mode registers to control operations. DDR memory comes in SRAM and DRAM varieties, with DRAM being more common due to its lower power consumption and use in main memory, though it requires constant refreshing to prevent data loss.

Interleaved memoryashishgy

Interleaved memory is a design that spreads memory addresses across multiple memory banks to compensate for the relatively slow speed of DRAM. It increases bandwidth and improves performance by allowing different modules to be accessed independently and in parallel by different processing units like a CPU and hard disk. There are two address formats for interleaved memory: low order interleaving which spreads addresses across banks, and high order interleaving which uses high order bits as the module address.

Bus structure in Computer Organization.pdfmvpk14486

Buses connect components in a computer system and allow for the transfer of data and control signals. There are three main types of buses: the address bus carries memory and I/O addresses, the data bus carries data and instructions, and the control bus carries signals that determine read, write, I/O, and synchronization operations. A system bus combines the functions of these three buses to connect major computer components like the processor, memory, and I/O devices.

System verilog assertionsHARINATH REDDY

This document discusses SystemVerilog assertions (SVA). It introduces SVA and explains that assertions are used to document design functionality, check design intent is met, and determine if verification tested the design. Assertions can be specified by the design or verification engineer. The document outlines the key building blocks of SVA like sequences, properties, and assertions. It provides examples of different types of assertions and how they are used. Key concepts discussed include implication, timing windows, edge detection, and repetition operators.

File replicationDhaval Chodavadiya

Introduction to Apache ZooKeeperSaurav Haloi

Apache ZooKeeperScott Leberknight

Introduction to ZooKeeper - TriHUG May 22, 2012mumrah

Dynamic Reconfiguration of Apache ZooKeeperDataWorks Summit

ZooKeeper allows for dynamic reconfiguration of servers in its ensemble. Manual reconfiguration is problematic as it requires changing the configuration, restarting servers, and can result in data loss. The presented solution allows ZooKeeper to reconfigure itself automatically through a speculative reconfiguration approach. It commits the reconfiguration once quorums of both the old and new ensembles acknowledge it, and gossips the new configuration to ensure all servers sync before activation. This allows reconfigurations to complete without failures in a transparent manner to clients.

Winter is coming? Not if ZooKeeper is there!Joydeep Banik Roy

[Download the slide to get the entire talk in the form of presentation note embedded in the ppt] Apache ZooKeeper is the chosen leader in distributed coordination. In this talk, I have explored the atomic elements of Apache ZooKeeper, how it fits everything together and some of its popular use cases. For ZooKeeper simplicity is the key and as a consumer of the API, our imagination enables us to push the limits of the ZooKeeper world.

Meetup on Apache ZookeeperAnshul Patel

Zookeeper is a distributed coordination service that provides naming, configuration, synchronization, and group services. It allows distributed processes to coordinate with each other through a shared hierarchical namespace of data registers called znodes. Zookeeper follows a leader-elected consensus protocol to guarantee atomic broadcast of state updates from the leader to followers. It uses a hierarchical namespace of znodes similar to a file system to store configuration data and other application-defined metadata. Zookeeper provides services like leader election, group membership, synchronization, and configuration management that are essential for distributed systems.

Zookeeper In Actionjuvenxu

Centralized Application Configuration with Spring and Apache ZookeeperRyan Gardner

From talk given at Spring One 2gx Dallas, 2014 Application configuration is an evolution. It starts as a hard-coded strings in your application and hopefully progresses to something external, such as a file or system property that can be changed without deployment. But what happens when other enterprise concerns enter the mix, such as audit requirements or access control around who can make changes? How do you maintain the consistency of values across too many application servers to manage at one time from a terminal window? The next step in the application configuration evolution is centralized configuration that can be accessed by your applications as they move through your various environments on their way to production. Such a service transfers the ownership of configuration from the last developer who touched the code to a well-versed application owner who is responsible for the configuration of the application across all environments. At Dealer.com, we have created one such solution that relies on Apache ZooKeeper to handle the storage and coordination of the configuration data and Spring to handle to the retrieval, creation and registration of configured objects in each application. The end result is a transparent framework that provides the same configured objects that could have been created using a Spring configuration, configuration file and property value wiring. This talk will cover both the why and how of our solution, with a focus on how we leveraged the powerful attributes of both Apache ZooKeeper and Spring to rid our application of local configuration files and provide a consistent mechanism for application configuration in our enterprise.

Apache Zookeeper 分布式服务框架Cabin WJ

Distributed Applications with Apache ZookeeperAlex Ehrnschwender

Apache Zookeeper Explained: Tutorial, Use Cases and Zookeeper Java API ExamplesBinu George

Introduction to Kafka and ZookeeperRahul Jain

OpenIO A3techlive in London - March 2016OpenIO Object Storage

Oscon 2015 - OpenIO: Enabling the petabyte-sized mailboxesOpenIO Object Storage

IT Press Tour #19 Slides OpenIO June 2016OpenIO Object Storage

This document discusses OpenIO, an open source software-defined storage platform. It can transform commodity x86 servers into a large storage and computing pool. OpenIO offers elastic, scalable object storage for up to 1000+ petabytes of data. It uses a grid architecture with no single point of failure and real-time load balancing. OpenIO also enables running applications directly on the storage infrastructure through its "Grid for Apps" feature. The document outlines how OpenIO provides ease of use, supports storage tiering, and can integrate with services like Backblaze B2 to offer hybrid cloud storage solutions.

IT Press Tour #17 - OpenIO & TechnologyOpenIO Object Storage

OpenIO ServerLess StorageOpenIO Object Storage

This document discusses serverless storage and scale-out object storage appliances. It introduces the OpenIO SLS, a scale-out object storage appliance that uses a distributed storage software (SDS) platform. The SLS provides high performance, efficiency, and scalability through its architecture of many small ARM-based nano-nodes within a single 4U chassis, with no single point of failure and linear scalability by adding disks one by one. It aims to make scale-out object storage easy to deploy, manage and use for organizations of all kinds.

Hybrid Cloud Data Storage with OpenIO and Backblaze B2Backblaze

ZooKeeper FuturesCloudera, Inc.

Henry Robinson gave a presentation on upcoming features for ZooKeeper. He discussed observers, which allow non-voting servers to scale client connections without impacting performance. He also covered dynamic ensembles, which would allow changing the ZooKeeper cluster membership without downtime. Finally, he announced that Cloudera's Distribution for Hadoop will include ZooKeeper packages to integrate it more fully.

Taming Pythons with ZooKeeperJyrki Pulliainen

Concurrency is hard. Consistency in distributed systems is hard. And then the whole thing should be highly-available and error resilient. Fear not, there are good news: There exists an awesome tool called ZooKeeper to help you with this. There even exists a plethora of Python libraries for it, but how to know what to use and how? This talk will walk you through ZooKeeper and how to use it with Python. We’ll be focusing on what I think is the most prominient ZooKeeper library out there for Python: Kazoo. You’ll see how to do things in ZooKeeper and how to implement them using Kazoo. We’ll also peek in to the recipes Kazoo offers, and if we have enough time, touch a real life application we’ve used Kazoo and ZooKeeper to build at Spotify.

Zookeeper Tutorial for beginnersjeetendra mandal

Apache ZooKeeper is an open-source distributed coordination service that helps manage large sets of hosts. It implements coordination protocols to provide a consistent view of shared state across distributed applications or servers. ZooKeeper uses a hierarchical namespacing system called znodes to store configuration data and other information. It ensures highly reliable distributed coordination through features like leader election, group membership, and notifications.

Automated Cluster Management and Recovery for Large Scale Multi-Tenant Sea...Lucidworks

The document describes Bloomreach's architecture for managing a large-scale SolrCloud cluster across multiple data centers. It discusses the challenges of serving real-time queries at scale, managing configurations and rankings across tenants and data centers, and providing high availability and recovery capabilities. The key components of Bloomreach's architecture include a cluster management suite, replication and ranking configuration APIs, and deployment/recovery services for adding or replacing data centers and collections.

More Related Content

Viewers also liked (20)

Introduction to Apache ZooKeeperSaurav Haloi

Apache ZooKeeperScott Leberknight

Introduction to ZooKeeper - TriHUG May 22, 2012mumrah

Dynamic Reconfiguration of Apache ZooKeeperDataWorks Summit

Winter is coming? Not if ZooKeeper is there!Joydeep Banik Roy

Meetup on Apache ZookeeperAnshul Patel

Zookeeper In Actionjuvenxu

Centralized Application Configuration with Spring and Apache ZookeeperRyan Gardner

Apache Zookeeper 分布式服务框架Cabin WJ

Distributed Applications with Apache ZookeeperAlex Ehrnschwender

Apache Zookeeper Explained: Tutorial, Use Cases and Zookeeper Java API ExamplesBinu George

Introduction to Kafka and ZookeeperRahul Jain

OpenIO A3techlive in London - March 2016OpenIO Object Storage

Oscon 2015 - OpenIO: Enabling the petabyte-sized mailboxesOpenIO Object Storage

IT Press Tour #19 Slides OpenIO June 2016OpenIO Object Storage

IT Press Tour #17 - OpenIO & TechnologyOpenIO Object Storage

OpenIO ServerLess StorageOpenIO Object Storage

Hybrid Cloud Data Storage with OpenIO and Backblaze B2Backblaze

ZooKeeper FuturesCloudera, Inc.

Taming Pythons with ZooKeeperJyrki Pulliainen

Introduction to Apache ZooKeeperSaurav Haloi

Apache ZooKeeperScott Leberknight

Introduction to ZooKeeper - TriHUG May 22, 2012mumrah

Dynamic Reconfiguration of Apache ZooKeeperDataWorks Summit

Winter is coming? Not if ZooKeeper is there!Joydeep Banik Roy

Meetup on Apache ZookeeperAnshul Patel

Zookeeper In Actionjuvenxu

Centralized Application Configuration with Spring and Apache ZookeeperRyan Gardner

Apache Zookeeper 分布式服务框架Cabin WJ

Distributed Applications with Apache ZookeeperAlex Ehrnschwender

Apache Zookeeper Explained: Tutorial, Use Cases and Zookeeper Java API ExamplesBinu George

Introduction to Kafka and ZookeeperRahul Jain

OpenIO A3techlive in London - March 2016OpenIO Object Storage

Oscon 2015 - OpenIO: Enabling the petabyte-sized mailboxesOpenIO Object Storage

IT Press Tour #19 Slides OpenIO June 2016OpenIO Object Storage

IT Press Tour #17 - OpenIO & TechnologyOpenIO Object Storage

OpenIO ServerLess StorageOpenIO Object Storage

Hybrid Cloud Data Storage with OpenIO and Backblaze B2Backblaze

ZooKeeper FuturesCloudera, Inc.

Taming Pythons with ZooKeeperJyrki Pulliainen

Similar to So we're running Apache ZooKeeper. Now What? By Camille Fournier (20)

Zookeeper Tutorial for beginnersjeetendra mandal

Automated Cluster Management and Recovery for Large Scale Multi-Tenant Sea...Lucidworks

[Hic2011] using hadoop lucene-solr-for-large-scale-search by systexJames Chen

Deploying and managing Solr at scaleAnshum Gupta

This document discusses deploying and managing Apache Solr at scale. It introduces the Solr Scale Toolkit, an open source tool for deploying and managing SolrCloud clusters in cloud environments like AWS. The toolkit uses Python tools like Fabric to provision machines, deploy ZooKeeper ensembles, configure and start SolrCloud clusters. It also supports benchmark testing and system monitoring. The document demonstrates using the toolkit and discusses lessons learned around indexing and query performance at scale.

InfluxEnterprise Architecture Patterns by Tim Hall & Sam DillardInfluxData

1. The document provides an overview of InfluxEnterprise, including its core open source functionality, high availability features, scalability, fine-grained authorization, support options, and on-premise or cloud deployment options. 2. It discusses signs that an organization may be ready for InfluxEnterprise, such as high CPU usage, issues with single node deployments, and needing improved data durability or throughput. 3. The document covers InfluxEnterprise cluster architecture including meta nodes, data nodes, replication patterns, ingestion and query rates for different replication configurations, and examples for mothership, durable data ingest, and integrating with ElasticSearch deployments.

InfluxEnterprise Architectural Patterns by Dean Sheehan, Senior Director, Pre...InfluxData

When it Absolutely, Positively, Has to be There: Reliability Guarantees in Ka...confluent

In the financial industry, losing data is unacceptable. Financial firms are adopting Kafka for their critical applications. Kafka provides the low latency, high throughput, high availability, and scale that these applications require. But can it also provide complete reliability? As a system architect, when asked “Can you guarantee that we will always get every transaction,” you want to be able to say “Yes” with total confidence. In this session, we will go over everything that happens to a message – from producer to consumer, and pinpoint all the places where data can be lost – if you are not careful. You will learn how developers and operation teams can work together to build a bulletproof data pipeline with Kafka. And if you need proof that you built a reliable system – we’ll show you how you can build the system to prove this too.

OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander DibboOpenNebula Project

The Science and Technology Facilities Council is a UK Research Council which funds research and provides large facilities to the UK Scientific Community. This includes running a Tier 1 site for the LHC computing project, the JASMIN Super Data Cluster and a number of other HPC and HTC facilities. The Scientific Computing Department at the Rutherford Appleton Laboratory has been developing a cloud for use across both sites of the Department and in the wider scientific community. This is an OpenNebula backed by Ceph block storage. I will give a brief background of the project, describe our set up, some use cases and the work we have done around OpenNebula (including a simplified web front-end and a number of hooks to provide us with traceability). I will also discuss how we are creating an elastic boundary between our HTC batch farm and cloud. Author Biography I am a Systems Administrator in the Scientific Computing Department of the UK’s Science and Technology Facilities Council. I work as part of the cloud team and I also work on a number of Grid services including our HTC batch farm for the LHC computing project. Prior to my position here I worked in IT at a SMB focusing on Storage and Virtualisation, in particular Hyper-V and VMWare.

Fault tolerant presentationskadyan1

Presentation was delivered in a fault tolerance class which talk about the achieving fault tolerance in databases by making use of the replication.Different commercial databases were studied and looked into the approaches they took for replication.Then based on the study an architecture was suggested for military database design using an asynchronous approach and making use of the cluster patterns.

Benchmarking Solr PerformanceLucidworks

The document discusses benchmarking the performance of Apache Solr. It describes testing the indexing performance of SolrCloud clusters of varying sizes. The results show that indexing performance scales nearly linearly as nodes are added. It also discusses using the Solr Scale Toolkit, which is a set of tools for deploying, managing, and benchmarking SolrCloud clusters. Future work mentioned includes benchmarking mixed workloads and integrating chaos monkey tests.

使用ZooKeeper打造軟體式負載平衡Lawrence Huang

ZooKeeper is a coordination service for distributed systems that allows nodes to perform tasks correctly. It provides features like atomic operations, order guarantees, and high availability. ZooKeeper uses the Zab protocol to elect a leader and achieve consensus. It stores data in a hierarchical structure of znodes that can be persistent or ephemeral. Software load balancers can use ZooKeeper to store cluster configuration and direct traffic. The Apache Curator library provides a higher-level API for common ZooKeeper patterns like leader election and locks.

Mtc learnings from isv & enterprise (dated - Dec -2014)Govind Kanshi

Mtc learnings from isv & enterprise interactionGovind Kanshi

Exhibitor IntroductionJordan Zimmerman

Exhibitor is a tool developed by Netflix to manage ZooKeeper ensembles. It provides features like instance monitoring to restart ZooKeeper servers if they crash, log cleanup for older ZooKeeper versions, backup and selective restore of ZooKeeper data, rolling changes to ZooKeeper ensembles while maintaining quorum, and a visualizer to view the ZooKeeper namespace. Exhibitor configurations are shared across an ensemble from a centralized source like S3 or a file system. It also integrates with tools like Curator to update client configurations when the ensemble changes.

Mini-Training: Netflix Simian ArmyBetclic Everest Group Tech Team

Introduction to SolrCloudVarun Thacker

Apache Kafka IntroductionAmita Mirajkar

Hadoop Ecosystem and Low Latency Streaming ArchitectureInSemble

DCSF19 Container Security: Theory & Practice at NetflixDocker, Inc.

Michael Wardrop, Netflix Usage of containers has undergone rapid growth at Netflix and it is still accelerating. Our container story started organically with developers downloading Docker and using it to improve their developer experience. The first production workloads were simple batch jobs, pioneering micro-services followed, then status as a first class platform running critical workloads. As the types of workloads changed and their importance increased, the security of our container ecosystem needed to evolve and adapt. This session will cover some security theory, architecture, along with practical considerations, and lessons we learnt along the way.

Tech Talks_25.04.15_Session 3_Tibor Sulyan_Distributed coordination with zook...EPAM_Systems_Bulgaria

This document provides an overview of distributed coordination using ZooKeeper and Curator. It discusses ZooKeeper features like its data model, API, guarantees such as consistency and availability. Common recipes for coordination like shared data, service discovery, locking, and leader election are presented. The document also covers performance considerations and advanced ZooKeeper topics like using observers and hierarchical quorums. Finally, Curator is introduced as a higher-level client API for ZooKeeper that implements many recipes and handles complexity.

Zookeeper Tutorial for beginnersjeetendra mandal

Automated Cluster Management and Recovery for Large Scale Multi-Tenant Sea...Lucidworks

[Hic2011] using hadoop lucene-solr-for-large-scale-search by systexJames Chen

Deploying and managing Solr at scaleAnshum Gupta

InfluxEnterprise Architecture Patterns by Tim Hall & Sam DillardInfluxData

InfluxEnterprise Architectural Patterns by Dean Sheehan, Senior Director, Pre...InfluxData

When it Absolutely, Positively, Has to be There: Reliability Guarantees in Ka...confluent

OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander DibboOpenNebula Project

Fault tolerant presentationskadyan1

Benchmarking Solr PerformanceLucidworks

使用ZooKeeper打造軟體式負載平衡Lawrence Huang

Mtc learnings from isv & enterprise (dated - Dec -2014)Govind Kanshi

Mtc learnings from isv & enterprise interactionGovind Kanshi

Exhibitor IntroductionJordan Zimmerman

Mini-Training: Netflix Simian ArmyBetclic Everest Group Tech Team

Introduction to SolrCloudVarun Thacker

Apache Kafka IntroductionAmita Mirajkar

Hadoop Ecosystem and Low Latency Streaming ArchitectureInSemble

DCSF19 Container Security: Theory & Practice at NetflixDocker, Inc.

Tech Talks_25.04.15_Session 3_Tibor Sulyan_Distributed coordination with zook...EPAM_Systems_Bulgaria

More from Hakka Labs (20)

Always Valid Inference (Ramesh Johari, Stanford)Hakka Labs

DataEngConf SF16 - High cardinality time series searchHakka Labs

The document discusses high cardinality time series search and scaling to large datasets. It describes the author's company which collects and analyzes machine data at terabytes per day. General purpose search systems are good for moderate scale but have limitations for high cardinality data with large retention. The author's company built Rocana Search to optimize for their time-oriented event data, with features like parallel ingestion and querying, dynamic partitioning, and keeping all data online without wasted resources. It can handle billions of events per day with low latency and utilizes modern hardware through full distribution.

DataEngConf SF16 - Data Asserts: Defensive Data ScienceHakka Labs

1) Complex data pipelines can introduce bugs that compound as dependencies increase. Engineers manage complexity through encapsulation, clear APIs, and integration tests. 2) Data scientists require semantic correctness but making assumptions introduces risks. Sanity checks on fields like verifying formats and constraints help identify potential errors. 3) Defensive data science through data asserts maintains quality by clearly defining trust boundaries and assumptions. Checks should match expectations and be revisited regularly as upstream changes can impact pipelines.

DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast DataHakka Labs

This document discusses Apache Kudu, an open source columnar storage system for analytics workloads on Hadoop. Kudu is designed to enable both fast analytics queries as well as real-time updates on fast changing data. It aims to fill gaps in the current Hadoop storage landscape by supporting simultaneous high throughput scans, low latency reads/writes, and ACID transactions. An example use case described is for real-time fraud detection on streaming financial data.

DataEngConf SF16 - Recommendations at InstacartHakka Labs

The document discusses recommendations at Instacart, an online grocery delivery service. It summarizes: 1) Instacart aims to provide personalized top N recommendations to promote discovery across its large and dynamic catalog of grocery products from various stores. 2) It also provides replacement product recommendations to help shoppers find substitutes when items are out of stock, drawing on models trained on product attributes and user purchase histories. 3) Additional recommendation types discussed include "frequently bought together" items and post-checkout suggestions to accommodate last-minute additions. The document outlines Instacart's recommendation system architecture and evaluation approach.

DataEngConf SF16 - Running simulations at scaleHakka Labs

This document summarizes Lyft's use of simulations to optimize key services like pricing, dispatching, and Lyft Line matching. It discusses how simulations allow Lyft to test many variations of models quickly under different conditions without disrupting live operations. The simulations replay historical ride and driver location data. Distributed workers on EC2 run the simulations asynchronously and in parallel. Challenges addressed include avoiding race conditions between workers, speeding up environment setup using conda, and handling failures resiliently.

DataEngConf SF16 - Deriving Meaning from Wearable Sensor DataHakka Labs

This document discusses deriving meaning from wearable sensor data to unlock its potential for health insights and behavior change. It describes the growth of wearable devices and sensor data collection over time. Infrastructure is proposed to aggregate sensor and context data from wearables and mobile phones at scale. Methods are outlined for processing, analyzing and exploring this data to develop models for activity and sleep detection, as well as personalized insights and behavioral recommendations. The goal is to leverage this wealth of health data for chronic disease prevention and management through behavioral nudges.

DataEngConf SF16 - Collecting and Moving Data at Scale Hakka Labs

This document summarizes Sada Furuhashi's presentation on Fluentd, an open source data collector. Fluentd provides a centralized way to collect, filter, and output log data from various sources like applications, servers, and databases. It addresses challenges with typical log collection architectures that have high latency, complex parsing, and a combination explosion of connections. Fluentd uses a plugin-based architecture with input, filter, and output components to flexibly collect, transform, and deliver log data at scale to targets like files, databases and visualization tools. Many large companies like Microsoft, Atlassian and Amazon use Fluentd for log collection and analytics in production environments.

DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQHakka Labs

The document discusses reasons for reimplementing messaging queue functionality instead of using an existing solution like RabbitMQ or Kafka. It notes that reimplementing provides full understanding and control, allowing for quick fixes and easy addition of features without needing a daemon. However, it also acknowledges that reimplementing takes a long time to reach stability. The document then describes the specific requirements for the reimplemented messaging queue and provides a brief overview of its design and performance.

DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...Hakka Labs

This document discusses the challenges of data engineering and introduces the Lambda Architecture as a solution. The Lambda Architecture unifies real-time and batch processing through five key concepts: parallel ingestion, a batch layer, a serving layer, a speed layer, and a query unifier. The document shares lessons learned from migrating an existing analytics platform to the Lambda Architecture at Keen IO, including challenges around cross-provider networking, tool versioning, and cultural debt.

DataEngConf SF16 - Three lessons learned from building a production machine l...Hakka Labs

This document discusses three lessons learned from building machine learning systems at Stripe. 1. Don't treat models as black boxes. Early on, Stripe focused only on training with more data and features without understanding algorithms, results, or deeper reasons behind results. This led to overfitting. Introspecting models using "score reasons" helped debug issues. 2. Have a plan for counterfactual evaluation before production. Stripe's validation results did not predict poor production performance because the environment changed. Counterfactual evaluation using A/B testing with probabilistic reversals of block decisions allows estimating true precision and recall. 3. Invest in production monitoring of models. Monitoring inputs, outputs, action rates, score

DataEngConf SF16 - Scalable and Reliable Logging at PinterestHakka Labs

DataEngConf SF16 - Bridging the gap between data science and data engineeringHakka Labs

DataEngConf SF16 - Multi-temporal Data StructuresHakka Labs

DataEngConf SF16 - Entity Resolution in Data Pipelines Using SparkHakka Labs

DataEngConf SF16 - Beginning with OurselvesHakka Labs

DataEngConf SF16 - Routing Billions of Analytics Events with High DeliverabilityHakka Labs

This document summarizes the architecture of Segment's analytics event routing system. It discusses constraints of high throughput and reliability. The initial system used RabbitMQ and MongoDB but scaled to NSQ and Go microservices. Metrics and queues were used to monitor and schedule fan-out of events. Microservices provided isolation and visibility while Docker enabled easy deployment. Looking ahead, the system may move to Kafka and standardize the microservices toolkit.

DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...Hakka Labs

DataEngConf SF16 - Methods for Content Relevance at LinkedInHakka Labs

DataEngConf SF16 - Spark SQL WorkshopHakka Labs

Always Valid Inference (Ramesh Johari, Stanford)Hakka Labs

DataEngConf SF16 - High cardinality time series searchHakka Labs

DataEngConf SF16 - Data Asserts: Defensive Data ScienceHakka Labs

DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast DataHakka Labs

DataEngConf SF16 - Recommendations at InstacartHakka Labs

DataEngConf SF16 - Running simulations at scaleHakka Labs

DataEngConf SF16 - Deriving Meaning from Wearable Sensor DataHakka Labs

DataEngConf SF16 - Collecting and Moving Data at Scale Hakka Labs

DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQHakka Labs

DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...Hakka Labs

DataEngConf SF16 - Three lessons learned from building a production machine l...Hakka Labs

DataEngConf SF16 - Scalable and Reliable Logging at PinterestHakka Labs

DataEngConf SF16 - Bridging the gap between data science and data engineeringHakka Labs

DataEngConf SF16 - Multi-temporal Data StructuresHakka Labs

DataEngConf SF16 - Entity Resolution in Data Pipelines Using SparkHakka Labs

DataEngConf SF16 - Beginning with OurselvesHakka Labs

DataEngConf SF16 - Routing Billions of Analytics Events with High DeliverabilityHakka Labs

DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...Hakka Labs

DataEngConf SF16 - Methods for Content Relevance at LinkedInHakka Labs

DataEngConf SF16 - Spark SQL WorkshopHakka Labs

Recently uploaded (20)

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Build Your Own Copilot & Agents For DevsBrian McKeiver

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

Cyber Awareness overview for 2025 month of securityriccardosl1

Splunk Security Update | Public Sector Summit Germany 2025Splunk

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Building 10x Organizations with Modern Productivity Metrics 10x developers may be a myth, but 10x organizations are very real, as proven by the influential study performed in the 1980s, ‘The Coding War Games.’ Right now, here in early 2025, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience. It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them. But which of these approaches actually work? DORA? SPACE? DevEx? What should we invest in and create urgency behind today, so that we don’t find ourselves having the same discussion again in a decade?

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

Talk at the final event of Data Fusion Dynamics: A Collaborative UK-Saudi Initiative in Cybersecurity and Artificial Intelligence funded by the British Council UK-Saudi Challenge Fund 2024, Cardiff Metropolitan University, 29th April 2025 https://alandix.com/academic/talks/CMet2025-AI-Changes-Everything/ Is AI just another technology, or does it fundamentally change the way we live and think? Every technology has a direct impact with micro-ethical consequences, some good, some bad. However more profound are the ways in which some technologies reshape the very fabric of society with macro-ethical impacts. The invention of the stirrup revolutionised mounted combat, but as a side effect gave rise to the feudal system, which still shapes politics today. The internal combustion engine offers personal freedom and creates pollution, but has also transformed the nature of urban planning and international trade. When we look at AI the micro-ethical issues, such as bias, are most obvious, but the macro-ethical challenges may be greater. At a micro-ethical level AI has the potential to deepen social, ethnic and gender bias, issues I have warned about since the early 1990s! It is also being used increasingly on the battlefield. However, it also offers amazing opportunities in health and educations, as the recent Nobel prizes for the developers of AlphaFold illustrate. More radically, the need to encode ethics acts as a mirror to surface essential ethical problems and conflicts. At the macro-ethical level, by the early 2000s digital technology had already begun to undermine sovereignty (e.g. gambling), market economics (through network effects and emergent monopolies), and the very meaning of money. Modern AI is the child of big data, big computation and ultimately big business, intensifying the inherent tendency of digital technology to concentrate power. AI is already unravelling the fundamentals of the social, political and economic world around us, but this is a world that needs radical reimagining to overcome the global environmental and human challenges that confront us. Our challenge is whether to let the threads fall as they may, or to use them to weave a better future.

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

Procurement Insights Cost To Value Guide.pptxJon Hansen

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.

tecnologias de las primeras civilizaciones.pdffjgm517

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

TrsLabs - Fintech Product & Business ConsultingTrs Labs

Hybrid Growth Mandate Model with TrsLabs Strategic Investments, Inorganic Growth, Business Model Pivoting are critical activities that business don't do/change everyday. In cases like this, it may benefit your business to choose a temporary external consultant. An unbiased plan driven by clearcut deliverables, market dynamics and without the influence of your internal office equations empower business leaders to make right choices. Getting things done within a budget within a timeframe is key to Growing Business - No matter whether you are a start-up or a big company Talk to us & Unlock the competitive advantage

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Build Your Own Copilot & Agents For DevsBrian McKeiver

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

Cyber Awareness overview for 2025 month of securityriccardosl1

Splunk Security Update | Public Sector Summit Germany 2025Splunk

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

Procurement Insights Cost To Value Guide.pptxJon Hansen

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

tecnologias de las primeras civilizaciones.pdffjgm517

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

TrsLabs - Fintech Product & Business ConsultingTrs Labs

So we're running Apache ZooKeeper. Now What? By Camille Fournier

1. So we’re running ZooKeeper. Now What? Camille Fournier, Rent the Runway @skamille

2. Big Data. Big Systems.

3. • Outages • Coordination • Operational Complexity Common Challenges

4. • Consistency guarantees Common Deficiency

5. Storm uses Zookeeper for coordinating the cluster. Zookeeper is not used for message passing, so the load Storm places on Zookeeper is quite low ZooKeeper in Storm

6. A centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services • • • • Distributed, Consistent Data Store Highly Available High performance Strictly ordered access ZooKeeper

7. • Tolerates the loss of a minority ((n/2) – 1) of ensemble members and still function Highly Available

8. • All data is stored in memory • Performance measured around 50,000 operations/second • Particularly fast for read performance, built for readdominant workloads High Performance

9. • Atomic Writes • In the order you sent them • Changes always seen in the order they occurred • Reliable, no writes acked will be dropped Strictly Ordered Access

10. leader cli cli follower follower cli cli cli cli Basics: Cluster Interactions

11. leader cli cli follower follower cli cli cli cli Basics: Cluster Interactions

12. leader cli cli follower cli cli cli cli Basics: Cluster Interactions

13. cli /a/b/myNode /a/b /a/b/d /a /a/c Data Structure /a/c/e000001

14. • Nodes can contain data, have children, or both • Ephemeral nodes are associated with the session that created them • They cannot have children, and disappear when that session ends • Sequential nodes have an ever-increasing number attached to them Basics: Data Structure

15. client leader getData “/foo” true client client Watches return “mydata” follower follower

16. client leader client follower setData “/foo” “bar” client Watches follower

17. client leader NOTIFICATION client follower client follower Watches

18. client leader getData “/foo” true client client Watches return “bar” follower follower

19. • Set against data or path changes • Ordered with respect to other events, other watches, and asynchronous replies. • A client will see a watch event for a node it is watching before seeing the new data that corresponds to that node. • The order of watch events corresponds to the order of the updates as seen by the ZooKeeper service • One time notifications; must be reset, changes can be missed between notification and reset of the watch Basics: Watches

20. • create • delete • setData Basics: Creation API

21. • exists • getData • getChildren Basics: Get/Watch API

22. • multi * new in 3.4 • sync Basics: API

23. Service Management Distributed Locking Common Uses

24. • In Storm, ZooKeeper is the source of communication between Nimbus and Supervisors • Nimbus finds Supervisors via ZooKeeper Coordination

25. Find servers doing job “Products” Encode as path in ZooKeeper: /servers/products Servers register as ephemeral nodes under this path with details about location, other connection info Discovery (Naming)

26. Read config from nodes Watch nodes for config changes Configuration

27. Shared Locks Barriers and Latches Leader Election Two-Phase Commit Locking

28. And Now, The Scary Part

29. The State Machine

30. NOPE

31. • Curator (Java) • Kazoo (Python) • Twitter Commons for Discovery Recommended Clients

32. ZooKeeper Owns Your Availability (maybe) Be Aware

33. • • • • • • Thank you to @zaa for the format of the slide on watches Tweet me! @skamille Email me! [email protected] Kazoo: http://kazoo.readthedocs.org/en/latest/ Curator: http://curator.incubator.apache.org/ Twitter commons: http://twitter.github.io/commons/ Credits and Contact

Editor's Notes

#4: Example of outage…Nodes Goes DownNetwork PartitionsDisk CorruptionCoordination: Task AssignmentOperational Complexity: Finding other cluster membersDynamic ConfigurationGroup Membership
#23: If you use the sync call before a read, ZooKeeper providesilnearizability for sync+read and write operations (this is true withcertain timing assumption made in ZooKeeper for efficiency).