Apache ZooKeeper TechTuesday

Aug 24, 20107 likes2,881 views

ZooKeeper is a highly available, scalable, distributed configuration, consensus, group membership, leader election, naming and coordination service. It provides a hierarchical namespace and basic operations like create, delete, and read data. It is useful for building distributed applications and services like queues. Future releases will focus on monitoring improvements, read-only mode, and failure detection models. The community is working on features like children for ephemeral nodes and viewing session information.

Apache ZooKeeper

Why use it? What to expect in the future?

Andrei Savu @TechTuesday

Outline

Why use it?
Crash Course
Practical Example

What to expect in the future (3.4.0 release)?
GSoC 2010
New Contrib
Work in Progress

What is ZooKeeper?

A highly available, scalable, distributed,
configuration, consensus, group membership,
leader election, naming and coordination
service.

What is ZooKeeper? (2)

replicated in memory tree data structure
somehow similar to a file system
no partial read / writes
no renames
ordered updates
strong persistence guarantees
conditional updates (version)
watches for data changes
ephemeral nodes
generated file names

ZooKeeper Data Model

hierarchical namespace
each znode has
data and children
data is read and
written in its entirety

Basic ZooKeeper API

string create(path, data, acl, flags)

delete(path, expected_version)

stat set_data(path, data, expected_version)

(data, stat) get_data(path, watch)

stat exists(path, watch)

string[] get_children(path, watch)

ZooKeeper Service

Facts: 1) all servers store a copy of the data in memory 2) the
leader is elected at startup 3) followers respond to clients 4) all
updates go through the leader 5) responses are sent when a
majority of servers have persisted the change

Distributed Queue (Python)

http://www.cloudera.com/blog/2009/05/building-a-
distributed-concurrent-queue-with-apache-zookeeper/
http://github.com/henryr/pyzk-recipes
Retry operation on ConnectionLoss:
http://github.com/andreisavu/pyzk-recipes

1. Monitoring & Web-based interface

Status: Committed to the trunk

1. JIRA: ZOOKEEPER-701
2. Progress Tracking Wiki
3. Monitoring for Ganglia, Nagios and Cacti
1. contrib / monitoring
2. 'mntr' 4letter word
4. Web interface available as a Hue application
1. contrib / huebrowser
2. complete install instructions
3. requirements: rest gateway, Hue 1.0

2. Read-Only Mode

Status: Under Review (Ready to be committed)

1. JIRA: ZOOKEEPER-704
2. Progress Tracking Wiki
3. Description: "When a ZooKeeper server loses contact with
over half of the other servers in an ensemble ('loses a
quorum'), it stops responding to client requests. For some
applications, it would be beneficial if a server still responded
to read requests when the quorum is lost, but caused an
error condition when a write request was attempted."

3. Failure Detector Model

Status: Under Review

1. JIRA: ZOOKEEPER-702
2. Progress Tracking Wiki
3. Detectors: Phi Accrual, Chen, Bertier, Fixed Heartbeat
4. Why? Check the concluding remarks on the wiki.
5. Conclusion snippet: "in scenarios where we have a
changing network behavior, such in a WAN, the adaptive
methods can be a good pick"

Large Scale Pub/Sub (hedwig)

1. JIRA: ZOOKEEPER-775
2. Uses ZooKeeper and BookKeeper
3. Committed to the trunk
4. Developed at Yahoo! Research
5. Used for PNUTS cross data center replication
6. http://vimeo.com/13282102

Work in Progress
only some interesting JIRAs

#834 Children for ephemerals

JIRA: ZOOKEEPER-834
Allow ephemeral nodes to have children owned by the same
session.
Useful when publishing status information. No need to do
serialization for basic data structures (hash tables)
Similar to /proc in *nix systems.
Examples: /agent-01/ip, /agent-01/memory, /agent-01/load

#829 /zookeeper/sessions/*

JIRA: ZOOKEEPER-829
Requested by HBase developers: " we'd like the ability to
forcible expire someone else's ZK session "

Plenty of bug fixes

Join the community!

Resources

http://hadoop.apache.org/zookeeper/
http://wiki.apache.org/hadoop/ZooKeeper/ProjectDescription
http://wiki.apache.org/hadoop/ZooKeeper/Tao

ZooKeeper is a distributed coordination service that allows distributed applications to synchronize data and configuration information. It uses a data model of directories and files, called znodes, that can contain small amounts of structured data. ZooKeeper maintains data consistency through a leader election process and quorum-based consensus algorithm called Paxos. It provides applications with synchronization primitives and configuration maintenance in a highly-available and reliable way.

Zookeeper In Actionjuvenxu

Introduction to ZooKeeper - TriHUG May 22, 2012mumrah

zookeeperProgrammersHiroshi Ono

This document provides a guide for developing distributed applications that use ZooKeeper. It discusses ZooKeeper's data model including znodes, ephemeral nodes, and sequence nodes. It describes ZooKeeper sessions, watches, consistency guarantees, and available bindings. It provides an overview of common ZooKeeper operations like connecting, reads, writes, and handling watches. It also discusses program structure, common problems, and troubleshooting. The guide is intended to help developers understand key ZooKeeper concepts and how to integrate ZooKeeper coordination services into their distributed applications.

Centralized Application Configuration with Spring and Apache ZookeeperRyan Gardner

From talk given at Spring One 2gx Dallas, 2014 Application configuration is an evolution. It starts as a hard-coded strings in your application and hopefully progresses to something external, such as a file or system property that can be changed without deployment. But what happens when other enterprise concerns enter the mix, such as audit requirements or access control around who can make changes? How do you maintain the consistency of values across too many application servers to manage at one time from a terminal window? The next step in the application configuration evolution is centralized configuration that can be accessed by your applications as they move through your various environments on their way to production. Such a service transfers the ownership of configuration from the last developer who touched the code to a well-versed application owner who is responsible for the configuration of the application across all environments. At Dealer.com, we have created one such solution that relies on Apache ZooKeeper to handle the storage and coordination of the configuration data and Spring to handle to the retrieval, creation and registration of configured objects in each application. The end result is a transparent framework that provides the same configured objects that could have been created using a Spring configuration, configuration file and property value wiring. This talk will cover both the why and how of our solution, with a focus on how we leveraged the powerful attributes of both Apache ZooKeeper and Spring to rid our application of local configuration files and provide a consistent mechanism for application configuration in our enterprise.

Meetup on Apache ZookeeperAnshul Patel

Zookeeper is a distributed coordination service that provides naming, configuration, synchronization, and group services. It allows distributed processes to coordinate with each other through a shared hierarchical namespace of data registers called znodes. Zookeeper follows a leader-elected consensus protocol to guarantee atomic broadcast of state updates from the leader to followers. It uses a hierarchical namespace of znodes similar to a file system to store configuration data and other application-defined metadata. Zookeeper provides services like leader election, group membership, synchronization, and configuration management that are essential for distributed systems.

Distributed system coordination by zookeeper and introduction to kazoo python...Jimmy Lai

Introduction to Apache ZooKeeperSaurav Haloi

Winter is coming? Not if ZooKeeper is there!Joydeep Banik Roy

[Download the slide to get the entire talk in the form of presentation note embedded in the ppt] Apache ZooKeeper is the chosen leader in distributed coordination. In this talk, I have explored the atomic elements of Apache ZooKeeper, how it fits everything together and some of its popular use cases. For ZooKeeper simplicity is the key and as a consumer of the API, our imagination enables us to push the limits of the ZooKeeper world.

So we're running Apache ZooKeeper. Now What? By Camille Fournier Hakka Labs

The ZooKeeper framework was originally built at Yahoo! to make it easy for the company’s applications to access configuration information in a robust and easy-to-understand way, but it has since grown to offer a lot of features that help coordinate work across distributed clusters. Apache Zookeeper became a de-facto standard for coordination service and used by Storm, Hadoop, HBase, ElasticSearch and other distributed computing frameworks.

Apache ZooKeeperScott Leberknight

Apache zookeeper seminar_trinh_viet_dung_03_2016Viet-Dung TRINH

Introduction to apache zoo keeper Omid Vahdaty

Apache ZookeeperNguyen Quang

ZooKeeper is an open-source coordination service for distributed applications that provides common services like naming, configuration management, synchronization, and groups. It uses a hierarchical data model where data is stored as znodes that can be configured with watches, versions, access control, and more. Common uses include distributed queues, leader election, and group membership. Recipes demonstrate how to implement queues and group membership using ZooKeeper.

Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLabCloudxLab

Big Data with Hadoop & Spark Training: http://bit.ly/2kvXlPd This CloudxLab Introduction to Apache ZooKeeper tutorial helps you to understand ZooKeeper in detail. Below are the topics covered in this tutorial: 1) Data Model 2) Znode Types 3) Persistent Znode 4) Sequential Znode 5) Architecture 6) Election & Majority Demo 7) Why Do We Need Majority? 8) Guarantees - Sequential consistency, Atomicity, Single system image, Durability, Timeliness 9) ZooKeeper APIs 10) Watches & Triggers 11) ACLs - Access Control Lists 12) Usecases 13) When Not to Use ZooKeeper

Curator introJordan Zimmerman

The document provides an overview and summary of Curator, a client library for Apache ZooKeeper. Curator aims to simplify ZooKeeper development by providing a friendlier API, handling retries, and implementing common patterns ("recipes") like leader election and locks. It consists of a client, framework, and recipes components. The framework handles connection management and retries, while recipes implement distributed primitives. Details about common recipes like locks and leader election are provided.

[2C1] 아파치 피그를 위한 테즈 연산 엔진 개발하기 최종NAVER D2

This document discusses the development of Apache Pig on Tez, an execution engine for Pig jobs. Pig on Tez allows Pig workflows to be executed as directed acyclic graphs (DAGs) using Tez, improving performance over the default MapReduce execution. Key benefits of Tez include eliminating intermediate data writes, reducing job launch overhead, and allowing more flexible data flows. However, challenges remain around automatically determining optimal parallelism and integrating Tez with user interface and monitoring tools. Future work is needed to address these issues.

使用ZooKeeper打造軟體式負載平衡Lawrence Huang

ZooKeeper is a coordination service for distributed systems that allows nodes to perform tasks correctly. It provides features like atomic operations, order guarantees, and high availability. ZooKeeper uses the Zab protocol to elect a leader and achieve consensus. It stores data in a hierarchical structure of znodes that can be persistent or ephemeral. Software load balancers can use ZooKeeper to store cluster configuration and direct traffic. The Apache Curator library provides a higher-level API for common ZooKeeper patterns like leader election and locks.

ZooKeeper - wait free protocol for coordinating processesJulia Proskurnia

Zookeeper big sonataAnh Le

ZooKeeper is an open-source coordination service for distributed applications that provides common services like leader election, configuration management, and locks in a simple interface to help distributed processes coordinate actions and share information. It provides guarantees around consistency, reliability, and timeliness to applications using its hierarchical data model and APIs. Popular distributed systems like Hadoop and Kafka use ZooKeeper for tasks such as cluster management, metadata storage, and detecting node failures.

A Python Petting Zoodevondjones

This document discusses Knewton's use of ZooKeeper and PettingZoo to implement distributed machine learning on a Python cluster. It begins by explaining what ZooKeeper is and how it provides services for distributed synchronization. It then discusses the state of ZooKeeper libraries for Python, including incomplete bindings and lack of high-level recipes. PettingZoo is introduced as Knewton's library that implements common ZooKeeper recipes for Python, allowing their machine learning models to be sharded and scaled across multiple machines. Distributed discovery, distributed bags, leader queues, and role matching are highlighted as key recipes that enable dynamic reconfiguration and load balancing of their distributed system.

Introduction to Apache ZooKeeperknowbigdata

This is Apache ZooKeeper session. ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. By the end of this presentation you should be fairly clear about Apache ZooKeeper. To watch the video or know more about the course, please visit http://www.knowbigdata.com/page/big-data-and-hadoop-online-instructor-led-training

Distributed Coordination with PythonOSCON Byrum

Running High Performance & Fault-tolerant Elasticsearch Clusters on DockerSematext Group, Inc.

This document discusses running Elasticsearch clusters on Docker containers. It describes how Docker containers are more lightweight than virtual machines and have less overhead. It provides examples of running official Elasticsearch Docker images and customizing configurations. It also covers best practices for networking, storage, constraints, and high availability when running Elasticsearch on Docker.

Introduction to Mesoskoboltmarky

Mesos is an open source cluster management framework that provides efficient resource isolation and sharing across distributed applications or frameworks. It was first released in 2009 at Berkeley and uses a master-slave architecture with 10,000 lines of C++ code. Mesos allows organizations to combine CPUs, memory, storage and other resources into a shared pool that can be allocated to applications known as frameworks.

How to Run Solr on Docker and WhySematext Group, Inc.

Docker is all the rage these days. While one doesn't hear much about Solr on Docker, we're here to tell you not only that it can be done, but also share how it's done. We'll quickly go over the basic Docker ideas - containers are lighter than VMs, they solve "but it worked on my laptop" issues - so we can dive into the specifics of running Solr on Docker. We'll do a live demo showing you how to run Solr master - slave as well as SolrCloud using containers, how to manage CPU assignments, constraint memory and use Docker data volumes when running Solr in containers. We will also show you how to create your own containers with custom configurations. Finally, we'll address one of the core Solr questions - which deployment type should I use? We will demonstrate performance differences between the following deployment types: - Single Solr instance running on a bare metal machine - Multiple Solr instances running on a single bare metal machine - Solr running in containers - Solr running on virtual machine - Solr running on virtual machine using unikernel For each deployment type we'll address how it impacts performance, operational flexibility and all other key pros and cons you ought to keep in mind.

Ansible not only for DummiesŁukasz Proszek

Elasticsearch for Logs & Metrics - a deep diveSematext Group, Inc.

Zookeeperltsllc

Dynamic Reconfiguration of Apache ZooKeeperDataWorks Summit

ZooKeeper allows for dynamic reconfiguration of servers in its ensemble. Manual reconfiguration is problematic as it requires changing the configuration, restarting servers, and can result in data loss. The presented solution allows ZooKeeper to reconfigure itself automatically through a speculative reconfiguration approach. It commits the reconfiguration once quorums of both the old and new ensembles acknowledge it, and gossips the new configuration to ensure all servers sync before activation. This allows reconfigurations to complete without failures in a transparent manner to clients.

More Related Content

What's hot (20)

Winter is coming? Not if ZooKeeper is there!Joydeep Banik Roy

So we're running Apache ZooKeeper. Now What? By Camille Fournier Hakka Labs

Apache ZooKeeperScott Leberknight

Apache zookeeper seminar_trinh_viet_dung_03_2016Viet-Dung TRINH

Introduction to apache zoo keeper Omid Vahdaty

Apache ZookeeperNguyen Quang

Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLabCloudxLab

Curator introJordan Zimmerman

[2C1] 아파치 피그를 위한 테즈 연산 엔진 개발하기 최종NAVER D2

使用ZooKeeper打造軟體式負載平衡Lawrence Huang

ZooKeeper - wait free protocol for coordinating processesJulia Proskurnia

Zookeeper big sonataAnh Le

A Python Petting Zoodevondjones

Introduction to Apache ZooKeeperknowbigdata

Distributed Coordination with PythonOSCON Byrum

Running High Performance & Fault-tolerant Elasticsearch Clusters on DockerSematext Group, Inc.

Introduction to Mesoskoboltmarky

How to Run Solr on Docker and WhySematext Group, Inc.

Ansible not only for DummiesŁukasz Proszek

Elasticsearch for Logs & Metrics - a deep diveSematext Group, Inc.

Winter is coming? Not if ZooKeeper is there!Joydeep Banik Roy

So we're running Apache ZooKeeper. Now What? By Camille Fournier Hakka Labs

Apache ZooKeeperScott Leberknight

Apache zookeeper seminar_trinh_viet_dung_03_2016Viet-Dung TRINH

Introduction to apache zoo keeper Omid Vahdaty

Apache ZookeeperNguyen Quang

Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLabCloudxLab

Curator introJordan Zimmerman

[2C1] 아파치 피그를 위한 테즈 연산 엔진 개발하기 최종NAVER D2

使用ZooKeeper打造軟體式負載平衡Lawrence Huang

ZooKeeper - wait free protocol for coordinating processesJulia Proskurnia

Zookeeper big sonataAnh Le

A Python Petting Zoodevondjones

Introduction to Apache ZooKeeperknowbigdata

Distributed Coordination with PythonOSCON Byrum

Running High Performance & Fault-tolerant Elasticsearch Clusters on DockerSematext Group, Inc.

Introduction to Mesoskoboltmarky

How to Run Solr on Docker and WhySematext Group, Inc.

Ansible not only for DummiesŁukasz Proszek

Elasticsearch for Logs & Metrics - a deep diveSematext Group, Inc.

Viewers also liked (16)

Zookeeperltsllc

Dynamic Reconfiguration of Apache ZooKeeperDataWorks Summit

Distributed Applications with Apache ZookeeperAlex Ehrnschwender

Introduction to Kafka and ZookeeperRahul Jain

Groovy to gradleGeng-Dian Huang

This document discusses how Groovy powers Gradle's domain-specific language (DSL). It outlines Groovy's features like optional syntax, closures, and Java integration. It then shows how Gradle uses Groovy to define its DSL, with examples like applying plugins and defining tasks. Finally, it mentions that AST transformations allow customizing Groovy's compilation for the Gradle DSL, and that Launcher.java loads Groovy scripts while TaskTransformer.java builds task method calls from the Groovy AST.

ZooKeeper FuturesCloudera, Inc.

Henry Robinson gave a presentation on upcoming features for ZooKeeper. He discussed observers, which allow non-voting servers to scale client connections without impacting performance. He also covered dynamic ensembles, which would allow changing the ZooKeeper cluster membership without downtime. Finally, he announced that Cloudera's Distribution for Hadoop will include ZooKeeper packages to integrate it more fully.

ZooKeeper (and other things)Jonathan Halterman

Taming Pythons with ZooKeeperJyrki Pulliainen

Concurrency is hard. Consistency in distributed systems is hard. And then the whole thing should be highly-available and error resilient. Fear not, there are good news: There exists an awesome tool called ZooKeeper to help you with this. There even exists a plethora of Python libraries for it, but how to know what to use and how? This talk will walk you through ZooKeeper and how to use it with Python. We’ll be focusing on what I think is the most prominient ZooKeeper library out there for Python: Kazoo. You’ll see how to do things in ZooKeeper and how to implement them using Kazoo. We’ll also peek in to the recipes Kazoo offers, and if we have enough time, touch a real life application we’ve used Kazoo and ZooKeeper to build at Spotify.

Taming Pythons with ZooKeeper (Pyconfi edition)Jyrki Pulliainen

Apache ZooKeeper is a distributed coordination service that allows distributed applications to synchronize data and configuration information and maintain consistency. It provides a hierarchical name space and simple primitives like counters and change notifications. ZooKeeper elects a leader automatically and guarantees consistency by requiring a quorum for all operations. It is useful for applications that need to maintain consistency but also remain available, as it allows choosing consistency and availability over partition tolerance. Spotify uses ZooKeeper to orchestrate distributed tasks across its infrastructure while maintaining a consistent order and remaining available within data centers.

ZooKeeper and Embedded ZooKeeper Support for IBM InfoSphere Streams V4.0lisanl

Yip-Hing Ng works as a senior software engineer on the Streams Platform development team. Yip's presentation describes how to setup and configure ZooKeeper for IBM InfoSphere Streams V4.0. View related presentations and recordings from the Streams V4.0 Developers Conference at: https://developer.ibm.com/answers/questions/183353/ibm-infosphere-streams-40-developers-conference-on.html?smartspace=streamsdev

Overview of Zookeeper, Helix and Kafka (Oakjug)Chris Richardson

Jcconf 2016 zookeeperMatt Ho

The document discusses using ZooKeeper to modify existing distributed systems. It introduces ZooKeeper and common ZooKeeper operations like starting servers, connecting clients, creating and deleting znodes, watching for changes. It then covers how to implement leader election using ZooKeeper. Finally, it discusses how to integrate ZooKeeper into a scheduling framework in Java by adapting scheduled methods and using a ZooKeeper-aware TaskScheduler. Sample code is provided and improvements like load balancing leaders are suggested.

ZookeeperGeng-Dian Huang

ZooKeeper is a centralized coordination service for distributed applications. It allows distributed processes to synchronize with each other using shared znodes. ZooKeeper uses a leader election algorithm and quorum consensus to ensure high availability. It provides recipes for common distributed coordination problems like queues, locks, and leader election. ZooKeeper's internals use the ZAB protocol for consensus between leader and followers through epochs and proposals.

Apache Zookeeper 分布式服务框架Cabin WJ

Zookeeperandres felipe cortes amaya

Este documento describe ZooKeeper, un servicio de coordinación para aplicaciones distribuidas. ZooKeeper permite que los procesos distribuidos se coordinen a través de un espacio de nombres jerárquico compartido. Proporciona servicios como administración de configuración, bloqueos, sincronización y grupos de servicios. Los desarrolladores pueden usar ZooKeeper en lugar de tener que escribir su propia lógica de coordinación distribuida.

Comparing ZooKeeper and ConsulIvan Glushkov

Zookeeperltsllc

Dynamic Reconfiguration of Apache ZooKeeperDataWorks Summit

Distributed Applications with Apache ZookeeperAlex Ehrnschwender

Introduction to Kafka and ZookeeperRahul Jain

Groovy to gradleGeng-Dian Huang

ZooKeeper FuturesCloudera, Inc.

ZooKeeper (and other things)Jonathan Halterman

Taming Pythons with ZooKeeperJyrki Pulliainen

Taming Pythons with ZooKeeper (Pyconfi edition)Jyrki Pulliainen

ZooKeeper and Embedded ZooKeeper Support for IBM InfoSphere Streams V4.0lisanl

Overview of Zookeeper, Helix and Kafka (Oakjug)Chris Richardson

Jcconf 2016 zookeeperMatt Ho

ZookeeperGeng-Dian Huang

Apache Zookeeper 分布式服务框架Cabin WJ

Zookeeperandres felipe cortes amaya

Comparing ZooKeeper and ConsulIvan Glushkov

Similar to Apache ZooKeeper TechTuesday (20)

Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...Accumulo Summit

Talk Abstract In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time. Speaker Joe Stein Principal Consultant, Big Data Open Source Security, LLC Joe Stein is an Apache Kafka committer and PMC member. Joe is the Founder and Principal Architect of Big Data Open Source Security LLC a professional services and product solutions company. Joe has been a developer, architect and technologist professionally for 15 years now having built back end systems that supported over one hundred million unique devices a day processing trillions of events. He blogs and hosts a podcast about Hadoop and related systems at All Things Hadoop and tweets @allthingshadoop

Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein

In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.

Hadoop Vectored IOSteve Loughran

StorageQuery: federated querying on object stores, powered by Alluxio and PrestoAlluxio, Inc.

Alluxio Global Online Meetup August 25, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Abner Ferreira, Simbiose Ventures Caio Pavanelli, Simbiose Ventures Bin Fan, Alluxio Over the last few years, organizations have worked towards the separation of storage and compute for a number of benefits in the areas of cost, data duplication and data latency. Cloud resolves most of these issues but comes to the expense of needing a way to query data on remote storages. Alluxio and Presto are a powerful combination to address the compute problem, which is part of the strategy used by Simbiose Ventures to create a product called StorageQuery - A platform to query files in cloud storages with SQL. This talk will focus on: - How Alluxio fits StorageQuery's tech stack; - Advantages of using Alluxio as a cache layer and its unified filesystem; - Development of new under file system for Backblaze B2 and fine-grained code documentation; - ShannonDB remote storage mode.

Devoxx Belgium 2015GiedriusTS

The document summarizes key topics from the DEVOXX BE 2015 conference including Java 9 modular programming, HTTP/2 and Java 9, JSON API upgrades for Java EE 8, Spring roadmap, understanding Git internals, principles of microservices, best practices for Java deployment, and visualizing architecture. It provides an overview of 10 top talks at the conference focused on Java and web development topics.

Базы данных. ZooKeeperVadim Tsesko

This document summarizes an Apache ZooKeeper presentation given on November 11, 2013. It introduces ZooKeeper as a high-availability and reliable service for distributed applications. It provides an overview of ZooKeeper's data model, APIs, common usages, architecture and throughput. The presentation also discusses how ZooKeeper provides solutions for distributed coordination problems like naming, configuration, leader election, and synchronization.

Streaming Processing with a Distributed Commit LogJoe Stein

Hadoop introductionmusrath mohammad

FHIR Server internals - sqlonfhirBrian Postlethwaite

OsqueryAnimesh Roy

This document provides an introduction to OS Query, an open source tool for querying information about operating systems. It outlines lessons that will cover OS Query basics, deploying OS Query, and running basic commands. Examples of queries are provided to find running processes, network connections, kernel modules, and more. Use cases are described for how OS Query can be used to detect new listening processes, outbound network activity, deleted process binaries, and loaded kernel modules.

Large Scale Crawling with Apache Nutch and Friendslucenerevolution

Presented by Julien Nioche, Director, DigitalPebble This session will give an overview of Apache Nutch. I will describe its main components and how it fits with other Apache projects such as Hadoop, SOLR, Tika or HBase. The second part of the presentation will be focused on the latest developments in Nutch, the differences between the 1.x and 2.x branch and what we can expect to see in Nutch in the future. This session will cover many practical aspects and should be a good starting point to crawling on a large scale with Apache Nutch and SOLR.

Large Scale Crawling with Apache Nutch and FriendsJulien Nioche

This session will give an overview of Apache Nutch. I will describe its main components and how it fits with other Apache projects such as Hadoop, SOLR, Tika or HBase. The second part of the presentation will be focused on the latest developments in Nutch, the differences between the 1.x and 2.x branch and what we can expect to see in Nutch in the future. This session will cover many practical aspects and should be a good starting point to crawling on a large scale with Apache Nutch and SOLR.

Jupyter Ascending: a practical hand guide to galactic scale, reproducible dat...John Fonner

This document summarizes a presentation about Jupyter Ascending, a practical guide to reproducible data science at a galactic scale. It discusses the challenges scientists face with programming and data management. It introduces Agave, an API and tools for provisioning systems, applications, and jobs on various infrastructures. It also describes AgavePy and an Agave-aware JupyterHub for running Jupyter notebooks that leverage Agave for scalable and reproducible data exploration and analysis. Future work includes a developer portal and production JupyterHub support.

Practical Chaos EngineeringSIGHUP

A Big, Fast and Persistent QueueWilliam Yang

BigQueue is a fast, big, and persistent queue that stores data on disk, making it crash resistant. It allows for concurrent enqueue and dequeue operations in close to O(1) time. The queue can scale to large sizes limited only by available disk space, and provides both consume-once and fanout queue semantics. It uses memory mapping and paging techniques to keep recently accessed data in memory efficiently while accessing the full data set on disk.

SFBay Area Solr Meetup - June 18th: Benchmarking Solr PerformanceLucidworks (Archived)

The document discusses benchmarking the performance of SolrCloud clusters. It describes Timothy Potter's experience operating a large SolrCloud cluster at Dachis Group. It outlines an methodology for benchmarking indexing performance by varying the number of servers, shards, and replicas. Results show near-linear scalability as nodes are added. The document also introduces the Solr Scale Toolkit for deploying and managing SolrCloud clusters using Python and AWS. It demonstrates integrating Solr with tools like Logstash and Kibana for log aggregation and dashboards.

JCache data store for Apache GoraKevin Ratnasekera

This document discusses Kevin Ratnasekera's proposal to implement a JCache data store for the Apache Gora project. It provides background on GSoC, Apache Gora, the need for a caching layer, how JCache works, the proposed design for a JCache data store in Gora using Hazelcast, possible improvements, sample code, documentation, and the project roadmap. The goal is to improve Gora performance by adding caching capabilities via the JCache API in a way that is decoupled from the underlying data store.

Everything you wanted to know about writing async, concurrent http apps in java Baruch Sadogursky

As presented at CodeMotion Tel Aviv: Facing tens of millions of clients continuously downloading binaries from its repositories, JFrog decided to offer an OSS client that natively supports these downloads. This session shares the main challenges of developing a highly concurrent, resumable, async download library on top of an Apache HTTP client. It also covers other libraries JFrog tested and why it decided to reinvent the wheel. Consider yourself forewarned: lots of HTTP internals, NIO, and concurrency ahead!

ApacheCon NA 2011 reportKoji Kawamura

The document summarizes the speaker's experience at ApacheCon NA 2011. It discusses several keynotes and sessions attended, including talks on building secure software, the success of Hadoop, Watson's use of Apache technologies, and new features in Lucene 4.0 like improved performance through UTF8 encoding and faster querying through new query types and indexing approaches. The document also mentions several projects that build user interfaces and experiences on top of Solr, like Prism, Blacklight, TwigKit, and Ajax Solr.

Sanger, upcoming Openstack for Bio-informaticiansPeter Clapham

Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...Accumulo Summit

Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein

Hadoop Vectored IOSteve Loughran

StorageQuery: federated querying on object stores, powered by Alluxio and PrestoAlluxio, Inc.

Devoxx Belgium 2015GiedriusTS

Базы данных. ZooKeeperVadim Tsesko

Streaming Processing with a Distributed Commit LogJoe Stein

Hadoop introductionmusrath mohammad

FHIR Server internals - sqlonfhirBrian Postlethwaite

OsqueryAnimesh Roy

Large Scale Crawling with Apache Nutch and Friendslucenerevolution

Large Scale Crawling with Apache Nutch and FriendsJulien Nioche

Jupyter Ascending: a practical hand guide to galactic scale, reproducible dat...John Fonner

Practical Chaos EngineeringSIGHUP

A Big, Fast and Persistent QueueWilliam Yang

SFBay Area Solr Meetup - June 18th: Benchmarking Solr PerformanceLucidworks (Archived)

JCache data store for Apache GoraKevin Ratnasekera

Everything you wanted to know about writing async, concurrent http apps in java Baruch Sadogursky

ApacheCon NA 2011 reportKoji Kawamura

Sanger, upcoming Openstack for Bio-informaticiansPeter Clapham

More from Andrei Savu (20)

The Evolving Landscape of Data EngineeringAndrei Savu

The document discusses the evolving landscape of data engineering. It provides context on the past, present, and future of data engineering. Specifically, it notes that in the past, data engineering was driven by open source communities and the early histories of AWS and Google Cloud. It describes common present-day patterns like serverless architectures and data locality. Finally, it outlines a future wish list, including data catalogs, monitoring systems, and more intelligent data infrastructure. The document concludes by offering recommendations on where to start with technologies, Google Cloud courses, and developing domain knowledge.

The Evolving Landscape of Data EngineeringAndrei Savu

Data Engineering is a relatively new, but fast evolving discipline that spans multiple environments and technologies, from traditional data centers to hyper-scale cloud providers, a discipline that combines closed-source, homegrown and open source software to create scalable data pipelines and power incredible new product features. In this presentation, we will go over the last 5-10 years of technology trends and advancements and bring all of that together in a story about modern day Data Engineering and the magic behind it.

Recap on AWS Lambda after re:Invent 2015Andrei Savu

One Hadoop, Multiple Clouds - NYC Big Data MeetupAndrei Savu

Introducing Cloudera Director at Big Data BashAndrei Savu

APIs & Underlying Protocols #APICraftSFAndrei Savu

Challenges for running Hadoop on AWS - AdvancedAWS MeetupAndrei Savu

Nowadays we've got all the tools we need to spin-up and tear-down clusters with hundreds of nodes in minutes and this puts more pressure on the tools we use to configure and monitor our applications. This challenge is even more interesting when we have to deal with long running distributed data storage and processing systems like Hadoop. In this talk we will look into some of the challenges we need to deal with when creating and managing Hadoop clusters in AWS, we will discuss improvement opportunities in monitoring (e.g. detecting and dealing with instance failure, resource contention & noisy neighbors) and a bit about the future and how we should go about disconnecting workload dispatch from cluster lifecycle.

Cloud as a Data PlatformAndrei Savu

Apache Provisionr (incubating) - Bucharest JUG 10Andrei Savu

Creating pools of Virtual Machines - ApacheCon NA 2013Andrei Savu

Data Scientist ToolboxAndrei Savu

This document provides an overview of the data science process and tools for a data science project. It discusses identifying important business questions to answer with data, extracting relevant data from sources, cleaning and sampling the data, analyzing samples to create models and check hypotheses, applying results to full data sets, visualizing findings, automating and deploying solutions, and continuously learning and improving through an iterative process. Key tools mentioned include Hadoop, R, Python, Excel, and various data wrangling, analysis, and visualization tools.

Axemblr Provisionr 0.3.x OverviewAndrei Savu

Simple Service for Managing Pools of 10s or 100s of Virtual Machines With Provisionr we want to solve the problem of cloud portability by hiding completely the API and only focusing on building a cluster that matches the same set of assumptions on all clouds, assumptions like: running a specific operating system (e.g. Ubuntu LTS), having the same set of pre-installed packages and binaries, sane dns settings (forward & reverse ip resolution - as needed for Hadoop), ntp settings, networking settings, ssh admin access, vpn access etc.

2012 in Review - Bucharest JUGAndrei Savu

The document summarizes the Bucharest JUG (Java User Group) meetings from May 2012 to November 2012. It provides details on the monthly meetings including average attendance of 25-80 people, topics covered such as Guava, Maven, JavaScript UI for REST, and thanks to speakers and sponsors. It outlines future plans such as live streaming international speakers, a donations/job board, and potential future topics around alternative build systems, deployment options, and monitoring tools. Contact details are provided.

Metrics for Web Applications - Netcamp 2012Andrei Savu

Counters with Riak on Amazon EC2 at HackoverAndrei Savu

The document discusses using distributed counters with Riak on Amazon EC2. It introduces Riak as a distributed key-value database focused on availability, fault tolerance, simplicity and scalability. It describes using Riak features like consistent hashing, replication and automatic load balancing to implement a REST API for counters with eventual consistency. The demo shows implementing and accessing counters across local machines and EC2 regions to demonstrate the architecture.

Simple REST with DropwizardAndrei Savu

This document provides an overview of using Dropwizard, an open-source Java framework, to build RESTful web services. It discusses REST concepts like resources and representations, REST verbs like GET and POST, and architectures for REST APIs. It then introduces Dropwizard and its components for building HTTP services with features like Jetty, Jersey, Jackson, and metrics support. The document demonstrates a sample Dropwizard TODO list application with REST endpoints and resources and discusses considerations for development, testing, and deployment.

Guava Overview Part 2 Bucharest JUG #2 Andrei Savu

This document provides an overview of Guava and discusses caches and services. Guava is Google's core Java library that contains utilities like caches, primitives, collections, and concurrency libraries. Caches can improve performance by storing values to avoid expensive re-computation. Services in Guava define lifecycles for objects with operational state and allow asynchronous starting and stopping. The document describes cache eviction strategies, service implementations, and where to find more information on Guava features like functional idioms and concurrency.

Guava Overview. Part 1 @ Bucharest JUG #1 Andrei Savu

Guava is a Java library developed by Google that includes common libraries such as collections, caching, primitives support, concurrency libraries, and generalized utility classes. The talk covered the basic utilities in Guava including using and avoiding null, preconditions, common object utilities, ordering, and primitive array utilities. It also discussed the collections in Guava including immutable collections, new collection types like multisets and multimaps, collection utilities, and ways to extend the collections framework with decorators.

Polyglot Persistence & Big Data in the CloudAndrei Savu

This document discusses polyglot persistence and deploying big data technologies in the cloud. It introduces databases like HBase, Cassandra, MongoDB, CouchDB, and Riak for storing large amounts of data. Technologies like Elasticsearch and Solr are presented for search. Hadoop is described as a framework for distributed processing of large datasets. The document concludes by discussing how the Apache Whirr project can be used to deploy these technologies on cloud infrastructure.

Building a Great Team in Open Source - Open Agile 2011Andrei Savu

The Evolving Landscape of Data EngineeringAndrei Savu

Recap on AWS Lambda after re:Invent 2015Andrei Savu

One Hadoop, Multiple Clouds - NYC Big Data MeetupAndrei Savu

Introducing Cloudera Director at Big Data BashAndrei Savu

APIs & Underlying Protocols #APICraftSFAndrei Savu

Challenges for running Hadoop on AWS - AdvancedAWS MeetupAndrei Savu

Cloud as a Data PlatformAndrei Savu

Apache Provisionr (incubating) - Bucharest JUG 10Andrei Savu

Creating pools of Virtual Machines - ApacheCon NA 2013Andrei Savu

Data Scientist ToolboxAndrei Savu

Axemblr Provisionr 0.3.x OverviewAndrei Savu

2012 in Review - Bucharest JUGAndrei Savu

Metrics for Web Applications - Netcamp 2012Andrei Savu

Counters with Riak on Amazon EC2 at HackoverAndrei Savu

Simple REST with DropwizardAndrei Savu

Guava Overview Part 2 Bucharest JUG #2 Andrei Savu

Guava Overview. Part 1 @ Bucharest JUG #1 Andrei Savu

Polyglot Persistence & Big Data in the CloudAndrei Savu

Building a Great Team in Open Source - Open Agile 2011Andrei Savu

Recently uploaded (20)

Asthma presentación en inglés abril 2025 pdfVanessaRaudez

How Can I use the AI Hype in my Business Context?Daniel Lehner

𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨? Everyone’s talking about AI but is anyone really using it to create real value? Most companies want to leverage AI. Few know 𝗵𝗼𝘄. ✅ What exactly should you ask to find real AI opportunities? ✅ Which AI techniques actually fit your business? ✅ Is your data even ready for AI? If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.

Build Your Own Copilot & Agents For DevsBrian McKeiver

Leading AI Innovation As A Product Manager - Michael JidaelMichael Jidael

Unlike traditional product management, AI product leadership requires new mental models, collaborative approaches, and new measurement frameworks. This presentation breaks down how Product Managers can successfully lead AI Innovation in today's rapidly evolving technology landscape. Drawing from practical experience and industry best practices, I shared frameworks, approaches, and mindset shifts essential for product leaders navigating the unique challenges of AI product development. In this deck, you'll discover: - What AI leadership means for product managers - The fundamental paradigm shift required for AI product development. - A framework for identifying high-value AI opportunities for your products. - How to transition from user stories to AI learning loops and hypothesis-driven development. - The essential AI product management framework for defining, developing, and deploying intelligence. - Technical and business metrics that matter in AI product development. - Strategies for effective collaboration with data science and engineering teams. - Framework for handling AI's probabilistic nature and setting stakeholder expectations. - A real-world case study demonstrating these principles in action. - Practical next steps to begin your AI product leadership journey. This presentation is essential for Product Managers, aspiring PMs, product leaders, innovators, and anyone interested in understanding how to successfully build and manage AI-powered products from idea to impact. The key takeaway is that leading AI products is about creating capabilities (intelligence) that continuously improve and deliver increasing value over time.

Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc.

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

Salesforce AI Associate 2 of 2 Certification.docxJosé Enrique López Rivera

#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018Lynda Kane

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.

Network Security. Different aspects of Network Security.gregtap1

Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3

Automation Hour 1/28/2022: Capture User Feedback from AnywhereLynda Kane

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including: -Artificial Intelligence Market Overview -Strategies for AI Adoption in 2025 -Anticipated drivers of AI adoption and transformative technologies -Benefits of AI and Big data for your business -Tips on how to prepare your business for innovation -AI and data privacy: Strategies for securing data privacy in AI models, etc. Download your free copy nowand implement the key findings to improve your business.

Mobile App Development Company in Saudi ArabiaSteve Jonas

EmizenTech is a globally recognized software development company, proudly serving businesses since 2013. With over 11+ years of industry experience and a team of 200+ skilled professionals, we have successfully delivered 1200+ projects across various sectors. As a leading Mobile App Development Company In Saudi Arabia we offer end-to-end solutions for iOS, Android, and cross-platform applications. Our apps are known for their user-friendly interfaces, scalability, high performance, and strong security features. We tailor each mobile application to meet the unique needs of different industries, ensuring a seamless user experience. EmizenTech is committed to turning your vision into a powerful digital product that drives growth, innovation, and long-term success in the competitive mobile landscape of Saudi Arabia.

UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity

Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts! 📕 Agenda Welcome & Introductions Orchestrator API Overview Exploring the Swagger Interface Test Manager API Highlights Streamlining Automation & Testing with APIs (Demo) Q&A and Open Discussion Perfect for developers, testers, and automation enthusiasts! 👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/ This session streamed live on April 29, 2025, 18:00 CET. Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.

tecnologias de las primeras civilizaciones.pdffjgm517

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

Asthma presentación en inglés abril 2025 pdfVanessaRaudez

How Can I use the AI Hype in my Business Context?Daniel Lehner

Build Your Own Copilot & Agents For DevsBrian McKeiver

Leading AI Innovation As A Product Manager - Michael JidaelMichael Jidael

Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc.

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

Salesforce AI Associate 2 of 2 Certification.docxJosé Enrique López Rivera

#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018Lynda Kane

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

Network Security. Different aspects of Network Security.gregtap1

Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3

Automation Hour 1/28/2022: Capture User Feedback from AnywhereLynda Kane

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

Mobile App Development Company in Saudi ArabiaSteve Jonas

UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity

tecnologias de las primeras civilizaciones.pdffjgm517

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

Apache ZooKeeper TechTuesday

1. Apache ZooKeeper Why use it? What to expect in the future? Andrei Savu @TechTuesday

2. Outline Why use it? Crash Course Practical Example What to expect in the future (3.4.0 release)? GSoC 2010 New Contrib Work in Progress

3. Crash Course

4. What is ZooKeeper? A highly available, scalable, distributed, configuration, consensus, group membership, leader election, naming and coordination service.

5. What is ZooKeeper? (2) replicated in memory tree data structure somehow similar to a file system no partial read / writes no renames ordered updates strong persistence guarantees conditional updates (version) watches for data changes ephemeral nodes generated file names

6. ZooKeeper Data Model hierarchical namespace each znode has data and children data is read and written in its entirety

7. Basic ZooKeeper API string create(path, data, acl, flags) delete(path, expected_version) stat set_data(path, data, expected_version) (data, stat) get_data(path, watch) stat exists(path, watch) string[] get_children(path, watch)

8. ZooKeeper Service Facts: 1) all servers store a copy of the data in memory 2) the leader is elected at startup 3) followers respond to clients 4) all updates go through the leader 5) responses are sent when a majority of servers have persisted the change

9. Practical Example

10. Distributed Queue (Python) http://www.cloudera.com/blog/2009/05/building-a- distributed-concurrent-queue-with-apache-zookeeper/ http://github.com/henryr/pyzk-recipes Retry operation on ConnectionLoss: http://github.com/andreisavu/pyzk-recipes

11. GSoC 2010 3 projects / 5 months

12. 1. Monitoring & Web-based interface Status: Committed to the trunk 1. JIRA: ZOOKEEPER-701 2. Progress Tracking Wiki 3. Monitoring for Ganglia, Nagios and Cacti 1. contrib / monitoring 2. 'mntr' 4letter word 4. Web interface available as a Hue application 1. contrib / huebrowser 2. complete install instructions 3. requirements: rest gateway, Hue 1.0

13. 2. Read-Only Mode Status: Under Review (Ready to be committed) 1. JIRA: ZOOKEEPER-704 2. Progress Tracking Wiki 3. Description: "When a ZooKeeper server loses contact with over half of the other servers in an ensemble ('loses a quorum'), it stops responding to client requests. For some applications, it would be beneficial if a server still responded to read requests when the quorum is lost, but caused an error condition when a write request was attempted."

14. 3. Failure Detector Model Status: Under Review 1. JIRA: ZOOKEEPER-702 2. Progress Tracking Wiki 3. Detectors: Phi Accrual, Chen, Bertier, Fixed Heartbeat 4. Why? Check the concluding remarks on the wiki. 5. Conclusion snippet: "in scenarios where we have a changing network behavior, such in a WAN, the adaptive methods can be a good pick"

15. Contrib

16. Large Scale Pub/Sub (hedwig) 1. JIRA: ZOOKEEPER-775 2. Uses ZooKeeper and BookKeeper 3. Committed to the trunk 4. Developed at Yahoo! Research 5. Used for PNUTS cross data center replication 6. http://vimeo.com/13282102

17. Work in Progress only some interesting JIRAs

18. #834 Children for ephemerals JIRA: ZOOKEEPER-834 Allow ephemeral nodes to have children owned by the same session. Useful when publishing status information. No need to do serialization for basic data structures (hash tables) Similar to /proc in *nix systems. Examples: /agent-01/ip, /agent-01/memory, /agent-01/load

19. #829 /zookeeper/sessions/* JIRA: ZOOKEEPER-829 Requested by HBase developers: " we'd like the ability to forcible expire someone else's ZK session "

20. Plenty of bug fixes Join the community!

21. Resources http://hadoop.apache.org/zookeeper/ http://wiki.apache.org/hadoop/ZooKeeper/ProjectDescription http://wiki.apache.org/hadoop/ZooKeeper/Tao

22. Thanks! Questions?

Apache ZooKeeper TechTuesday

Recommended

More Related Content

What's hot (20)

Viewers also liked (16)

Similar to Apache ZooKeeper TechTuesday (20)

More from Andrei Savu (20)

Recently uploaded (20)

Apache ZooKeeper TechTuesday