Building a custom time series db - Colin Hemmings at #DOXLON

Jun 30, 20142 likes1,209 views

Colin talks about how he architected and built a high performance time series database from the ground up at Dataloop.io. Handling hundreds of thousands of metrics per second. One of the objectives was to provide real time graphing and alerting. If you're 'rolling your own' metrics, are interested in Node.JS, highly scalable architectures and like listening to plenty of war stories you should enjoy this talk. Video: http://youtu.be/vx6Ms5TNtqo DevOps Exchange Meetup Group: http://bit.ly/doxlonmeetup

www.dataloop.io | @dataloopio | info@dataloop.io
Colin Hemmings | Architect
Time-series Datastore
on Riak

www.dataloop.io | @dataloopio | info@dataloop.io
•Collection
•Storage
•Analytics
Architecture

www.dataloop.io | @dataloopio | info@dataloop.io
Just stick it in a database, right?
The Storage Problem

www.dataloop.io | @dataloopio | info@dataloop.io
Past Solutions
TempoDB - the phantom menace

www.dataloop.io | @dataloopio | info@dataloop.io
Past Solutions
MongoDB - return of the Jedi

www.dataloop.io | @dataloopio | info@dataloop.io
Riak - Our New Hope
• Scales
• Ops Friendly
• Actually works
• No random JVM crashes here

www.dataloop.io | @dataloopio | info@dataloop.io
Objectives
• Handle the load
• Semi-arbitrary queries
• Data retention windows
• Low latency

www.dataloop.io | @dataloopio | info@dataloop.io
Data structure
• Resolution/rollup based queries
• Minimum 24 hours at 1 second resolution
• Second, minute and hour resolution

www.dataloop.io | @dataloopio | info@dataloop.io
Data structure
• 86,400 data points per resolution
• 1 second -> 24 hour retention
• 1 minute -> 60 day retention
• 1 hour -> 10 year retention

www.dataloop.io | @dataloopio | info@dataloop.io
Data structure
• per metric -> 250k data points
• 1000 metric per host -> 2.5M data points
• 300 hosts per user -> 750M data points
• 1000 customers -> 750B data points!!!!!

www.dataloop.io | @dataloopio | info@dataloop.io
Simple Riak Storage
• Timestamp keyed object per metric value
• 2i and MapReduce are too slow
• Especially across millions of keys
• Writes would soon cripple our Riak cluster

www.dataloop.io | @dataloopio | info@dataloop.io
Intelligent Riak Storage
• Units of storage: time based data blocks
• Compute keys
• Mutable data windows

www.dataloop.io | @dataloopio | info@dataloop.io
Query
Get cpu metrics for host A for period t1-t4 at 1 second resolution
• Pull the correct blocks from riak, based on block boundaries
• GET /buckets/host_a/keys/cpu_second_t1b
• GET /buckets/host_a/keys/cpu_second_t2b
• GET /buckets/host_a/keys/cpu_second_t3b
• GET /buckets/host_a/keys/cpu_second_t4b

www.dataloop.io | @dataloopio | info@dataloop.io
Query
• Filter points outside of our query range
• Aggregate all the data points
• Perform other operation if more complex query

www.dataloop.io | @dataloopio | info@dataloop.io
Expiring
• Cleanup worker
• Removes keys out of retention window
• Host keyed, easier to clear all hosts or account data

www.dataloop.io | @dataloopio | info@dataloop.io
Our cluster
• Riak 2.0
• 5 nodes on LevelDB
• Each 2 x 500GB striped SSDs
• Average 1ms GET and PUT latencies

www.dataloop.io | @dataloopio | info@dataloop.io

www.dataloop.io | @dataloopio | info@dataloop.io
Comments
• Awesome, especially for ops
• A bit more work in application tier
• Always compute keys avoid 2i and MapReduce
• Looking forward to using the new data types

Follow on from Back to Basics: An Introduction to NoSQL and MongoDB •Covers more advanced topics: Storage Engines • What storage engines are and how to pick them Aggregation Framework • How to deploy advanced analytics processing right inside the database The BI Connector • How to create visualizations and dashboards from your MongoDB data Authentication and Authorisation • How to secure MongoDB, both on-premise and in the cloud

Presto Summit 2018 - 09 - Netflix Icebergkbajda

Apache Flink Adoption at ShopifyYaroslav Tkachenko

At the beginning of 2021, Shopify Data Platform decided to adopt Apache Flink to enable modern stateful stream-processing. Shopify had a lot of experience with other streaming technologies, but Flink was a great fit due to its state management primitives. After about six months, Shopify now has a flourishing ecosystem of tools, tens of prototypes from many teams across the company and a few large use-cases in production. Yaroslav will share a story about not just building a single data pipeline but building a sustainable ecosystem. You can learn about how they planned their platform roadmap, the tools and libraries Shopify built, the decision to fork Flink, and how Shopify partnered with other teams and drove the adoption of streaming at the company.

Challenges in Building a Data PipelineManish Kumar

The document discusses challenges in building a data pipeline including making it highly scalable, available with low latency and zero data loss while supporting multiple data sources. It covers expectations around real-time vs batch processing and streaming vs batch data. Implementation approaches like ETL vs ELT are examined along with replication modes, challenges around schema changes and NoSQL. Effective implementations should address transformations, security, replays, monitoring and more. Reference architectures like Lambda and Kappa are briefly outlined.

Presto Summit 2018 - 04 - Netflix Containerskbajda

Iceberg: A modern table format for big data (Strata NY 2018)Ryan Blue

Hive tables are an integral part of the big data ecosystem, but the simple directory-based design that made them ubiquitous is increasingly problematic. Netflix uses tables backed by S3 that, like other object stores, don’t fit this directory-based model: listings are much slower, renames are not atomic, and results are eventually consistent. Even tables in HDFS are problematic at scale, and reliable query behavior requires readers to acquire locks and wait. Owen O’Malley and Ryan Blue offer an overview of Iceberg, a new open source project that defines a new table layout addresses the challenges of current Hive tables, with properties specifically designed for cloud object stores, such as S3. Iceberg is an Apache-licensed open source project. It specifies the portable table format and standardizes many important features, including: * All reads use snapshot isolation without locking. * No directory listings are required for query planning. * Files can be added, removed, or replaced atomically. * Full schema evolution supports changes in the table over time. * Partitioning evolution enables changes to the physical layout without breaking existing queries. * Data files are stored as Avro, ORC, or Parquet. * Support for Spark, Pig, and Presto.

Presto Summit 2018 - 01 - Facebook Prestokbajda

Presto is a distributed SQL query engine used at Facebook for analytics workloads like warehouse ETL, dashboards, and A/B testing. Over the past year there have been improvements to workload management, security, execution performance, and language features. Upcoming work includes improving scalability, adding more connectors, and advanced query planning techniques like dynamic filtering. Future projects aim to enhance failure recovery, temporary tables, and introduce user-defined functions.

Presto Summit 2018 - 03 - Starburst CBOkbajda

Introduction to Data Engineer and Data Pipeline at Credit OKKriangkrai Chaonithi

The document discusses the role of data engineers and data pipelines. It begins with an introduction to big data and why data volumes are increasing. It then covers what data engineers do, including building data architectures, working with cloud infrastructure, and programming for data ingestion, transformation, and loading. The document also explains data pipelines, describing extract, transform, load (ETL) processes and batch versus streaming data. It provides an example of Credit OK's data pipeline architecture on Google Cloud Platform that extracts raw data from various sources, cleanses and loads it into BigQuery, then distributes processed data to various applications. It emphasizes the importance of data engineers in processing and managing large, complex data sets.

Presto@Netflix Presto Meetup 03-19-15Zhenxiao Luo

Presto is used at Netflix for interactive queries against their 10PB data warehouse stored in S3. Some key points: - Presto was chosen for its open source nature, speed, scalability on AWS, and integration with Hadoop. - Netflix contributes to Presto's development, including improvements to S3 support and Parquet integration. - Current work includes optimizations like vectorized reading and predicate pushdown. Integration with BI tools and monitoring systems is also a focus. - Future work includes better resource management, support for additional data types, and techniques for handling large joins.

Iceberg: a fast table format for S3DataWorks Summit

Netflix’s Big Data Platform team manages data warehouse in Amazon S3 with over 60 petabytes of data and writes hundreds of terabytes of data every day. With a data warehouse at this scale, it is a constant challenge to keep improving performance. This talk will focus on Iceberg, a new table metadata format that is designed for managing huge tables backed by S3 storage. Iceberg decreases job planning time from minutes to under a second, while also isolating reads from writes to guarantee jobs always use consistent table snapshots. In this session, you'll learn: • Some background about big data at Netflix • Why Iceberg is needed and the drawbacks of the current tables used by Spark and Hive • How Iceberg maintains table metadata to make queries fast and reliable • The benefits of Iceberg's design and how it is changing the way Netflix manages its data warehouse • How you can get started using Iceberg Speaker Ryan Blue, Software Engineer, Netflix

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...HostedbyConfluent

If a real-time dashboard takes 5 minutes to refresh, it’s not real-time. With data lakes increasingly enabling massive amounts of unprocessed data sets, delivering low-latency analytics is not for the faint-hearted. Learn how to stream massive amounts of data which used to be impossible to handle from Kafka, to serve real-time applications using lake-scale optimized approaches to storage and indexing.

An Intro to Elasticsearch and KibanaObjectRocket

TiDB IntroductionMorgan Tocker

This document provides an overview and summary of TiDB, an open-source distributed SQL database compatible with MySQL. It discusses TiDB's architecture which includes TiDB for the SQL layer, TiKV for storage, and PD for placement driving. TiDB provides features like horizontal scalability, distributed transactions, and high availability. Example use cases are also presented, like Mobike's use of TiDB for locking/unlocking bikes and real-time analytics of bike usage data across 200 cities in China.

Presto Summit 2018 - 02 - LinkedInkbajda

Building an open data platform with apache icebergAlluxio, Inc.

Presto@UberZhenxiao Luo

Presto is Uber's distributed SQL query engine for their Hadoop data warehouse. Some key points: - Presto allows interactive SQL queries directly on Uber's petabyte-scale Hadoop data lake without needing to first load the data into another database. - It provides fast performance at scale by leveraging columnar data formats like Parquet and optimizing for distributed execution across many nodes. - Uber deployed a 200 node Presto cluster that handles 30,000 queries per day, serving both ad hoc queries and real-time applications accessing data in Hadoop and improving on the performance of alternative solutions like Hive.

A Walkthrough of InfluxCloud 2.0 by Tim HallInfluxData

Key Challenges in Cloud Computing and How Yahoo! is Approaching ThemYahoo Developer Network

Presto meetup 2015-03-19 @FacebookTreasure Data, Inc.

This document provides an overview of Presto as a Service in Treasure Data, including how Treasure Data deploys and monitors Presto. Key points include: - Treasure Data offers Presto as an interactive query engine accessible through its API and web console. - Treasure Data uses blue-green deployments and a private Maven repository to deploy new Presto versions with no downtime. - Treasure Data monitors Presto using its REST API and collects query logs to analyze performance and detect anomalies. - Treasure Data implements multi-tenancy in Presto by allocating resources like worker nodes based on customers' price plans and resource usage.

Presto Summit 2018 - 07 - Lyftkbajda

Presto talk @ Global AI conference 2018 Bostonkbajda

Presented at Global AI Conference in Boston 2018: http://www.globalbigdataconference.com/boston/global-artificial-intelligence-conference-106/speaker-details/kamil-bajda-pawlikowski-62952.html Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Facebook, Airbnb, Netflix, Uber, Twitter, LinkedIn, Bloomberg, and FINRA, Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments in the last few years. Presto is really a SQL-on-Anything engine in a single query can access data from Hadoop, S3-compatible object stores, RDBMS, NoSQL and custom data stores. This talk will cover some of the best use cases for Presto, recent advancements in the project such as Cost-Based Optimizer and Geospatial functions as well as discuss the roadmap going forward.

Bleeding Edge DatabasesLynn Langit

Unifying Frontend and Backend Development with Scala - ScalaCon 2021Taro L. Saito

Scala can be used for developing both frontend (Scala.js) and backend (Scala JVM) applications. A missing piece has been bridging these two worlds using Scala. We built Airframe RPC, a framework that uses Scala traits as a unified RPC interface between servers and clients. With Airframe RPC, you can build HTTP/1 (Finagle) and HTTP/2 (gRPC) services just by defining Scala traits and case classes. It simplifies web application design as you only need to care about Scala interfaces without using existing web standards like REST, ProtocolBuffers, OpenAPI, etc. Scala.js support of Airframe also enables building interactive Web applications that can dynamically render DOM elements while talking with Scala-based RPC servers. With Airframe RPC, the value of Scala developers will be much higher both for frontend and backend areas.

Setting Up InfluxDB for IoT by David G SimmonsInfluxData

InfluxData Internals by Ryan BettsInfluxData

InfluxData builds a time series platform primarily deployed for DevOps and IoT monitoring. This talk presents several lessons learned while scaling the platform across a large number of deployments—from single server open source instances to highly available high-throughput clusters. This talk presents a number of failure conditions that informed subsequent design choices. Ryan Betts (Director of Engineering at InfluxData) will discuss designing backpressure in an AP system with tens of thousands of resource-limited writers; trade-offs between monolithic and service-oriented database implementations; and lessons learned implementing multiple query processing systems.

InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxData

Convergent Replicated Data Types in Riak 2.0Big Data Spain

Talk by Gordon Guthrie, Senior Software Engineer at Basho Summary A review of the CAP Theorem and the difficulties of resolving conflicts in highly distributed systems. Covering the issues and various theories on how to resolve including the use CRDTs in Riak Details CRDTs are used to replicate data across multiple computers in a network, executing updates without the need for remote synchronisation. This leads to merge conflicts in systems using conventional eventual consistency technology, but CRDTs are designed such that conflicts are mathematically impossible. Under the constraints of the CAP theorem they provide the strongest consistency guarantees for available/partition-tolerant (AP) settings. The CRDT concept was first formally defined in 2007 by Marc Shapiro and Nuno Preguiça in terms of operation commutativity, and development was initially motivated by collaborative text editing. The concept of semilattice evolution of replicated states was first defined by Baquero and Moura in 1997, and development was initially motivated by mobile computing. The two concepts were later unified in 2011. Basho has worked with the EU and Marc Shapiro's team to push CRDTs into distributed systems. Riak v2.x is the first commercial product to include this functionality

Getting started with Riak in the CloudInes Sombra

Getting started with Riak in the Cloud involves provisioning a Riak cluster on Engine Yard and optimizing it for performance. Key steps include choosing instance types like m1.large or m1.xlarge that are EBS-optimized, having at least 5 nodes, setting the ring size to 256, disabling swap, using the Bitcask backend, enabling kernel optimizations, and monitoring and backing up the cluster. Benchmarks show best performance from high I/O instance types like hi1.4xlarge that use SSDs rather than EBS storage.

More Related Content

What's hot (20)

Presto Summit 2018 - 03 - Starburst CBOkbajda

Introduction to Data Engineer and Data Pipeline at Credit OKKriangkrai Chaonithi

Presto@Netflix Presto Meetup 03-19-15Zhenxiao Luo

Iceberg: a fast table format for S3DataWorks Summit

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...HostedbyConfluent

An Intro to Elasticsearch and KibanaObjectRocket

TiDB IntroductionMorgan Tocker

Presto Summit 2018 - 02 - LinkedInkbajda

Building an open data platform with apache icebergAlluxio, Inc.

Presto@UberZhenxiao Luo

A Walkthrough of InfluxCloud 2.0 by Tim HallInfluxData

Key Challenges in Cloud Computing and How Yahoo! is Approaching ThemYahoo Developer Network

Presto meetup 2015-03-19 @FacebookTreasure Data, Inc.

Presto Summit 2018 - 07 - Lyftkbajda

Presto talk @ Global AI conference 2018 Bostonkbajda

Bleeding Edge DatabasesLynn Langit

Unifying Frontend and Backend Development with Scala - ScalaCon 2021Taro L. Saito

Setting Up InfluxDB for IoT by David G SimmonsInfluxData

InfluxData Internals by Ryan BettsInfluxData

InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxData

Presto Summit 2018 - 03 - Starburst CBOkbajda

Introduction to Data Engineer and Data Pipeline at Credit OKKriangkrai Chaonithi

Presto@Netflix Presto Meetup 03-19-15Zhenxiao Luo

Iceberg: a fast table format for S3DataWorks Summit

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...HostedbyConfluent

An Intro to Elasticsearch and KibanaObjectRocket

TiDB IntroductionMorgan Tocker

Presto Summit 2018 - 02 - LinkedInkbajda

Building an open data platform with apache icebergAlluxio, Inc.

Presto@UberZhenxiao Luo

A Walkthrough of InfluxCloud 2.0 by Tim HallInfluxData

Key Challenges in Cloud Computing and How Yahoo! is Approaching ThemYahoo Developer Network

Presto meetup 2015-03-19 @FacebookTreasure Data, Inc.

Presto Summit 2018 - 07 - Lyftkbajda

Presto talk @ Global AI conference 2018 Bostonkbajda

Bleeding Edge DatabasesLynn Langit

Unifying Frontend and Backend Development with Scala - ScalaCon 2021Taro L. Saito

Setting Up InfluxDB for IoT by David G SimmonsInfluxData

InfluxData Internals by Ryan BettsInfluxData

InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxData

Similar to Building a custom time series db - Colin Hemmings at #DOXLON (20)

Convergent Replicated Data Types in Riak 2.0Big Data Spain

Getting started with Riak in the CloudInes Sombra

How Shopify Scales Railsjduff

Near Real-Time Analytics with Apache Spark: Ingestion, ETL, and Interactive Q...Databricks

Near real-time analytics has become a common requirement for many data teams as the technology has caught up to the demand. One of the hardest aspects of enabling near-realtime analytics is making sure the source data is ingested and deduplicated often enough to be useful to analysts while writing the data in a format that is usable by your analytics query engine. This is usually the domain of many tools since there are three different aspects of the problem: streaming ingestion of data, deduplication using an ETL process, and interactive analytics. With Spark, this can be done with one tool. This talk with walk you through how to use Spark Streaming to ingest change-log data, use Spark batch jobs to perform major and minor compaction, and query the results with Spark.SQL. At the end of this talk you will know what is required to setup near-realtime analytics at your organization, the common gotchas including file formats and distributed file systems, and how to handle data the unique data integrity issues that arise from near-realtime analytics.

Is hadoop for youGwen (Chen) Shapira

1) Hadoop is well-suited for organizations that have large amounts of non-relational or unstructured data from sources like logs, sensor data, or social media. It allows for the distributed storage and parallel processing of such large datasets across clusters of commodity hardware. 2) Hadoop uses the Hadoop Distributed File System (HDFS) to reliably store large files across nodes in a cluster and allows for the parallel processing of data using the MapReduce programming model. This architecture provides benefits like scalability, flexibility, reliability, and low costs compared to traditional database solutions. 3) To get started with Hadoop, organizations should run some initial proof-of-concept projects using freely available cloud resources

How does Riak compare to Cassandra? [Cassandra London User Group July 2011]Rainforest QA

Using Riak for Events storage and analysis at Booking.comDamien Krotkine

At Booking.com, we have a constant flow of events coming from various applications and internal subsystems. This critical data needs to be stored for real-time, medium and long term analysis. Events are schema-less, making it difficult to use standard analysis tools.This presentation will explain how we built a storage and analysis solution based on Riak. The talk will cover: data aggregation and serialization, Riak configuration, solutions for lowering the network usage, and finally, how Riak's advanced features are used to perform real-time data crunching on the cluster nodes.

Chirp 2010: Scaling TwitterJohn Adams

This document summarizes a keynote speech given by John Adams, an early Twitter engineer, about scaling Twitter operations from 2008-2009. Some key points: 1) Twitter saw exponential growth rates from 2008-2009, processing over 55 million tweets per day and 600 million searches per day. 2) Operations focused on improving performance, reducing errors and outages, and using metrics to identify weaknesses and bottlenecks like network latency and database delays. 3) Technologies like Unicorn, memcached, Flock, Cassandra, and daemons were implemented to improve scalability beyond a traditional RDBMS and handle Twitter's data volumes and real-time needs. 4) Caching,

EM12c: Capacity Planning with OEM MetricsMaaz Anjum

DrupalSouth 2015 - Performance: Not an AfterthoughtNick Santamaria

MongoDB Scalability Best PracticesJason Terpko

PrestoChen Chun

Presto is an open source distributed SQL query engine for running queries against large datasets stored in Hadoop/HDFS clusters. It uses in-memory parallel processing, pipelining, data locality, caching, and dynamic compilation to byte code for low query latency. Key techniques include caching frequently used metadata and compiled plans, processing data locally on nodes where it resides, and controlling garbage collection to optimize native code generation. Presto has been tested on TPC-H benchmarks and is used at Meituan to query their 300+PB dataset across Hadoop clusters.

Riak seattle-meetup-augustpharkmillups

This document summarizes the open source Riak database from Basho Technologies. Riak is a distributed, fault-tolerant key-value database written in Erlang. It provides simple, elegant scalability through a gossip protocol and consistent hashing to distribute data across nodes. Riak aims for ease of operations, resiliency to failures, and easy development through RESTful APIs and client libraries. It is used by both startups and Fortune 100 companies for applications such as session storage, timelines, and full-text search.

Suburface 2021 IBM Cloud Data LakeTorsten Steinbach

Serverless Cloud Data Lake with Spark for Serving Weather Data 1) The document discusses using a serverless architecture with IBM Cloud services like SQL Query powered by Spark, Cloud Object Storage, and Cloud Functions to build a cost-effective cloud data lake for serving historical weather data on demand. 2) It describes how data skipping techniques and geospatial indexes in SQL Query can accelerate queries by an order of magnitude by pruning irrelevant data. 3) The new serverless solution provides unlimited storage, global coverage, and supports large queries for machine learning and analytics at an order of magnitude lower cost than the previous implementation.

Operational-AnalyticsNiloy Mukherjee

Planet-scale Data Ingestion Pipeline: BigdamSATOSHI TAGOMORI

Bigdam is a planet-scale data ingestion pipeline designed for large-scale data ingestion. It addresses issues with the traditional pipeline such as imperfectqueue throughput limitations, latency in queries from event collectors, difficulty maintaining event collector code, many small temporary and imported files. The redesigned pipeline includes Bigdam-Gateway for HTTP endpoints, Bigdam-Pool for distributed buffer storage, Bigdam-Scheduler to schedule import tasks, Bigdam-Queue as a high throughput queue, and Bigdam-Import for data conversion and import. Consistency is ensured through at-least-once design and deduplication is performed at the end of the pipeline for simplicity and reliability. Components are designed to scale out horizontally.

Webinar: The Future of SQLCrate.io

NoSQL databases like MongoDB, Elasticsearch, and Cassandra are synonymous with scalability, search, and developer agility. But there’s a downside...having to give up the ease and comfort of SQL. Or do you? Join this webcast to learn how the newest databases, like CrateDB and CockroachDB deliver the benefits of NoSQL with the ease of SQL by building SQL engines on top of custom NoSQL technology stacks. Database industry veteran Andy Ellicott, who helped launch Vertica, VoltDB, Cloudant, and now with Crate.io, will provide a no-BS view of current DBMS architectures and predictions for the future of data. If you’re a DBMS user, this webcast will help you make sense of a very crowded DBMS market and make better-informed decisions for your new tech stacks.

Practical Medium Data Analytics with Python (10 Things I Hate About pandas, P...Wes McKinney

This document discusses pandas, a popular Python library for data analysis, and its limitations. It introduces Badger, a new project from DataPad that aims to address some of pandas' shortcomings like slow performance on large datasets and lack of tight database integration. The creator describes Badger as using compressed columnar storage, immutable data structures, and C kernels to perform analytics queries much faster than pandas or databases on benchmark tests of a multi-million row dataset. He envisions Badger becoming a distributed, multicore analytics platform that can also be used for ETL jobs.

Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data StreamingMichael Rainey

This document provides an overview and summary of a presentation on integrating Oracle GoldenGate and Apache Kafka for real-time data streaming. It introduces the speaker, describes Rittman Mead as a specialist in Oracle data integration and analytics, and outlines the challenges of integrating new data sources. The bulk of the document then dives into a step-by-step example of using GoldenGate to replicate transactional data from an Oracle database to Kafka in real-time via Kafka's publish-subscribe capabilities.

Performance Tuning RocksDB for Kafka Streams’ State Storesconfluent

Convergent Replicated Data Types in Riak 2.0Big Data Spain

Getting started with Riak in the CloudInes Sombra

How Shopify Scales Railsjduff

Near Real-Time Analytics with Apache Spark: Ingestion, ETL, and Interactive Q...Databricks

Is hadoop for youGwen (Chen) Shapira

How does Riak compare to Cassandra? [Cassandra London User Group July 2011]Rainforest QA

Using Riak for Events storage and analysis at Booking.comDamien Krotkine

Chirp 2010: Scaling TwitterJohn Adams

EM12c: Capacity Planning with OEM MetricsMaaz Anjum

DrupalSouth 2015 - Performance: Not an AfterthoughtNick Santamaria

MongoDB Scalability Best PracticesJason Terpko

PrestoChen Chun

Riak seattle-meetup-augustpharkmillups

Suburface 2021 IBM Cloud Data LakeTorsten Steinbach

Operational-AnalyticsNiloy Mukherjee

Planet-scale Data Ingestion Pipeline: BigdamSATOSHI TAGOMORI

Webinar: The Future of SQLCrate.io

Practical Medium Data Analytics with Python (10 Things I Hate About pandas, P...Wes McKinney

Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data StreamingMichael Rainey

Performance Tuning RocksDB for Kafka Streams’ State Storesconfluent

More from Outlyer (20)

Murat Karslioglu, VP Solutions @ OpenEBS - Containerized storage for containe...Outlyer

1) OpenEBS provides container-attached storage that aims to make storage for containerized applications as containerized as the applications themselves. 2) It does this by running storage services within Kubernetes as microservices, allowing each application pod to have its own dedicated storage with policies for data protection, availability and performance. 3) This approach aims to eliminate many of the concerns of traditional distributed storage systems while providing the benefits of DAS with increased utilization, mobility and cloud independence.

How & When to Feature FlagOutlyer

Feature flags are a valuable DevOps technique to deliver better, more reliable software faster. Feature flags can be used for both release management (dark launches, canary rollouts, betas) as well as long term control (entitlement management, user segmentation personalization). However, if not managed properly, feature flags can be very destructive technical debt. Feature flags need to be managed properly with visibility and control to both engineering and business users.

Why You Need to Stop Using "The" Staging ServerOutlyer

Old staging methodology is broken for modern development. In fact, the staging server is left over from when we built monolithic applications. Find out why microservice architectures are driving ephemeral testing environments & why every sized dev shop should deliver true continuous deployment. Staging servers slow down development with merge conflicts, slow iteration loops, and manhour intensive processes. To build better software faster containers and infrastructure as code are key in 2017. Dev Ops professionals miss this talk at their own peril.

How GitHub combined with CI empowers rapid product delivery at Credit Karma Outlyer

Packaging Services with NixOutlyer

Minimum Viable Docker: our journey towards orchestrationOutlyer

Ops is dead. long live ops.Outlyer

Ops is the past! DevOps is the present ! SRE is for giants! NoOps is the future! Fowler even says that a DevOps Engineer is an anti-pattern! So will our job disappear in 10 years? What can we do about it? What is the next set of skills that we need? A startup is often a precursor to larger changes. I'll tell you what we are trying to do at Curve, a Fintech startup where developers build Kubernetes clusters and the SRE team codes microservices.

The service mesh: resilient communication for microservice applicationsOutlyer

Modern application architecture is shifting from monolith to microservices: componentized, containerized, and orchestrated with systems like Kubernetes, Mesos, and Docker Swarm. While this environment is resilient to many failures of both hardware and software, applications require more than this to be truly resilient. In this talk, we introduce the notion of a "service mesh": a userspace infrastructure layer designed to manage service-to-service communication in microservice applications, including handling partial failures and unexpected load, while reducing tail latencies and degrading gracefully in the presence of component failure.

Microservices: Why We Did It (and should you?) Outlyer

Renan Dias: Using Alexa to deploy applications to KubernetesOutlyer

It's time to bring voice commands into continuous deployment pipelines. In this talk, Renan will walk you through the steps of setting up a powerful and cutting-edge continuous deployment pipeline, which will allow you to deploy your products to Kubernetes clusters using just your voice. "Alexa, deploy API to production". If you have never imagined yourself doing that, or you have but don't know where to start, this talk is definitely for you.

Alex Dias: how to build a docker monitoring solution Outlyer

How to build a container monitoring solution - David Gildeh, CEO and Co-Found...Outlyer

This document discusses monitoring microservices running in containers at scale. Some key points: 1) Containers introduce additional complexity for monitoring due to their dynamic nature - the average host runs 8 containers and each container only runs for 2 days on average. 2) Traditional VM-based monitoring agents can't monitor inside containers so monitoring must be done from outside the containers, like monitoring remote hosts. 3) Orchestration platforms like Kubernetes add labels to collected metrics to provide context on nodes, pods, services and more. 4) The large number of short-lived containers results in high metric series churn that must be addressed to avoid exploding storage requirements. Dimensional labeling and partitioning metrics by time help

Heresy in the church of - Corey Quinn, Principal at The Quinn Advisory Group Outlyer

Docker (and by extension, microservices based architecture) has expanded our horizons with respect to how the industry builds and supports applications at scale. It’s changed the way we think about our code, what production looks like, and how we live. But in our rush to embrace this exciting new paradigm, are we throwing away the lessons of the past? In this entertaining and somewhat irreverent talk, Corey presents the ”other side” of the containerization craze: how configuration management fits into a world consumed by the Docker Docker Docker madness, how ”containers all the way down” can let you down when you least expect it, and how promising technologies should perhaps be vetted a bit more thoroughly before you try to run critical services on top of them.

Anatomy of a real-life incident -Alex Solomon, CTO and Co-Founder of PagerDutyOutlyer

Major incidents can be very stressful, frustrating and chaotic experiences, especially if the on-call responders lack the proper process, training and coordination. In this talk, we will walk through a real incident from PagerDuty’s own history, to illustrate what an effective incident response looks like. We will recreate the incident timeline step by step and go over all of the different roles involved, including the incident commander, scribe, customer/business liaison and subject matter experts. We will also cover the process and tooling needed to respond quickly and effectively to major incidents in order to minimize customer and business impact.

A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering...Outlyer

The Network Knows—Avi Freedman, CEO & Co-Founder of Kentik Outlyer

Building a production-ready, fully-scalable Docker Swarm using Terraform & Pa...Outlyer

Zero Downtime Postgres UpgradesOutlyer

On the surface, the tech behind a payments API may look like any other startup’s. You'll probably find some Rails apps, a database, and a bunch of stuff off to the sides to glue it together. GoCardless found it's mostly not the tech that differs, but the approach. Using their high-availability Postgres cluster as a running example, they explore how reliability became so important to them, and dive into the most recent feature they built into the cluster: zero-downtime patch upgrades.

DOXLON November 2016: Facebook Engineering on cgroupv2Outlyer

DOXLON November 2016 - ELK Stack and Beats Outlyer