Sharding - patterns & antipatterns, Константин Осипов, Алексей Рыбак

Nov 10, 2014Download as PPTX, PDF2 likes4,160 views

This document discusses sharding patterns and antipatterns for scalable databases. It covers selecting good shard keys like user IDs, routing types like using smart clients or proxies, and approaches for re-sharding like moving data instead of redistributing it. The key topics are sharding functions, routing, and re-sharding strategies to minimize disruption when updating shard configurations.

Sharding: patterns and
antipatterns
Konstantin Osipov (Mail.Ru, Tarantool)
Alexey Rybak (Badoo)

Big picture: scalable databases
● replication
● sharding and re-sharding
● distributed queries & jobs, Map/Reduce
● DDL
● will focus on sharding/re-sharding only

Contents
I. sharding function
II. routing
III.re-sharding

Selecting a good shard key
● the identified object
should be small
● some data you won’t be
able to shard (and have to
duplicate in each shard)
● don’t store the key if you
don’t have to

Good and bad shard keys
● good: user session, shopping order
● maybe: user (if user data isn’t too thick)
● bad: inventory item, order date

Garage sharding: numbers
● replication based doubling (2, 4, 8, out of
cash)
● the magic number 48 (2✕3✕4)

Garage sharding thru hashing
● good: remainders
o f(key) ≡ key % n_srv
o f(key) ≡ crc32(key) % n_srv
● bad: first login letter

Sharding for grown-ups
● table function
● consistent hashing

Table functions
● virtual buckets: key -> bucket -> shard
o “key -> bucket” function, “bucket -> shard” table
o “key -> bucket” table, “bucket -> shard” table

Consistent hashing
● Danny Lewin RIP
● Kinda ring and like...
uhm... points, you
know ...
● Libraries: Ketama

Guava/Sumbur
● f(key, n_servers) => server_id
● strictly uniform key-to-server mapping
● recurrence formula (15 lines of code)

Routing types
● smart client
● coordinator
● proxy
● local proxy on every app server
● intra-database routing

Smart Client
● no extra hops
● all clients
(PHP/Python/C...)
should implement
it
● resharding is hard

Proxy
● encapsulates routing logic
● extra hop, traffic
● +1 service
● SPOF
=> local proxy

Coordinator
● centralized
knowledge
● SPOF

Intra-database routing
● too many nodes
● redundancy is high
● ad-hoc requests

Re-sharding is a pain
● redistribution impacts:
o clients
o network performance
o consistency
=> maintenance time window
● forget about it on petabyte scale

Best practice: no data redistribution
● update is a move
● data expiration (new data on new servers)
● new data on selected servers

DDL
● upgrade your app
● upgrade your database
● update your app and remove any trace of old
schema

Thank you! Questions?
kostja@tarantool.org
fisher@corp.badoo.com

This 8-hours tutorial was given at various conferences including Percona conference (London), DevConf (Moscow), Highload++ (Moscow). ABSTRACT During this tutorial we will cover various topics related to high scalability for the LAMP stack. This workshop is divided into three sections. The first section covers basic principles of shared nothing architectures and horizontal scaling for the app//cache/database tiers. Section two of this tutorial is devoted to MySQL sharding techniques, queues and a few performance-related tips and tricks. In section three we will cover the practical approach for measuring site performance and quality, porviding a "lean" support philosophy, connecting buesiness and technology metrics. In addition we will cover a very useful Pinba real-time statistical server, it's features and various use cases. All of the sections will be based on real-world examples built in Badoo, one of the biggest dating sites on the Internet.

The immutable database datomicLaurence Chen

The document discusses problems with traditional mutable databases like losing history data when records are updated and the inability to retrieve past states. It introduces Datomic as an immutable database that can retrieve past states using its "as-of" operator similar to a git checkout. It also explains that Datomic improves scalability by running queries on clients instead of the server and uses transactions through data forms instead of SQL to prevent injection attacks. Workarounds without Datomic include using as-of in SQL 2011, Kafka with CQRS/event sourcing, or wrapping SQL in APIs. Overall, Datomic provides lessons on separating reaction from perception and solving impedance mismatch through an alternative approach of decomposing databases.

Mr hadoop seedrocketSeedRocket

MapReduce is a programming model for processing large datasets in a distributed manner. It involves splitting the data into chunks which are processed in parallel by map tasks, and then combining the outputs of those maps via reduce tasks. Hadoop is an open-source software framework that allows distributed processing of large datasets across clusters of computers using MapReduce. It works by distributing data storage across nodes as a filesystem, and distributing computations as MapReduce jobs across clusters. Hadoop provides reliable storage and parallel processing of large datasets in a distributed environment.

Dynamo db and Cross Region MigrationAnamika Gupta

DB reading group may 16, 2018Keisuke Suzuki

No sqlBhanu Sekhar

Amazon DynamoDB is a scalable and distributed NoSQL database that scales horizontally and seamlessly over hundreds of servers. It has flexible schemas and automatic data replication but has limitations on row and query sizes. HBase is an open-source, distributed, scalable big data store that provides real-time random access to big data and is highly scalable and reliable but has single points of failure. MongoDB is a scalable, schema-less document database ideal for flexible schemas with easy horizontal scaling and high availability but has higher data sizes than relational databases and less querying flexibility. Redis is an in-memory data structure store with no query language and offers fast data access but lacks security and recoverability of relational databases.

Hadoop @ eBuddyBennie Schut

The document discusses eBuddy's use of Hadoop for data warehousing and analytics needs. It describes eBuddy's logging history and current large volume of login records. It outlines the needs of different teams for analysis. It then reviews eBuddy's timeline of using traditional RDBMS, custom MapReduce code, Pig scripts and migrating to Hive for SQL queries. The document discusses their current use of Hive, ETL processes, future plans to use Cassandra and streaming ETL to Flume and questions about Hive partition pruning and left outer joins in Pig.

Geo data analyticsDaniel Marcous

TriHUG 3/14: HBase in Productiontrihug

This document discusses Bronto's use of HBase for their marketing platform. Some key points: - Bronto uses HBase for high volume scenarios, realtime data access, batch processing, and as a staging area for HDFS. - HBase tables at Bronto are designed with the read/write patterns and necessary queries in mind. Row keys and column families are structured to optimize for these access patterns. - Operations of HBase at scale require tuning of JVM settings, monitoring tools, and custom scripts to handle compactions and prevent cascading failures during high load. Table design also impacts operations and needs to account for expected workloads.

DemonstrationSean Murphy

This document outlines a demonstration of a Cassandra and Hadoop cluster configured on Amazon EC2 instances. It will show launching the cluster nodes, configuring Cassandra, and interacting with the Cassandra database. It will also demonstrate using Hadoop and Pig to perform analytics on real user check-in data imported from the Gowalla dataset. The data models used to store the user, location, check-in, and friendship relationship data in Cassandra will also be explained.

openTSDB - Metrics for a distributed worldOliver Hankeln

Caching inRichardWarburton

Modern computationally intensive tasks are rarely bottlenecked by the absolute performance of your processor cores. The real bottleneck in 2013 is getting data out of memory. CPU caches are designed to alleviate the difference in performance between CPU core clock speed and main memory clock speed, but developers rarely understand how this interaction works or how to measure or tune their application accordingly. This session aims to address this by: • Describing how CPU caches work in the latest Intel hardware • Showing what and how to measure in order to understand the caching behavior of software • Giving examples of how this affects Java program performance and what can be done to address poor cache utilization

Pain points with M3, some things to address them and how replication worksRob Skillington

This document discusses some pain points with running the M3 time series database in a clustered configuration for metrics collection. It notes that clustering takes significant work and learning. Key challenges include long bootstrap times for large instances, difficulties controlling and monitoring memory usage, and harder implementation of downsampling in a central model versus a sidecar model. The document provides updates on work being done to address these issues through improvements to bootstrapping, memory management, and downsampling support.

Apache Flink Training Workshop @ HadoopCon2016 - #2 DataSet API Hands-OnApache Flink Taiwan User Group

FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...Rob Skillington

Rob Skillington gave a presentation on observability and M3, Uber's open source time series database. Some key points: - M3 was created at Uber to handle high dimensionality metrics at massive scale, storing over 11 billion unique time series. - It uses techniques like Roaring Bitmaps to efficiently store and query metrics with many dimensions or tag values. - M3 can ingest metrics from Prometheus and Graphite, storing over 33 million metrics per second while powering dashboards and 150,000 alerts. - The open source M3DB component can run standalone or on Kubernetes, providing a scalable time series storage solution for complex monitoring needs.

OpenTSDB: HBaseCon2017HBaseCon

Another year, another talk about OpenTSDB running on HBase. We'll discuss topics like: Yahoo's append co-processor saving CPU resources by resolving atomic appends at compaction or query time. The pros and cons of HBASE-15181, Date Tiered compaction for time series data. Yahoo's experiments with unbounded secondary index on HBase. OpenTSDB's 3.0 featuring a new query engine and API. by Chris Larsen of Yahoo!

OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...NETWAYS

At Uber we use high cardinality monitoring to observe and detect issues with our 4,000 microservices running on Mesos and across our infrastructure systems and servers. We’ll cover how we put the resulting 6 billion plus time series to work in a variety of different ways, auto-discovering services and their usage of other systems at Uber, setting up and tearing down alerts automatically for services, sending smart alert notifications that rollup different failures into individual high level contextual alerts, and more. We’ll also talk about how we accomplish all this with a global view of our systems with M3, our open source metrics platform. We’ll take a deep dive look at how we use M3DB, now available as an open source Prometheus long term storage backend, to horizontally scale our metrics platform in a cost efficient manner with a system that’s still sane to operate with petabytes of metrics data.

Data Lessons Learned at ScaleCharlie Reverte

Charlie Reverte, VP of Engineering at AddThis, discusses lessons learned from processing large-scale web data. AddThis processes data from 14 million domains, including 100 billion monthly page views and 50,000 events per second. Reverte outlines challenges around distributed ID generation, counting unique values, joining distributed data, sampling large datasets, and deploying systems that invalidate over 1.4 billion browser caches. He advocates for loose coupling between systems using approaches like Kafka for asynchronous event logging. Reverte also discusses techniques for columnar compression, tunable quality of service, and open sourcing Hydra, AddThis' custom processing system optimized for real-time data.

Analytical data processingPolad Saruxanov

MongoDB is a cross-platform, document-oriented database that is free, open source, and scalable. It stores data in flexible, JSON-like documents, allowing for easy addition of new document structures, and uses map-reduce functions for complex data processing and aggregation. Key features include horizontal scalability, replication for high availability, and the ability to handle structured, semi-structured, and unstructured data.

Tms trainingChi Lee

.NET Memory Primer (Martin Kulov)ITCamp

Feel++ webinar 9 27 2012Université de Strasbourg

The document summarizes the state of Feel++, an open source C++ finite element library. Recent developments include improvements to the programming environment, numerics, language, and coupling capabilities. Projects using Feel++ were presented at conferences. Upcoming events include a workshop on Feel++ and reduced order modeling. Work continues on certification reduced basis methods and non-conforming discretization techniques for parallel computing.

Mongo nyc nyt + mongodbDeep Kapadia

The document discusses lessons learned from using MongoDB at the New York Times over 6 months. It covers initial setup without backups or monitoring, improving to replication and monitoring, optimizing storage, backups, restores, querying, indexing and administration. Key lessons include using replication and backups, monitoring all aspects of MongoDB and storage, optimizing data and indexes for queries, and understanding data and access patterns.

FOSDEM 2020: Querying over millions and billions of metrics with M3DB's indexRob Skillington

The cardinality of monitoring data we are collecting today continues to rise, in no small part due to the ephemeral nature of containers and compute platforms like Kubernetes. Querying a flat dataset comprised of an increasing number of metrics requires searching through millions and in some cases billions of metrics to select a subset to display or alert on. The ability to use wildcards or regex within the tag name and values of these metrics and traces are becoming less of a nice-to-have feature and more useful for the growing popularity of ad-hoc exploratory queries. In this talk we will look at how Prometheus introduced the concept of a reverse index existing side-by-side with a traditional column based TSDB in a single process. We will then walk through the evolution of M3’s metric index, starting with ElasticSearch and evolving over the years to the current M3DB reverse index. We will give an in depth overview of the alternate designs and dive deep into the architecture of the current distributed index and the optimizations we’ve made in order to fulfill wildcards and regex queries across billions of metrics.

Redis as a Main Database, Scaling and HADave Nielsen

Iskren Chernev, an Independent developer, uses a lot of Redis. In this talk, Iskren will look at a particular Redis use-case -- using it as the main database (not cache). Iskren will show how to achieve reasonable guarantees about data integrity, speed, high-availability in an event of failure and infinite horizontal scalability. This particular approach has proven successful in managing clusters of up to 2400 nodes, and storing data north of 7TB before replication. We'll cover ways to separate your data appropriately into many nodes, performing different types of migrations (from another database, from one cluster to another, scaling migrations and migrating out of Redis), moving nodes without downtime, some configuration tips and monitoring.

DBpedia Viewer - LDOW 2014Dimitris Kontokostas

The document describes DBpedia Viewer, an interface for exploring DBpedia data. It leverages DBpedia services and tools to provide an intuitive interface that reduces information overload. The viewer uses AngularJS for dynamic rendering of entities from Virtuoso. It features entity summaries, search, language filtering, mapping integration, and "triple actions" that enable custom interactions and functionality based on entity properties.

erlang 101Gokhan Boranalp

Erlang 101 provides an overview of the Erlang programming language. It discusses Erlang's history and current status, key features like concurrency, distribution, and fault tolerance. These features make Erlang well-suited for building large, distributed, highly-available systems. Examples are given of companies using Erlang like Amazon, Facebook, Twitter, and of applications and frameworks built with Erlang like Apache CouchDB, RabbitMQ, and Riak. Resources for learning more about Erlang are also provided.

Caching in (DevoxxUK 2013)RichardWarburton

Modern computationally intensive tasks are rarely bottlenecked on the absolute performance of your processor cores, the real bottleneck in 2012 is getting data out of memory. CPU Caches are designed to alleviate the difference in performance between CPU Core Clockspeed and main memory clockspeed, but developers rarely understand how this interaction works or how to measure or tune their application accordingly. This Talk aims to solve that by: 1. Describing how the CPU caches work in the latest Intel Hardware. 2. Showing people what and how to measure in order to understand the caching behaviour of their software. 3. Giving examples of how this affects Java Program performance and what can be done to address things.

Python's slippy path and Tao of thick Pandas: give my data, Rrrrr...Alexey Zinoviev

This document contains a summary of a presentation about data mining and machine learning tools in Python. It discusses popular Python packages like NumPy, SciPy, Pandas, Scikit-learn, Spark, and NetworkX. It also covers data sources, preprocessing, common algorithms like classification, regression and clustering. Finally, it provides examples of network graphs and discusses scaling tools from small to big data using databases, HDFS and analysis frameworks like Hive and Spark.

More Related Content

What's hot (20)

Geo data analyticsDaniel Marcous

TriHUG 3/14: HBase in Productiontrihug

DemonstrationSean Murphy

openTSDB - Metrics for a distributed worldOliver Hankeln

Caching inRichardWarburton

Pain points with M3, some things to address them and how replication worksRob Skillington

Apache Flink Training Workshop @ HadoopCon2016 - #2 DataSet API Hands-OnApache Flink Taiwan User Group

FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...Rob Skillington

OpenTSDB: HBaseCon2017HBaseCon

OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...NETWAYS

Data Lessons Learned at ScaleCharlie Reverte

Analytical data processingPolad Saruxanov

Tms trainingChi Lee

.NET Memory Primer (Martin Kulov)ITCamp

Feel++ webinar 9 27 2012Université de Strasbourg

Mongo nyc nyt + mongodbDeep Kapadia

FOSDEM 2020: Querying over millions and billions of metrics with M3DB's indexRob Skillington

Redis as a Main Database, Scaling and HADave Nielsen

DBpedia Viewer - LDOW 2014Dimitris Kontokostas

erlang 101Gokhan Boranalp

Geo data analyticsDaniel Marcous

TriHUG 3/14: HBase in Productiontrihug

DemonstrationSean Murphy

openTSDB - Metrics for a distributed worldOliver Hankeln

Caching inRichardWarburton

Pain points with M3, some things to address them and how replication worksRob Skillington

Apache Flink Training Workshop @ HadoopCon2016 - #2 DataSet API Hands-OnApache Flink Taiwan User Group

FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...Rob Skillington

OpenTSDB: HBaseCon2017HBaseCon

OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...NETWAYS

Data Lessons Learned at ScaleCharlie Reverte

Analytical data processingPolad Saruxanov

Tms trainingChi Lee

.NET Memory Primer (Martin Kulov)ITCamp

Feel++ webinar 9 27 2012Université de Strasbourg

Mongo nyc nyt + mongodbDeep Kapadia

FOSDEM 2020: Querying over millions and billions of metrics with M3DB's indexRob Skillington

Redis as a Main Database, Scaling and HADave Nielsen

DBpedia Viewer - LDOW 2014Dimitris Kontokostas

erlang 101Gokhan Boranalp

Similar to Sharding - patterns & antipatterns, Константин Осипов, Алексей Рыбак (20)

Caching in (DevoxxUK 2013)RichardWarburton

Python's slippy path and Tao of thick Pandas: give my data, Rrrrr...Alexey Zinoviev

DruidDori Waldman

This document discusses Druid in production at Fyber, a company that indexes 5 terabytes of data daily from various sources into Druid. It describes the hardware used, including 30 historical nodes and 2 broker nodes. Issues addressed include slow query times with many dimensions, some as lists, and data cleanup steps to reduce cardinality like replacing values. Segment sizing and partitioning are also discussed. Hardware, data ingestion, querying, and optimizations used to scale Druid for Fyber's analytics needs are covered in under 3 sentences.

Database Performance at Scale Masterclass: Driver Strategies by Piotr SarnaScyllaDB

Caching inRichardWarburton

This document discusses CPU caching and provides strategies for optimizing code to improve cache performance. It explains that (1) modern CPUs use multilevel caching with cache misses causing costly stalls, (2) performance counters can measure cache hits and misses to identify hotspots, and (3) principles like data locality, sequential access, and avoiding holes in data structures can improve cache behavior and performance.

2013 05 nySri Ambati

- H2O is an open source platform for distributed and parallel machine learning that allows for fast in-memory computation across clusters. - It performs logistic regression and generalized linear models (GLMs) through an iterative process of computing Gram matrices in a distributed, parallel manner across nodes. - The core computation involves calculating the Gram matrix, which requires touching all the data, in a way that can be distributed and parallelized efficiently for large datasets.

AWS big-data-demystified #1.1 | Big Data Architecture Lessons Learned | EnglishOmid Vahdaty

Sv big datascience_cliffclick_5_2_2013Sri Ambati

This document discusses how big data and machine learning can be used to gain insights from large datasets and answer complex questions. It describes challenges in working with big data like data cleaning, modeling large datasets, and limitations of traditional tools. It then introduces H2O as a platform for performing fast, distributed machine learning on big data through an in-memory key-value store, distributed fork/join framework, and APIs for math hacking and model building. H2O aims to allow users to manipulate big data interactively like small data through its distributed, parallel architecture.

Web-scale data processing: practical approaches for low-latency and batchEdward Capriolo

The document is a slide deck presentation about batch processing, stream processing, and relational and NoSQL databases. It introduces the speaker and their experience with Hadoop, Cassandra, and Hive. It then covers batch processing using Hadoop, describing common architectures and use cases like processing web server logs. It discusses limitations of batch processing and then introduces stream processing concepts like Kafka and Storm. It provides an example of using Storm to perform word counting on streams of text data and discusses storing streaming results. Finally, it covers temporal databases and storing streaming results incrementally in Cassandra.

Which DBMS and Why?Majid Azimi

The document discusses various database management systems (DBMS) and considerations for choosing the right one based on different data and usage scenarios. It covers relational databases like MySQL and PostgreSQL, NoSQL databases like Redis, MongoDB, Cassandra and HBase, and specialized databases for graphs (Neo4j), time series data, and file storage. Key factors discussed include scalability, data structure, consistency guarantees, and query performance.

PostgreSQL and Redis - talk at pgcon 2013Andrew Dunstan

This document discusses using PostgreSQL and Redis together for a high performance ad serving system. Redis is used as a fast in-memory database to store frequently accessed data like ad impressions and clicks. A Redis foreign data wrapper allows PostgreSQL to efficiently retrieve this data from Redis and load it into the recording database. The PostgreSQL databases store longer term data and handle reporting. Materialized views are used to optimize ad filtering queries. Together PostgreSQL and Redis allow the system to serve over 10,000 ads per second while recording analytics data for reporting.

Piano Media - approach to data gathering and processingMartinStrycek

kranonit S06E01 Игорь Цинько: High loadKrivoy Rog IT Community

This document summarizes a presentation about designing systems to handle high loads when Chuck Norris is your customer. It discusses scaling architectures vertically and horizontally, RESTful principles, using NoSQL databases like MongoDB, caching with Memcached, search engines like Sphinx, video/image storage, and bandwidth management. It emphasizes that the right technology depends on business needs, and high-load systems require robust architectures, qualified developers, and avoiding single points of failure.

UKOUG 2011: Practical MySQL TuningFromDual GmbH

This document provides a summary of a presentation on practical MySQL tuning. It discusses measuring critical system resources like CPU, memory, I/O and network usage to identify bottlenecks. It also covers rough tuning of MySQL parameters like the InnoDB buffer pool size, log file size and key buffer size. Further tuning includes application optimizations like query tuning with EXPLAIN, index tuning, and schema design. The presentation also discusses scaling MySQL through approaches like caching, sharding, replication and optimizing architecture and data distribution. Regular performance monitoring is emphasized to simulate increased load and aid capacity planning.

Large Scale NoSql DB Migration Under Fire - Ido Barkan - DevOpsDays Tel Aviv ...DevOpsDays Tel Aviv

This document summarizes a large scale migration from CouchBase to Aerospike by Appsflyer due to scaling issues. Key points include: - Appsflyer processes 25B events per day and needed to replace CouchBase due to poor scale out, replication issues, and backup times over 24 hours. - The migration plan involved initially writing new data to both databases, then migrating older data via views and Kafka to avoid downtime or data loss. - Testing was crucial between stages to validate sizing, performance, and metrics. Data was not deleted from the old database until necessary to minimize risk. - The new Aerospike cluster saw improvements, with half the machines and a third of

How to build TiDBPingCAP

This is the speech Max Liu gave at Percona Live Open Source Database Conference 2016. Max Liu: Co-founder and CEO, a hacker with a free soul The slide covered the following topics: - Why another database? - What kind of database we want to build? - How to design such a database, including the principles, the architecture, and design decisions? - How to develop such a database, including the architecture and the core technologies for TiKV and TiDB? - How to test the database to ensure the quality and stability?

Big Data in 200 km/h | AWS Big Data Demystified #1.3 Omid Vahdaty

What we're about A while ago I entered the challenging world of Big Data. As an engineer, at first, I was not so impressed with this field. As time went by, I realised more and more, The technological challenges in this area are too great to master by one person. Just look at the picture in this articles, it only covers a small fraction of the technologies in the Big Data industry… Consequently, I created a meetup detailing all the challenges of Big Data, especially in the world of cloud. I am using AWS infrastructure to answer the basic questions of anyone starting their way in the big data world. how to transform data (TXT, CSV, TSV, JSON) into Parquet, ORCwhich technology should we use to model the data ? EMR? Athena? Redshift? Spectrum? Glue? Spark? SparkSQL?how to handle streaming?how to manage costs?Performance tips?Security tip?Cloud best practices tips? Some of our online materials: Website: https://big-data-demystified.ninja/ Youtube channels: https://www.youtube.com/channel/UCzeGqhZIWU-hIDczWa8GtgQ?view_as=subscriber https://www.youtube.com/channel/UCMSdNB0fGmX5dXI7S7Y_LFA?view_as=subscriber Meetup: https://www.meetup.com/AWS-Big-Data-Demystified/ https://www.meetup.com/Big-Data-Demystified Facebook Group : https://www.facebook.com/groups/amazon.aws.big.data.demystified/ Facebook page (https://www.facebook.com/Amazon-AWS-Big-Data-Demystified-1832900280345700/) Audience: Data Engineers Data Science DevOps Engineers Big Data Architects Solution Architects CTO VP R&D

Joker'14 Java as a fundamental working tool of the Data ScientistAlexey Zinoviev

This document discusses Java as a fundamental tool for data scientists. It provides an overview of data mining and machine learning techniques like classification, regression, clustering, and decision trees. It promotes using Java tools like Weka, Mahout, Spark and MLlib for big data analysis and machine learning. These tools allow processing large datasets across clusters using Java APIs and libraries. The document argues that Java is better suited than other languages like Python, R or Octave for building production machine learning systems at scale.

Etl confessions pg conf us 2017Corey Huinker

This document discusses strategies for efficiently loading and transforming large datasets in PostgreSQL for analytics use cases. It presents several case studies: 1) Loading a large CSV file - different methods like pgloader, COPY, and temporary foreign tables are compared. Temporary foreign tables perform best when filtering columns. 2) Pre-aggregating ("rolling up") data into multiple tables at different granularities for optimized querying. Chained INSERTs and CTEs are more efficient than individual inserts. 3) Creating a "dumb rollup table" using GROUPING SETS to pre-aggregate into a single temp table and insert into final tables in one pass. This outperforms multiple round trips or inserts.

Austin bdug 2011_01_27_small_and_big_dataAlex Pinkin

Caching in (DevoxxUK 2013)RichardWarburton

Python's slippy path and Tao of thick Pandas: give my data, Rrrrr...Alexey Zinoviev

DruidDori Waldman

Database Performance at Scale Masterclass: Driver Strategies by Piotr SarnaScyllaDB

Caching inRichardWarburton

2013 05 nySri Ambati

AWS big-data-demystified #1.1 | Big Data Architecture Lessons Learned | EnglishOmid Vahdaty

Sv big datascience_cliffclick_5_2_2013Sri Ambati

Web-scale data processing: practical approaches for low-latency and batchEdward Capriolo

Which DBMS and Why?Majid Azimi

PostgreSQL and Redis - talk at pgcon 2013Andrew Dunstan

Piano Media - approach to data gathering and processingMartinStrycek

kranonit S06E01 Игорь Цинько: High loadKrivoy Rog IT Community

UKOUG 2011: Practical MySQL TuningFromDual GmbH

Large Scale NoSql DB Migration Under Fire - Ido Barkan - DevOpsDays Tel Aviv ...DevOpsDays Tel Aviv

How to build TiDBPingCAP

Big Data in 200 km/h | AWS Big Data Demystified #1.3 Omid Vahdaty

Joker'14 Java as a fundamental working tool of the Data ScientistAlexey Zinoviev

Etl confessions pg conf us 2017Corey Huinker

Austin bdug 2011_01_27_small_and_big_dataAlex Pinkin

More from Ontico (20)

One-cloud — система управления дата-центром в Одноклассниках / Олег Анастасье...Ontico

HighLoad++ 2017 Зал «Калининград», 8 ноября, 15:00 Тезисы: http://www.highload.ru/2017/abstracts/2964.html Одноклассники состоят из более чем восьми тысяч железных серверов, расположенных в нескольких дата-центрах. Каждая из этих машин была специализированной под конкретную задачу - как для обеспечения изоляции отказов, так и для обеспечения автоматизированного управления инфраструктурой. ...

Масштабируя DNS / Артем Гавриченков (Qrator Labs)Ontico

HighLoad++ 2017 Зал «Калининград», 8 ноября, 16:00 Тезисы: http://www.highload.ru/2017/abstracts/3032.html Протокол DNS на семь лет старше, чем Всемирная паутина. Стандарты RFC 882 и 883, определяющие основную функциональность системы доменных имён, появились в конце 1983 года, а первая реализация последовала уже годом позже. Естественно, что у технологии столь старой и при этом по сей день активнейшим образом используемой просто не могли не накопиться особенности, неочевидные обыкновенным пользователям. ...

Создание BigData-платформы для ФГУП Почта России / Андрей Бащенко (Luxoft)Ontico

HighLoad++ 2017 Зал «Калининград», 8 ноября, 13:00 Тезисы: http://www.highload.ru/2017/abstracts/3010.html В этом докладе я расскажу, как BigData-платформа помогает трансформировать Почту России, как мы управляем построением и развитием платформы. Расскажу про найденные удачные решения, например, как разбиение на продукты с понятными SLA и интерфейсами между ними помогло нам сохранять управляемость с ростом масштабов проекта. ...

Готовим тестовое окружение, или сколько тестовых инстансов вам нужно / Алекса...Ontico

HighLoad++ 2017 Зал «Кейптаун», 8 ноября, 10:00 Тезисы: http://www.highload.ru/2017/abstracts/2914.html Казалось бы, что нужно для организации тестового окружения? Тестовая железка и копия боевого окружения - и тестовый сервер готов. Но как быть, когда проект сложный? А когда большой? А если нужно тестировать одновременно много версий? А если все это вместе? Организация тестирования большого развивающегося проекта, где одновременно в разработке и тестировании около полусотни фич - достаточно непростая задача. Ситуация обычно осложняется тем, что иногда есть желание потрогать еще не полностью готовый функционал. В таких ситуациях часто возникает вопрос: "А куда это можно накатить и где покликать?" ...

Новые технологии репликации данных в PostgreSQL / Александр Алексеев (Postgre...Ontico

HighLoad++ 2017 Зал «Кейптаун», 8 ноября, 18:00 Тезисы: http://www.highload.ru/2017/abstracts/2854.html Из этого доклада вы узнаете о возможностях репликации и автофейловера PostgreSQL, в том числе о возможностях, ставших доступных в PostgreSQL 10. Среди прочих, будет затронуты следующие темы: * Виды репликации и решаемые с ее помощью проблемы. * Настройка потоковой репликации. * Настройка логической репликации. * Настройка автофейловера / HA средствами Stolon и Consul. После прослушивания доклада вы сможете самостоятельно настраивать репликацию и автофейловер PostgreSQL.

PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)Ontico

HighLoad++ 2017 Зал «Кейптаун», 8 ноября, 17:00 Тезисы: http://www.highload.ru/2017/abstracts/3096.html PostgreSQL is the world’s most advanced open source database. Indeed! With around 270 configuration parameters in postgresql.conf, plus all the knobs in pg_hba.conf, it is definitely ADVANCED! How many parameters do you tune? 1? 8? 32? Anyone ever tuned more than 64? No tuning means below par performance. But how to start? Which parameters to tune? What are the appropriate values? Is there a tool --not just an editor like vim or emacs-- to help users manage the 700-line postgresql.conf file? Join this talk to understand the performance advantages of appropriately tuning your postgresql.conf file, showcase a new free tool to make PostgreSQL configuration possible for HUMANS, and learn the best practices for tuning several relevant postgresql.conf parameters.

Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...Ontico

HighLoad++ 2017 Зал «Кейптаун», 8 ноября, 16:00 Тезисы: http://www.highload.ru/2017/abstracts/3115.html During this session we will cover the last development in ProxySQL to support regular expressions (RE2 and PCRE) and how we can use this strong technique in correlation with ProxySQL's query rules to anonymize live data quickly and transparently. We will explain the mechanism and how to generate these rules quickly. We show live demo with all challenges we got from the Community and we finish the session by an interactive brainstorm testing queries from the audience.

Опыт разработки модуля межсетевого экранирования для MySQL / Олег Брославский...Ontico

HighLoad++ 2017 Зал «Кейптаун», 8 ноября, 15:00 Тезисы: http://www.highload.ru/2017/abstracts/2957.html Расскажем о нашем опыте разработки модуля межсетевого экрана для MySQL с использованием генератора парсеров ANTLR и языка Kotlin. Подробно рассмотрим следующие вопросы: — когда и почему целесообразно использовать ANTLR; — особенности разработки ANTLR-грамматики для MySQL; — сравнение производительности рантаймов для ANTLR в рамках задачи синтаксического анализа MySQL (C#, Java, Kotlin, Go, Python, PyPy, C++); — вспомогательные DSL; — микросервисная архитектура модуля экранирования SQL; — полученные результаты.

ProxySQL Use Case Scenarios / Alkin Tezuysal (Percona)Ontico

HighLoad++ 2017 Зал «Кейптаун», 8 ноября, 14:00 Тезисы: http://www.highload.ru/2017/abstracts/3114.html ProxySQL aims to be the most powerful proxy in the MySQL ecosystem. It is protocol-aware and able to provide high availability (HA) and high performance with no changes in the application, using several built-in features and integration with clustering software. During this session we will quickly introduce its main features, so to better understand how it works. We will then describe multiple use case scenarios in which ProxySQL empowers large MySQL installations to provide HA with zero downtime, read/write split, query rewrite, sharding, query caching, and multiplexing using SSL across data centers.

MySQL Replication — Advanced Features / Петр Зайцев (Percona)Ontico

HighLoad++ 2017 Зал «Кейптаун», 8 ноября, 13:00 Тезисы: http://www.highload.ru/2017/abstracts/2954.html MySQL Replication is powerful and has added a lot of advanced features through the years. In this presentation we will look into replication technology in MySQL 5.7 and variants focusing on advanced features, what do they mean, when to use them and when not, Including. When should you use STATEMENT, ROW or MIXED binary log format? What is GTID in MySQL and MariaDB and why do you want to use them? What is semi-sync replication and how is it different from lossless semi-sync? ...

Внутренний open-source. Как разрабатывать мобильное приложение большим количе...Ontico

HighLoad++ 2017 Зал «Кейптаун», 8 ноября, 12:00 Тезисы: http://www.highload.ru/2017/abstracts/3120.html Количество разработчиков мобильных приложений Сбербанк Онлайн с начала 2016 года выросло на порядок. Для того чтобы продолжать выпускать качественный продукт, мы кардинально перестраиваем процесс разработки. Количество внутренних заказчиков тех или иных доработок в какой-то момент выросло настолько, что разработчики стали узким местом. Мы внедрили культуру разработки, которую можно условно назвать "внутренним open-source", сохранив за собой контроль над архитектурой и качеством проекта, но позволив разрабатывать новые фичи всем желающим. ...

Подробно о том, как Causal Consistency реализовано в MongoDB / Михаил Тюленев...Ontico

HighLoad++ 2017 Зал «Мумбай», 8 ноября, 18:00 Тезисы: http://www.highload.ru/2017/abstracts/2836.html При использовании Eventually Consistent распределенных баз данных нет гарантий, что чтение возвращает результаты последних изменений данных, если чтение и запись производятся на разных узлах. Это ограничивает пропускную способность системы. Поддержка свойства Causal Consistency снимает это ограничение, что позволяет улучшить масштабируемость, не требуя изменений в коде приложения. ...

Балансировка на скорости проводов. Без ASIC, без ограничений. Решения NFWare ...Ontico

HighLoad++ 2017 Зал «Мумбай», 8 ноября, 16:00 Тезисы: http://www.highload.ru/2017/abstracts/2858.html Аудитория Одноклассников превышает 73 миллиона человек в России, СНГ и странах дальнего зарубежья. При этом ОК.ru - первая социальная сеть по просмотрам видео в рунете и крупнейшая сервисная платформа. Качественный и количественный рост DDoS-атак за последние годы превращает их в одну из первоочередных проблем для крупнейших интернет-ресурсов. В зависимости от вектора атаки “узким” местом становится та или иная часть инфраструктуры. В частности, при SYN-flood первый удар приходится на систему балансировки трафика. От ее производительности зависит успех в противостоянии атаке. ...

Перехват трафика — мифы и реальность / Евгений Усков (Qrator Labs)Ontico

HighLoad++ 2017 Зал «Мумбай», 8 ноября, 15:00 Тезисы: http://www.highload.ru/2017/abstracts/3008.html Никогда не было и вот снова случилось! Компания Google в результате перенаправления трафика сделала недостпуными в Японии несколько тысяч различных сервисов, большинство из которых никак не связано с самой компанией Google. Однако, подобные инциденты происходят с завидной регулярностью, вот только не всегда попадают в большие СМИ. У таких инцидентов могут быть разные причины, начиная от ошибок сетевых инженеров и заканчивая государственным регулированием. ...

И тогда наверняка вдруг запляшут облака! / Алексей Сушков (ПЕТЕР-СЕРВИС)Ontico

HighLoad++ 2017 Зал «Мумбай», 8 ноября, 14:00 Тезисы: http://www.highload.ru/2017/abstracts/2925.html Облака и виртуализация – современные тренды развития IT-технологий. Операторы связи строят свои TelcoClouds на стандартах NFV (Network Functions Virtualization) и SDN (Software-Defined Networking). В докладе начнем с основ виртуализации, далее разберемся, для чего используются NFV и SDN, потом полетим к облакам и вернемся на землю для решения практических задач! ...

Как мы заставили Druid работать в Одноклассниках / Юрий Невиницин (OK.RU)Ontico

HighLoad++ 2017 Зал «Мумбай», 8 ноября, 10:00 Тезисы: http://www.highload.ru/2017/abstracts/3045.html Как мы заставили Druid работать в Одноклассниках. «Druid is a high-performance, column-oriented, distributed data store» http://druid.io. Мы расскажем о том, как, внедрив Druid, мы справились с ситуацией, когда MSSQL-based система статистики на 50 терабайт стала: - медленной: средняя скорость ответа была в разы меньше требуемой (и увеличилась в 20 раз); - нестабильной: в час пик статистика отставала до получаса (теперь ничего не отстает); - дорогой: изменилась политика лицензирования Microsoft, расходы на лицензии могли составить миллионы долларов. ...

Разгоняем ASP.NET Core / Илья Вербицкий (WebStoating s.r.o.)Ontico

HighLoad++ 2017 Зал «Рио-де-Жанейро», 8 ноября, 18:00 Тезисы: http://www.highload.ru/2017/abstracts/2905.html Прошло более года с того момента, как Microsoft выпустила первую версию своего нового фреймворка для разработки web-приложений ASP.NET Core, и с каждым днем он находит все больше поклонников. ASP.NET Core базируется на платформе .NET Core, кроссплатформенной версии платформы .NET c открытым исходным кодом. Теперь у С#-разработчиков появилась возможность использовать Mac в качестве среды разработки, и запускать приложения на Linux или внутри Docker-контейнеров. ...

100500 способов кэширования в Oracle Database или как достичь максимальной ск...Ontico

HighLoad++ 2017 Зал «Рио-де-Жанейро», 8 ноября, 14:00 Тезисы: http://www.highload.ru/2017/abstracts/2913.html Изначально будут раскрыты базовые причины, которые заставили появиться такой части механизма СУБД, как кэш результатов, и почему в ряде СУБД он есть или отсутствует. Будут рассмотрены различные варианты кэширования результатов как sql-запросов, так и результатов хранимой в БД бизнес-логики. Произведено сравнение способов кэширования (программируемые вручную кэши, стандартный функционал) и даны рекомендации, когда и в каких случаях данные способы оптимальны, а порой опасны. ...

Apache Ignite Persistence: зачем Persistence для In-Memory, и как он работает...Ontico

HighLoad++ 2017 Зал «Рио-де-Жанейро», 8 ноября, 13:00 Тезисы: http://www.highload.ru/2017/abstracts/2947.html Apache Ignite — Open Source платформа для высокопроизводительной распределенной работы с большими данными с применением SQL или Java/.NET/C++ API. Ignite используют в самых разных отраслях. Сбербанк, ING, RingCentral, Microsoft, e-Therapeutics — все эти компании применяют решения на основе Ignite. Размеры кластеров разнятся от всего одного узла до нескольких сотен, узлы могут быть расположены в одном ЦОД-е или в нескольких геораспределенных. ...

Механизмы мониторинга баз данных: взгляд изнутри / Дмитрий Еманов (Firebird P...Ontico

HighLoad++ 2017 Зал «Рио-де-Жанейро», 8 ноября, 12:00 Тезисы: http://www.highload.ru/2017/abstracts/3005.html Когда мы говорим о нагруженных системах и базах данных с большим числом параллельных коннектов, особый интерес представляет практика эксплуатации и сопровождения таких проектов. В том числе инструменты и механизмы СУБД, которые могут быть использованы DBA и DevOps-инженерами для решения задач мониторинга жизнедеятельности базы данных и ранней диагностики возможных проблем. ...