MongoDB sharded cluster. How to design your topology ?
2 likes1,470 views
This slides was presented at Mydbops Database Meetup 4 on Aug-03 2019 by Vinodh Krishnaswamy ( Percona ). This talk focuses on when to go for sharing topology in MongoDB and their benefits and impact.
1 of 47
Downloaded 27 times
Ad
Recommended
Introduction to Storm 
Introduction to Storm Chandler Huang Storm is a distributed and fault-tolerant realtime computation system. It was created at BackType/Twitter to analyze tweets, links, and users on Twitter in realtime. Storm provides scalability, reliability, and ease of programming. It uses components like Zookeeper, ØMQ, and Thrift. A Storm topology defines the flow of data between spouts that read data and bolts that process data. Storm guarantees processing of all data through its reliability APIs and guarantees no data loss even during failures.
Sharding Methods for MongoDB
Sharding Methods for MongoDBMongoDB Sharding in MongoDB allows for horizontal scaling of data and operations across multiple servers. When determining if sharding is needed, factors like available storage, query throughput, and response latency on a single server are considered. The number of shards can be calculated based on total required storage, working memory size, and input/output operations per second across servers. Different types of sharding include range, tag-aware, and hashed sharding. Choosing a high cardinality shard key that matches query patterns is important for performance. Reasons to shard include scaling to large data volumes and query loads, enabling local writes in a globally distributed deployment, and improving backup and restore times.
MySQL Server Settings Tuning
MySQL Server Settings Tuningguest5ca94b The document discusses tuning MySQL server settings for performance. Some key points covered include:
- Settings are workload-specific and depend on factors like storage engine, OS, hardware. Tuning involves getting a few settings right rather than maximizing all settings.
- Monitoring tools like SHOW STATUS, SHOW INNODB STATUS, and OS tools can help evaluate performance and identify tuning opportunities.
- Memory allocation and settings like innodb_buffer_pool_size, key_buffer_size, query_cache_size are important to configure based on the workload and available memory.
MongoDB Performance Debugging
MongoDB Performance DebuggingMongoDB The document discusses debugging performance problems in MongoDB. It describes solutions tried such as denormalizing data, adding indexes, sharding, scaling hardware, and tagging shards. While performance improved with these solutions at times, slowness still occasionally occurred. The document advocates using monitoring tools like New Relic and Skylight to further analyze issues and find additional solutions.
What is in a Lucene index?
What is in a Lucene index?lucenerevolution Presented by Adrien Grand, Software Engineer, Elasticsearch
Although people usually come to Lucene and related solutions in order to make data searchable, they often realize that it can do much more for them. Indeed, its ability to handle high loads of complex queries make Lucene a perfect fit for analytics applications and, for some use-cases, even a credible replacement for a primary data-store. It is important to understand the design decisions behind Lucene in order to better understand the problems it can solve and the problems it cannot solve. This talk will explain the design decisions behind Lucene, give insights into how Lucene stores data on disk and how it differs from traditional databases. Finally, there will be highlights of recent and future changes in Lucene index file formats.
Cassandra Introduction & Features
Cassandra Introduction & FeaturesDataStax Academy This presentation shortly describes key features of Apache Cassandra. It was held at the Apache Cassandra Meetup in Vienna in January 2014. You can access the meetup here: http://www.meetup.com/Vienna-Cassandra-Users/
Introduction to Redis
Introduction to RedisMaarten Smeets Redis is an open source in memory database which is easy to use. In this introductory presentation, several features will be discussed including use cases. The datatypes will be elaborated, publish subscribe features, persistence will be discussed including client implementations in Node and Spring Boot. After this presentation, you will have a basic understanding of what Redis is and you will have enough knowledge to get started with your first implementation!
PostgreSQL, performance for queries with grouping
PostgreSQL, performance for queries with groupingAlexey Bashtanov The talk will cover PostgreSQL grouping and aggregation facilities and best practices of using them in fast and efficient manner.
In 40 minutes the audience will learn several techniques to optimise queries containing GROUP BY, DISTINCT or DISTINCT ON keywords.
Towards Functional Programming through Hexagonal Architecture
Towards Functional Programming through Hexagonal ArchitectureCodelyTV Slides of for the talk "Towards Functional Programming through Hexagonal Architecture" delivered at the Software Crafters Barcelona 2018 conference #scbcn18 by Juanma Serrano from Habla Computing and Javier Ferrer from CodelyTV
Redis
Redisimalik8088 Redis is an in-memory key-value database that is used by companies like Instagram and Stack Overflow for caching and storing non-relational data. It supports common data structures like strings, hashes, lists, sets and sorted sets. Redis provides very fast performance of over 100,000 writes and 80,000 reads per second. While Redis does not support complex queries, it is well suited for simple, fast queries and retrievals. The document discusses how Redis could be used to query and retrieve flight booking data by airport in both coarse and fine-grained data modeling approaches.
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Databricks Spark SQL is a highly scalable and efficient relational processing engine with ease-to-use APIs and mid-query fault tolerance. It is a core module of Apache Spark. Spark SQL can process, integrate and analyze the data from diverse data sources (e.g., Hive, Cassandra, Kafka and Oracle) and file formats (e.g., Parquet, ORC, CSV, and JSON). This talk will dive into the technical details of SparkSQL spanning the entire lifecycle of a query execution. The audience will get a deeper understanding of Spark SQL and understand how to tune Spark SQL performance.
Node.js API 서버 성능 개선기
Node.js API 서버 성능 개선기JeongHun Byeon The document discusses performance testing of an API server. It provides details on:
1) Preparing for performance testing by generating test data and configuring tools like Artillery to simulate user traffic.
2) Conducting the performance tests on different server configurations to identify bottlenecks, including testing with different Node.js versions.
3) Analyzing the results of the performance tests by profiling requests and examining response times to further optimize the server performance.
alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)
alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)Alphorm La formation complète est disponible ici:
http://www.alphorm.com/tutoriel/formation-en-ligne-oracle-database-11g-dba-1-1z0-052
Grâce à cette formation, vous pouvez commencer votre chemin pour devenir l'indispensable DBA Oracle dans votre entreprise.
Durant cette formation, Noureddine DRISSI, vous apprend à installer et gérer une base de données Oracle. Il présente l'architecture et les composants d'une base de données, ainsi que les interactions entre les différents éléments. Il montre comment créer une base opérationnelle et comment gérer correctement et efficacement les différentes structures, notamment via le contrôle des performances, la sécurité, la gestion des utilisateurs et les techniques de sauvegarde/récupération.
A la fin de cette formation vous serez en mesure de passer l'examen Oracle Certified Associate 1Z0-052 Oracle, une certification qui est presque obligatoire sur le marché du travail.
ClickHouse Introduction by Alexander Zaitsev, Altinity CTO
ClickHouse Introduction by Alexander Zaitsev, Altinity CTOAltinity Ltd This document summarizes a ClickHouse meetup agenda. The meetup included an opening by Javier Santana, an introduction to ClickHouse by Alexander Zaitsev of Altinity, a presentation on 2019 new ClickHouse features by Alexey Milovidov of Yandex, a coffee break, a presentation from Idealista on migrating from a legacy system to ClickHouse, a presentation from Corunet on analyzing 1027 predictive models in 10 seconds using ClickHouse, a presentation from Adjust on shipping data from Postgres to ClickHouse, closing remarks, and a networking session. The document then provides an overview of what ClickHouse is, how fast it can be, how flexible it is in deployment options, how
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsCloudera, Inc. The document discusses 5 common mistakes people make when writing Spark applications:
1) Not properly sizing executors for memory and cores.
2) Having shuffle blocks larger than 2GB which can cause jobs to fail.
3) Not addressing data skew which can cause joins and shuffles to be very slow.
4) Not properly managing the DAG to minimize shuffles and stages.
5) Classpath conflicts from mismatched dependencies causing errors.
Logical Replication in PostgreSQL - FLOSSUK 2016
Logical Replication in PostgreSQL - FLOSSUK 2016Petr Jelinek Slides from FLOSSUK Spring 2016 conference about the logical replication in PostgreSQL using pglogical extension.
Physical Plans in Spark SQL
Physical Plans in Spark SQLDatabricks In Spark SQL the physical plan provides the fundamental information about the execution of the query. The objective of this talk is to convey understanding and familiarity of query plans in Spark SQL, and use that knowledge to achieve better performance of Apache Spark queries. We will walk you through the most common operators you might find in the query plan and explain some relevant information that can be useful in order to understand some details about the execution. If you understand the query plan, you can look for the weak spot and try to rewrite the query to achieve a more optimal plan that leads to more efficient execution.
The main content of this talk is based on Spark source code but it will reflect some real-life queries that we run while processing data. We will show some examples of query plans and explain how to interpret them and what information can be taken from them. We will also describe what is happening under the hood when the plan is generated focusing mainly on the phase of physical planning. In general, in this talk we want to share what we have learned from both Spark source code and real-life queries that we run in our daily data processing.
Node.js and the MySQL Document Store
Node.js and the MySQL Document StoreRui Quelhas This document provides an overview of using Node.js with the MySQL Document Store. It discusses how MySQL can be used as a NoSQL database with the new MySQL Document Store API. Key components that enable this include the X DevAPI, Router, X Plugin, and X Protocol. The API allows for schemaless data storage and retrieval using JSON documents stored in MySQL collections. Documents can be added, retrieved, modified, and removed from collections using the MySQL Connector/Node.js driver.
ProxySQL Tutorial - PLAM 2016
ProxySQL Tutorial - PLAM 2016Derek Downey This document provides an overview of ProxySQL, a high performance proxy for MySQL. It discusses ProxySQL's main features such as query routing, caching, load balancing, and high availability capabilities including seamless failover. The document also describes ProxySQL's internal architecture including modules for queries processing, user authentication, hostgroup management, and more. Examples are given showing how hostgroups can be used for read/write splitting and replication topologies.
Spark SQL Bucketing at Facebook
Spark SQL Bucketing at FacebookDatabricks Bucketing is a popular data partitioning technique to pre-shuffle and (optionally) pre-sort data during writes. This is ideal for a variety of write-once and read-many datasets at Facebook, where Spark can automatically avoid expensive shuffles/sorts (when the underlying data is joined/aggregated on its bucketed keys) resulting in substantial savings in both CPU and IO.
Over the last year, we’ve added a series of optimizations in Apache Spark as a means towards achieving feature parity with Hive and Spark. These include avoiding shuffle/sort when joining/aggregating/inserting on tables with mismatching buckets, allowing user to skip shuffle/sort when writing to bucketed tables, adding data validators before writing bucketed data, among many others. As a direct consequence of these efforts, we’ve witnessed over 10x growth (spanning 40% of total compute) in queries that read one or more bucketed tables across the entire data warehouse at Facebook.
In this talk, we’ll take a deep dive into the internals of bucketing support in SparkSQL, describe use-cases where bucketing is useful, touch upon some of the on-going work to automatically suggest bucketing tables based on query column lineage, and summarize the lessons learned from developing bucketing support in Spark at Facebook over the last 2 years
MySQL Index Cookbook
MySQL Index CookbookMYXPLAIN This document provides an overview of indexing in MySQL. It begins with definitions of terminology like B-Trees, keys, indexes, and clustering. It then covers topics like primary keys, compound indexes, and optimization techniques. Use cases are presented to demonstrate how to design indexes for different querying needs, such as normalization, geospatial queries, and pagination. The document aims to explain indexing concepts and help the reader design efficient indexes.
Introduction to Apache solr
Introduction to Apache solrKnoldus Inc. Apache Solr is a popular, open source enterprise search platform built on the Java based search engine library Apache Lucene. It powers the search and navigation features of many of the world's largest companies like Netflix, Instagram, LinkedIn, Twitter and eBay, etc.
Spark Summit EU talk by Ross Lawley
Spark Summit EU talk by Ross LawleySpark Summit 1) Ross Lawley presented on connecting Spark to MongoDB. The MongoDB Spark connector started as an intern project in 2015 and was officially launched in 2016, written in Scala with Python and R support.
2) To read data from MongoDB, the connector partitions the collection, optionally using preferred shard locations for locality. It computes each partition's data as an iterator to be consumed by Spark.
3) For writing data, the connector groups data into batches by partition and inserts into MongoDB collections. DataFrames/Datasets will upsert if there is an ID.
4) The connector supports structured data in Spark by inferring schemas, creating relations, and allowing multi-language access from Scala, Python and R
2023 COSCUP - Whats new in PostgreSQL 16
2023 COSCUP - Whats new in PostgreSQL 16José Lin This document contains an agenda for a presentation on PostgreSQL. The agenda includes 10 items: 1) improving adaptability to large environments, 2) monitoring, 3) client applications, 4) server management and operations, 5) localization, 6) security, 7) SQL functions and commands, 8) performance, 9) logical replication, and 10) conclusion. It also includes several links to external resources providing additional information on topics related to PostgreSQL.
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)Aurimas Mikalauskas Is my MySQL server configured properly? Should I run Community MySQL, MariaDB, Percona or WebScaleSQL? How many innodb buffer pool instances should I run? Why should I NOT use the query cache? How do I size the innodb log file size and what IS that innodb log anyway? All answers are inside.
Aurimas Mikalauskas is a former Percona performance consultant and architect currently writing and teaching at speedemy.com. He's been involved with MySQL since 1999, scaling and optimizing MySQL backed systems since 2004 for companies such as BBC, EngineYard, famous social networks and small shops like EstanteVirtual, Pine Cove and hundreds of others.
Additional content mentioned in the presentation can be found here: http://speedemy.com/17
Optimizing Apache Spark SQL Joins
Optimizing Apache Spark SQL JoinsDatabricks Join operations in Apache Spark is often the biggest source of performance problems and even full-blown exceptions in Spark. After this talk, you will understand the two most basic methods Spark employs for joining DataFrames – to the level of detail of how Spark distributes the data within the cluster. You’ll also find out how to work out common errors and even handle the trickiest corner cases we’ve encountered! After this talk, you should be able to write performance joins in Spark SQL that scale and are zippy fast!
This session will cover different ways of joining tables in Apache Spark.
Speaker: Vida Ha
This talk was originally presented at Spark Summit East 2017.
A Deep Dive into Query Execution Engine of Spark SQL
A Deep Dive into Query Execution Engine of Spark SQLDatabricks Spark SQL enables Spark to perform efficient and fault-tolerant relational query processing with analytics database technologies. The relational queries are compiled to the executable physical plans consisting of transformations and actions on RDDs with the generated Java code. The code is compiled to Java bytecode, executed at runtime by JVM and optimized by JIT to native machine code at runtime. This talk will take a deep dive into Spark SQL execution engine. The talk includes pipelined execution, whole-stage code generation, UDF execution, memory management, vectorized readers, lineage based RDD transformation and action.
From HDFS to S3: Migrate Pinterest Apache Spark Clusters
From HDFS to S3: Migrate Pinterest Apache Spark ClustersDatabricks The document discusses Pinterest migrating their Apache Spark clusters from HDFS to S3 storage. Some key points:
1) Migrating to S3 provided significantly better performance due to the higher IOPS of modern EC2 instances compared to their older HDFS nodes. Jobs saw 25-35% improvements on average.
2) S3 is eventually consistent while HDFS is strongly consistent, so they implemented the S3Committer to handle output consistency issues during job failures.
3) Metadata operations like file moves were very slow in S3, so they optimized jobs to reduce unnecessary moves using techniques like multipart uploads to S3.
Lambda Architecture: The Best Way to Build Scalable and Reliable Applications!
Lambda Architecture: The Best Way to Build Scalable and Reliable Applications!Tugdual Grall Lambda Architecture is a useful framework to think about designing big data applications. This framework has been built initially at Twitter. In this presentation you will learn, based on concrete examples how to build deploy scalable and fault tolerant applications, with a focus on Big Data and Hadoop.
This presentation was delivered at the OOP conference, Munich, Feb 2016
Ad
More Related Content
What's hot (20)
PostgreSQL, performance for queries with grouping
PostgreSQL, performance for queries with groupingAlexey Bashtanov The talk will cover PostgreSQL grouping and aggregation facilities and best practices of using them in fast and efficient manner.
In 40 minutes the audience will learn several techniques to optimise queries containing GROUP BY, DISTINCT or DISTINCT ON keywords.
Towards Functional Programming through Hexagonal Architecture
Towards Functional Programming through Hexagonal ArchitectureCodelyTV Slides of for the talk "Towards Functional Programming through Hexagonal Architecture" delivered at the Software Crafters Barcelona 2018 conference #scbcn18 by Juanma Serrano from Habla Computing and Javier Ferrer from CodelyTV
Redis
Redisimalik8088 Redis is an in-memory key-value database that is used by companies like Instagram and Stack Overflow for caching and storing non-relational data. It supports common data structures like strings, hashes, lists, sets and sorted sets. Redis provides very fast performance of over 100,000 writes and 80,000 reads per second. While Redis does not support complex queries, it is well suited for simple, fast queries and retrievals. The document discusses how Redis could be used to query and retrieve flight booking data by airport in both coarse and fine-grained data modeling approaches.
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Databricks Spark SQL is a highly scalable and efficient relational processing engine with ease-to-use APIs and mid-query fault tolerance. It is a core module of Apache Spark. Spark SQL can process, integrate and analyze the data from diverse data sources (e.g., Hive, Cassandra, Kafka and Oracle) and file formats (e.g., Parquet, ORC, CSV, and JSON). This talk will dive into the technical details of SparkSQL spanning the entire lifecycle of a query execution. The audience will get a deeper understanding of Spark SQL and understand how to tune Spark SQL performance.
Node.js API 서버 성능 개선기
Node.js API 서버 성능 개선기JeongHun Byeon The document discusses performance testing of an API server. It provides details on:
1) Preparing for performance testing by generating test data and configuring tools like Artillery to simulate user traffic.
2) Conducting the performance tests on different server configurations to identify bottlenecks, including testing with different Node.js versions.
3) Analyzing the results of the performance tests by profiling requests and examining response times to further optimize the server performance.
alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)
alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)Alphorm La formation complète est disponible ici:
http://www.alphorm.com/tutoriel/formation-en-ligne-oracle-database-11g-dba-1-1z0-052
Grâce à cette formation, vous pouvez commencer votre chemin pour devenir l'indispensable DBA Oracle dans votre entreprise.
Durant cette formation, Noureddine DRISSI, vous apprend à installer et gérer une base de données Oracle. Il présente l'architecture et les composants d'une base de données, ainsi que les interactions entre les différents éléments. Il montre comment créer une base opérationnelle et comment gérer correctement et efficacement les différentes structures, notamment via le contrôle des performances, la sécurité, la gestion des utilisateurs et les techniques de sauvegarde/récupération.
A la fin de cette formation vous serez en mesure de passer l'examen Oracle Certified Associate 1Z0-052 Oracle, une certification qui est presque obligatoire sur le marché du travail.
ClickHouse Introduction by Alexander Zaitsev, Altinity CTO
ClickHouse Introduction by Alexander Zaitsev, Altinity CTOAltinity Ltd This document summarizes a ClickHouse meetup agenda. The meetup included an opening by Javier Santana, an introduction to ClickHouse by Alexander Zaitsev of Altinity, a presentation on 2019 new ClickHouse features by Alexey Milovidov of Yandex, a coffee break, a presentation from Idealista on migrating from a legacy system to ClickHouse, a presentation from Corunet on analyzing 1027 predictive models in 10 seconds using ClickHouse, a presentation from Adjust on shipping data from Postgres to ClickHouse, closing remarks, and a networking session. The document then provides an overview of what ClickHouse is, how fast it can be, how flexible it is in deployment options, how
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsCloudera, Inc. The document discusses 5 common mistakes people make when writing Spark applications:
1) Not properly sizing executors for memory and cores.
2) Having shuffle blocks larger than 2GB which can cause jobs to fail.
3) Not addressing data skew which can cause joins and shuffles to be very slow.
4) Not properly managing the DAG to minimize shuffles and stages.
5) Classpath conflicts from mismatched dependencies causing errors.
Logical Replication in PostgreSQL - FLOSSUK 2016
Logical Replication in PostgreSQL - FLOSSUK 2016Petr Jelinek Slides from FLOSSUK Spring 2016 conference about the logical replication in PostgreSQL using pglogical extension.
Physical Plans in Spark SQL
Physical Plans in Spark SQLDatabricks In Spark SQL the physical plan provides the fundamental information about the execution of the query. The objective of this talk is to convey understanding and familiarity of query plans in Spark SQL, and use that knowledge to achieve better performance of Apache Spark queries. We will walk you through the most common operators you might find in the query plan and explain some relevant information that can be useful in order to understand some details about the execution. If you understand the query plan, you can look for the weak spot and try to rewrite the query to achieve a more optimal plan that leads to more efficient execution.
The main content of this talk is based on Spark source code but it will reflect some real-life queries that we run while processing data. We will show some examples of query plans and explain how to interpret them and what information can be taken from them. We will also describe what is happening under the hood when the plan is generated focusing mainly on the phase of physical planning. In general, in this talk we want to share what we have learned from both Spark source code and real-life queries that we run in our daily data processing.
Node.js and the MySQL Document Store
Node.js and the MySQL Document StoreRui Quelhas This document provides an overview of using Node.js with the MySQL Document Store. It discusses how MySQL can be used as a NoSQL database with the new MySQL Document Store API. Key components that enable this include the X DevAPI, Router, X Plugin, and X Protocol. The API allows for schemaless data storage and retrieval using JSON documents stored in MySQL collections. Documents can be added, retrieved, modified, and removed from collections using the MySQL Connector/Node.js driver.
ProxySQL Tutorial - PLAM 2016
ProxySQL Tutorial - PLAM 2016Derek Downey This document provides an overview of ProxySQL, a high performance proxy for MySQL. It discusses ProxySQL's main features such as query routing, caching, load balancing, and high availability capabilities including seamless failover. The document also describes ProxySQL's internal architecture including modules for queries processing, user authentication, hostgroup management, and more. Examples are given showing how hostgroups can be used for read/write splitting and replication topologies.
Spark SQL Bucketing at Facebook
Spark SQL Bucketing at FacebookDatabricks Bucketing is a popular data partitioning technique to pre-shuffle and (optionally) pre-sort data during writes. This is ideal for a variety of write-once and read-many datasets at Facebook, where Spark can automatically avoid expensive shuffles/sorts (when the underlying data is joined/aggregated on its bucketed keys) resulting in substantial savings in both CPU and IO.
Over the last year, we’ve added a series of optimizations in Apache Spark as a means towards achieving feature parity with Hive and Spark. These include avoiding shuffle/sort when joining/aggregating/inserting on tables with mismatching buckets, allowing user to skip shuffle/sort when writing to bucketed tables, adding data validators before writing bucketed data, among many others. As a direct consequence of these efforts, we’ve witnessed over 10x growth (spanning 40% of total compute) in queries that read one or more bucketed tables across the entire data warehouse at Facebook.
In this talk, we’ll take a deep dive into the internals of bucketing support in SparkSQL, describe use-cases where bucketing is useful, touch upon some of the on-going work to automatically suggest bucketing tables based on query column lineage, and summarize the lessons learned from developing bucketing support in Spark at Facebook over the last 2 years
MySQL Index Cookbook
MySQL Index CookbookMYXPLAIN This document provides an overview of indexing in MySQL. It begins with definitions of terminology like B-Trees, keys, indexes, and clustering. It then covers topics like primary keys, compound indexes, and optimization techniques. Use cases are presented to demonstrate how to design indexes for different querying needs, such as normalization, geospatial queries, and pagination. The document aims to explain indexing concepts and help the reader design efficient indexes.
Introduction to Apache solr
Introduction to Apache solrKnoldus Inc. Apache Solr is a popular, open source enterprise search platform built on the Java based search engine library Apache Lucene. It powers the search and navigation features of many of the world's largest companies like Netflix, Instagram, LinkedIn, Twitter and eBay, etc.
Spark Summit EU talk by Ross Lawley
Spark Summit EU talk by Ross LawleySpark Summit 1) Ross Lawley presented on connecting Spark to MongoDB. The MongoDB Spark connector started as an intern project in 2015 and was officially launched in 2016, written in Scala with Python and R support.
2) To read data from MongoDB, the connector partitions the collection, optionally using preferred shard locations for locality. It computes each partition's data as an iterator to be consumed by Spark.
3) For writing data, the connector groups data into batches by partition and inserts into MongoDB collections. DataFrames/Datasets will upsert if there is an ID.
4) The connector supports structured data in Spark by inferring schemas, creating relations, and allowing multi-language access from Scala, Python and R
2023 COSCUP - Whats new in PostgreSQL 16
2023 COSCUP - Whats new in PostgreSQL 16José Lin This document contains an agenda for a presentation on PostgreSQL. The agenda includes 10 items: 1) improving adaptability to large environments, 2) monitoring, 3) client applications, 4) server management and operations, 5) localization, 6) security, 7) SQL functions and commands, 8) performance, 9) logical replication, and 10) conclusion. It also includes several links to external resources providing additional information on topics related to PostgreSQL.
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)Aurimas Mikalauskas Is my MySQL server configured properly? Should I run Community MySQL, MariaDB, Percona or WebScaleSQL? How many innodb buffer pool instances should I run? Why should I NOT use the query cache? How do I size the innodb log file size and what IS that innodb log anyway? All answers are inside.
Aurimas Mikalauskas is a former Percona performance consultant and architect currently writing and teaching at speedemy.com. He's been involved with MySQL since 1999, scaling and optimizing MySQL backed systems since 2004 for companies such as BBC, EngineYard, famous social networks and small shops like EstanteVirtual, Pine Cove and hundreds of others.
Additional content mentioned in the presentation can be found here: http://speedemy.com/17
Optimizing Apache Spark SQL Joins
Optimizing Apache Spark SQL JoinsDatabricks Join operations in Apache Spark is often the biggest source of performance problems and even full-blown exceptions in Spark. After this talk, you will understand the two most basic methods Spark employs for joining DataFrames – to the level of detail of how Spark distributes the data within the cluster. You’ll also find out how to work out common errors and even handle the trickiest corner cases we’ve encountered! After this talk, you should be able to write performance joins in Spark SQL that scale and are zippy fast!
This session will cover different ways of joining tables in Apache Spark.
Speaker: Vida Ha
This talk was originally presented at Spark Summit East 2017.
A Deep Dive into Query Execution Engine of Spark SQL
A Deep Dive into Query Execution Engine of Spark SQLDatabricks Spark SQL enables Spark to perform efficient and fault-tolerant relational query processing with analytics database technologies. The relational queries are compiled to the executable physical plans consisting of transformations and actions on RDDs with the generated Java code. The code is compiled to Java bytecode, executed at runtime by JVM and optimized by JIT to native machine code at runtime. This talk will take a deep dive into Spark SQL execution engine. The talk includes pipelined execution, whole-stage code generation, UDF execution, memory management, vectorized readers, lineage based RDD transformation and action.
Similar to MongoDB sharded cluster. How to design your topology ? (20)
From HDFS to S3: Migrate Pinterest Apache Spark Clusters
From HDFS to S3: Migrate Pinterest Apache Spark ClustersDatabricks The document discusses Pinterest migrating their Apache Spark clusters from HDFS to S3 storage. Some key points:
1) Migrating to S3 provided significantly better performance due to the higher IOPS of modern EC2 instances compared to their older HDFS nodes. Jobs saw 25-35% improvements on average.
2) S3 is eventually consistent while HDFS is strongly consistent, so they implemented the S3Committer to handle output consistency issues during job failures.
3) Metadata operations like file moves were very slow in S3, so they optimized jobs to reduce unnecessary moves using techniques like multipart uploads to S3.
Lambda Architecture: The Best Way to Build Scalable and Reliable Applications!
Lambda Architecture: The Best Way to Build Scalable and Reliable Applications!Tugdual Grall Lambda Architecture is a useful framework to think about designing big data applications. This framework has been built initially at Twitter. In this presentation you will learn, based on concrete examples how to build deploy scalable and fault tolerant applications, with a focus on Big Data and Hadoop.
This presentation was delivered at the OOP conference, Munich, Feb 2016
Healthcare Claim Reimbursement using Apache Spark
Healthcare Claim Reimbursement using Apache SparkDatabricks The document discusses rewriting a claims reimbursement system using Spark. It describes how Spark provides better performance, scalability and cost savings compared to the previous Oracle-based system. Key points include using Spark for ETL to load data into a Delta Lake data lake, implementing the business logic in a reusable Java library, and seeing significant increases in processing volumes and speeds compared to the prior system. Challenges and tips for adoption are also provided.
Spark to DocumentDB connector
Spark to DocumentDB connectorDenny Lee Combining the awesomeness of Apache Spark and Azure DocumentDB together for real-time machine learning on globally-distributed data.
ClickHouse on Plug-n-Play Cloud, by Som Sikdar, Kodiak Data
ClickHouse on Plug-n-Play Cloud, by Som Sikdar, Kodiak DataAltinity Ltd Kodiak provides a private cloud solution called MemCloud that offers faster performance and lower costs compared to public clouds like AWS. MemCloud can be deployed on-premises or at the edge to power analytics, big data, and AI/ML workloads. Benchmarks show the Kodiak solution is up to 5x faster than AWS for similar configurations. It also reduces the complexity, costs, and maintenance challenges of building and tuning physical data lake clusters that combine different software like HDFS, Kafka, Spark and ClickHouse.
Working with SAP Business Warehouse Elements in SAP Datasphere_.pdf
Working with SAP Business Warehouse Elements in SAP Datasphere_.pdfPanduM7 SAP Datasphere enables a business data fabric architecture that uniquely harmonizes mission-critical data throughout the organization, unleashing business experts to make the most impactful decisions. It combines previously discrete capabilities into a unified service for data integration, cataloging, semantic modeling and data-warehousing. It also virtualizes workloads spanning SAP and non-SAP data. SAP Datasphere preserves the full meaning and context of SAP data across systems and clouds. It integrates with other data vendor’s platforms, delivering seamless and scalable access to one authoritative source for your most valuable enterprise data
Analyzing Real-World Data with Apache Drill
Analyzing Real-World Data with Apache DrillTomer Shiran The document describes a demo of analyzing real-world data using Apache Drill. The demo involves running Drill, configuring storage plugins for HDFS and MongoDB, and exploring sample data from Yelp including reviews, users, and business data stored in JSON files and MongoDB collections. Queries are run against this data using SQL to analyze basics as well as complex data structures.
The Modern Data Team for the Modern Data Stack: dbt and the Role of the Analy...
The Modern Data Team for the Modern Data Stack: dbt and the Role of the Analy...Databricks A traditional data team has roles including data engineer, data scientist, and data analyst. However, many organizations are finding success by integrating a new role – the analytics engineer. The analytics engineer develops a code-based data infrastructure that can serve both analytics and data science teams. He or she develops re-usable data models using the software engineering practices of version control and unit testing, and provides the critical domain expertise that ensures that data products are relevant and insightful. In this talk we’ll talk about the role and skill set of the analytics engineer, and discuss how dbt, an open source programming environment, empowers anyone with a SQL skillset to fulfill this new role on the data team. We’ll demonstrate how to use dbt to build version-controlled data models on top of Delta Lake, test both the code and our assumptions about the underlying data, and orchestrate complete data pipelines on Apache Spark™.
How The Weather Company Uses Apache Spark to Serve Weather Data Fast at Low Cost
How The Weather Company Uses Apache Spark to Serve Weather Data Fast at Low CostDatabricks The Weather Company (TWC) collects weather data across the globe at the rate of 34 million records per hour, and the TWC History on Demand application serves that historical weather data to users via an API, averaging 600,000 requests per day. Users are increasingly consuming large quantities of historical data to train analytics models, and require efficient asynchronous APIs in addition to existing synchronous ones which use ElasticSearch. We present our architecture for asynchronous data retrieval and explain how we use Spark together with leading edge technologies to achieve an order of magnitude cost reduction while at the same time boosting performance by several orders of magnitude and tripling weather data coverage from land only to global.
The SparkSQL things you maybe confuse
The SparkSQL things you maybe confusevito jeng SparkSQL, SchemaRDD, DataFrame, and Dataset are Apache Spark APIs for structured data processing. SparkSQL is a high-level module introduced in Spark 1.0. SchemaRDD was introduced in Spark 1.0 from the Shark project and was later renamed to DataFrame in Spark 1.3. Dataset, introduced experimentally in Spark 1.6, allows SparkSQL optimizations while working with RDDs. DataFrame and Dataset were unified under a single API in Spark 2.0.
Intro to Data Vault 2.0 on Snowflake
Intro to Data Vault 2.0 on SnowflakeKent Graziano This document provides an introduction and overview of implementing Data Vault 2.0 on Snowflake. It begins with an agenda and the presenter's background. It then discusses why customers are asking for Data Vault and provides an overview of the Data Vault methodology including its core components of hubs, links, and satellites. The document applies Snowflake features like separation of workloads and agile warehouse scaling to support Data Vault implementations. It also addresses modeling semi-structured data and building virtual information marts using views.
IDERA Slides: Managing Complex Data Environments
IDERA Slides: Managing Complex Data EnvironmentsDATAVERSITY Companies are expanding their information systems beyond relational databases to incorporate big data and cloud deployments, creating hybrid configurations. Database professionals have the challenges of managing multiple data sources and running queries for analytics against diverse databases in these complex environments.
IDERA’s Lisa Waugh will discuss how to deal with the growing challenges of having data residing on different database platforms by using a single IDE.
Foundations strata sf-2019_final
Foundations strata sf-2019_finalJonathan Seidman The document summarizes key considerations for managing successful data projects, including understanding the problem, selecting appropriate software, managing risk, building effective teams, and architecting maintainable solutions. It covers major data project types like data pipelines, processing, and applications. It also discusses evaluating and selecting data management solutions by considering factors like solution lifecycles, tipping points, demand, fit, visibility, and risks. The overall goal is to provide foundations for architecting successful data solutions.
SQL-on-Hadoop with Apache Drill
SQL-on-Hadoop with Apache DrillMapR Technologies The open source project Apache Drill gives you SQL-on-Hadoop, but with some big differences. The biggest difference is that Drill extends ANSI SQL from a strongly typed language to also a late binding language without losing performance. This allows Drill to process complex structured data like JSON in addition to relational data. By dynamically generating a schema at read time that matches the data types and structures observed in the data, Drill gives you both self-service agility and speed.
Drill also introduces a view-based security model that uses file system permissions to control access to data at an extremely fine-grained level that makes secure access easy to control. These extensions have huge practical impact when it comes to writing real applications.
In these slides, Tugdual Grall, Technical Evangelist at MapR, gives several practical examples of how Drill makes it easy to analyze data, using SQL in your Java application with a simple JDBC driver.
Empower Hive with Spark
Empower Hive with SparkDataWorks Summit This document discusses empowering Apache Hive with Apache Spark. It provides background on Hive and Spark, outlines the architecture and design principles for integrating the two, discusses challenges, and provides benchmarks comparing performance of Hive on Spark versus Hive on MapReduce and Tez. The key points are: Hive on Spark allows reusing existing Hive code and features while benefiting from Spark's improved performance; efforts involved contributions from both Hive and Spark communities; benchmarks on 320GB and 4TB datasets showed Hive on Spark was sometimes faster than Hive on Tez and significantly faster than Hive on MapReduce, though Tez performance improved with dynamic partition pruning not yet implemented for Hive on Spark.
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar.
In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR.
Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios.
Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects.
Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.
Fun with Fabric in 15
Fun with Fabric in 15Neo4j Fabric allows distributing different data across multiple Neo4j databases and querying across them through a single query. The demo sets up a local Neo4j database with Stack Overflow data and connects to a remote AuraDB instance. A Fabric database is configured to connect to these two shards. Queries are run directly against each shard and through the Fabric database to retrieve results from both in a single query. The results are intelligently combined by collecting the results into maps and merging matching entries.
Qubism and scala nlp
Qubism and scala nlpJerome Banks Talk given at the SF Scala online meetup on 3/26/2020
https://www.meetup.com/SF-Scala/events/268998404/
Intro to Apache Spark
Intro to Apache SparkCloudera, Inc. Spark is a fast and general engine for large-scale data processing. It provides APIs in Java, Scala, and Python and an interactive shell. Spark applications operate on resilient distributed datasets (RDDs) that can be cached in memory for faster performance. RDDs are immutable and fault-tolerant via lineage graphs. Transformations create new RDDs from existing ones while actions return values to the driver program. Spark's execution model involves a driver program that coordinates tasks on executor machines. RDD caching and lineage graphs allow Spark to efficiently run jobs across clusters.
MySQL Conference 2011 -- The Secret Sauce of Sharding -- Ryan Thiessen
MySQL Conference 2011 -- The Secret Sauce of Sharding -- Ryan Thiessenryanthiessen This document discusses sharding strategies at Facebook. It describes Facebook's "universal database" approach which shards MySQL databases across multiple instances per physical host and groups related objects together. It also covers shard management, hashing techniques, re-sharding processes, and operational implications of sharding such as increased monitoring challenges and harder schema changes and upgrades.
Ad
More from Mydbops (20)
Scaling TiDB for Large-Scale Application
Scaling TiDB for Large-Scale ApplicationMydbops Mydbops MyWebinar 42: Scaling TiDB for Large-Scale Applications
Presenter: Kabilesh P.R., Founding Partner, Mydbops
Is your database slowing down as your business grows?
Scaling databases is a challenge, especially when dealing with high traffic and large workloads. TiDB is designed for scalability, but without the right approach, you may face slow queries, downtime, and migration hurdles.
Join Kabilesh P.R. as he shares real-world use cases, proven strategies, and common pitfalls in scaling TiDB for large applications. This session will help you understand how to optimize performance, improve reliability, and scale seamlessly.
What You'll Learn:
* How to scale TiDB efficiently for large applications
* Common mistakes in scaling and how to avoid them
* Real-world case studies of successful migrations
* Best practices for maintaining performance and reliability
https://www.mydbops.com/
[email protected]
AWS MySQL Showdown - RDS vs RDS Multi AZ vs Aurora vs Serverless - Mydbops...
AWS MySQL Showdown - RDS vs RDS Multi AZ vs Aurora vs Serverless - Mydbops...Mydbops AWS MySQL Showdown - RDS vs RDS Multi AZ vs Aurora vs Serverless - Mydbops Webinar 41
Key takeaways:
* Performance & Scalability – How each service handles workloads
* High Availability & Failover – Ensuring uptime and reliability
* Cost & Efficiency – Which solution gives the best value
* Architecture Deep Dive – Comparing Multi-AZ RDS and Aurora’s distributed model
Who Should Attend?
* Database Architects & Engineers
* DevOps & Cloud Professionals
* CTOs & Tech Decision-Makers
Don't miss out!
#aws #mysql #rds #aurora #serverless #cloud #database #scalability #highavailability #performance #cloudcomputing #devops #tech #engineering #webinar #automation #costoptimization #failover #replication #opensource #datamanagement #cloudarchitecture #cloudservices #datastorage #techcommunity #itprofessionals #dba #event #community #databasemanagement
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: [email protected]
Visit: https://www.mydbops.com/
Mastering Vector Search with MongoDB Atlas - Manosh Malai - Mydbops MyWebinar 39
Mastering Vector Search with MongoDB Atlas - Manosh Malai - Mydbops MyWebinar 39Mydbops Mastering Vector Search with MongoDB Atlas - Manosh Malai - Mydbops MyWebinar 39
In this session, explore how to harness MongoDB's native vector search capabilities to enhance your database and search functionality. From the basics to advanced techniques, gain insights into building intelligent solutions that drive innovation.
What You’ll Learn:
* The fundamentals of vector search in MongoDB Atlas.
* How to store vector embeddings and create efficient indexes.
* Performing similarity queries for applications like semantic search and personalized recommendations.
* Best practices for optimizing performance and scaling vector-based systems effectively.
Whether you’re a developer, data scientist, or database administrator, this webinar will equip you with practical skills to elevate your projects with MongoDB’s advanced features.
Download presentation here: https://www.mydbops.com/webinars/mastering-vector-search-with-mongodb-atlas
This webinar is ideal for database administrators, data engineers, system architects, and anyone involved in MongoDB database management.
#Webinar #mongodb #mongodbatlas #MyWebinar #Mydbops #DatabaseManagement #DevOps #TechWebinar #database #dbms #dba #vectorsearch
Migration Journey To TiDB - Kabilesh PR - Mydbops MyWebinar 38
Migration Journey To TiDB - Kabilesh PR - Mydbops MyWebinar 38Mydbops Migration Journey To TiDB - Kabilesh PR - Mydbops MyWebinar 38
Youtube video link: https://youtu.be/_WgXm1Ykj8c
What You Will Learn
* Data Migration Strategies – Understand the best approaches for transferring data to TiDB with minimal disruption.
* Seamless Replication – Learn how to maintain data consistency and minimize downtime during the migration process.
* Schema Design Adjustments – Explore the key schema design adjustments necessary for optimal TiDB performance.
* Challenges & Solutions – Gain practical insights into tackling common migration challenges to ensure a smooth transition.
This webinar is ideal for database administrators, data engineers, system architects, and anyone involved in database management and migrations. Whether you are considering TiDB as a new solution or already exploring it, this session will equip you with valuable knowledge to streamline your migration journey.
#Webinar #TiDB #MyWebinar #Mydbops #DatabaseManagement #migration #DevOps #TechWebinar #database #dbms #dba #distributedsql #sql #HTAP
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, PostgreSQL and TiDB.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: [email protected]
Visit: https://www.mydbops.com/
AWS Blue Green Deployment for Databases - Mydbops
AWS Blue Green Deployment for Databases - MydbopsMydbops Mastering AWS Blue/Green Deployment for Databases - Mydbops MyWebinar 37
What You Will Learn
* Key Principles of Blue/Green Deployment: Understand the fundamental concepts that drive this deployment strategy.
* Step-by-Step Implementation: A detailed walkthrough of the processes involved in setting up Blue/Green deployments using AWS services.
* Best Practices: Discover industry best practices to minimize risks and avoid common pitfalls during deployments.
* Database Management with AWS: Learn how to effectively use AWS services like RDS and Aurora for safe database upgrades, including rollback options in the event of deployment issues.
This webinar is ideal for database administrators, DevOps engineers, cloud architects, and anyone interested in mastering AWS deployment strategies. Whether you are new to AWS or looking to enhance your skills, this session will provide valuable insights and practical knowledge.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: [email protected]
Visit: https://www.mydbops.com/
What's New In MySQL 8.4 LTS Mydbops MyWebinar Edition 36
What's New In MySQL 8.4 LTS Mydbops MyWebinar Edition 36Mydbops What's New in MySQL 8.4? Mydbops MyWebinar Edition 36 - Vinoth Kanna, Founding Partner, Mydbops
Join us as we explore the latest advancements in MySQL 8.4 and discover how these updates can enhance your database management.
Key highlights:
* GTID Tags for improved replication
* Automatic histogram updates for query optimization
* Clone Plugin for faster replication
* Backward-compatible backups with mysqldump
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: [email protected]
What's New in PostgreSQL 17? - Mydbops MyWebinar Edition 35
What's New in PostgreSQL 17? - Mydbops MyWebinar Edition 35Mydbops What's New in PostgreSQL 17? - Mydbops MyWebinar Edition 35
Key Features of PostgreSQL 17:
• Discover how PostgreSQL 17 has optimized performance, making your queries run faster and more efficiently.
• Learn about the new indexing techniques that provide quicker access to data and reduce the load on your system.
• Explore the expanded support for various data types, allowing for more flexibility in how you store and manipulate data.
• PostgreSQL 17 introduces new functions that simplify data manipulation and enhance your ability to handle complex queries.
• Understand the improvements in logical replication that make data synchronization more robust and easier to manage.
• Get insights into the latest security enhancements designed to protect your data more effectively than ever before.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: [email protected]
Visit: https://www.mydbops.com/
What's New in MongoDB 8.0 - Mydbops MyWebinar Edition 34
What's New in MongoDB 8.0 - Mydbops MyWebinar Edition 34Mydbops What's New in MongoDB 8.0 - Mydbops MyWebinar Edition 34
* Performance Enhancements: Discover the impressive speed boosts in write and read performance, with benchmarks showing up to a 54% improvement in write-heavy workloads and a 27% improvement in read-heavy workloads.
* Time Series Enhancements: Learn about the new block processing feature and the transition to columnar storage, which promises faster queries and smarter use of storage space.
* Command Path Optimization: Understand the major overhaul of the command path for faster response times and more efficient database operations.
* Express Path Efficiency: Explore the new Express Path designed to optimize specific queries for speed and reduced overhead.
* Resource Efficiency: Learn about the reduced memory fragmentation and enhanced peak load behavior for better overall system performance.
* Advanced Sharding Capabilities: Discover the new capabilities for moving and converting collections between shards.
* Queryable Encryption Enhancements: Gain insights into the support for range queries within encrypted fields, enhancing security and functionality.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: [email protected]
Visit: https://www.mydbops.com/
Follow us on LinkedIn: https://in.linkedin.com/company/mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsMydbops This presentation, delivered at the Postgres Bangalore (PGBLR) Meetup-2 on June 29th, 2024, dives deep into connection pooling for PostgreSQL databases. Aakash M, a PostgreSQL Tech Lead at Mydbops, explores the challenges of managing numerous connections and explains how connection pooling optimizes performance and resource utilization.
Key Takeaways:
* Understand why connection pooling is essential for high-traffic applications
* Explore various connection poolers available for PostgreSQL, including pgbouncer
* Learn the configuration options and functionalities of pgbouncer
* Discover best practices for monitoring and troubleshooting connection pooling setups
* Gain insights into real-world use cases and considerations for production environments
This presentation is ideal for:
* Database administrators (DBAs)
* Developers working with PostgreSQL
* DevOps engineers
* Anyone interested in optimizing PostgreSQL performance
Contact [email protected] for PostgreSQL Managed, Consulting and Remote DBA Services
Read/Write Splitting using MySQL Router - Mydbops Meetup16
Read/Write Splitting using MySQL Router - Mydbops Meetup16Mydbops Read/Write Splitting using MySQL Router - Mydbops Meetup16
Topic: Scale Your Database Traffic with Read/Write Splitting Using MySQL Router
Date & Time: 8th June | 10 AM - 1 PM IST
Abstract:
This session dives deep into the power of Read/Write splitting, a technique that significantly improves database performance and application scalability.
* Challenges of managing read/write workloads on a single server.
* How MySQL Router enables transparent read/write splitting.
* Step-by-step guidance for implementation in your MySQL environment.
* Real-world use cases and benefits.
* No code changes required!
TiDB - From Data to Discovery: Exploring the Intersection of Distributed Dat...
TiDB - From Data to Discovery: Exploring the Intersection of Distributed Dat...Mydbops Speaker: Sreedharma Vijayan, India Director at PingCAP, at Mydbops Open Source Meetup 16.
Topic: From Data to Discovery: Exploring the Intersection of Distributed Databases and AI
Date & Time: 8th June | 10 AM - 1 PM IST
In this session, Sreedharma Vijayan delves into the exciting intersection of distributed databases and AI.
You'll discover how TiDB empowers digital industries to:
* Handle explosive data growth that challenges traditional databases.
* Optimize data distribution and access logic within your applications.
* Unlock valuable insights to fuel AI-powered workflows.
MySQL InnoDB Storage Engine: Deep Dive - Mydbops
MySQL InnoDB Storage Engine: Deep Dive - MydbopsMydbops This presentation, titled "MySQL - InnoDB" and delivered by Mayank Prasad at the Mydbops Open Source Database Meetup 16 on June 8th, 2024, covers dynamic configuration of REDO logs and instant ADD/DROP columns in InnoDB.
This presentation dives deep into the world of InnoDB, exploring two ground-breaking features introduced in MySQL 8.0:
• Dynamic Configuration of REDO Logs: Enhance your database's performance and flexibility with on-the-fly adjustments to REDO log capacity. Unleash the power of the snake metaphor to visualize how InnoDB manages REDO log files.
• Instant ADD/DROP Columns: Say goodbye to costly table rebuilds! This presentation unveils how InnoDB now enables seamless addition and removal of columns without compromising data integrity or incurring downtime.
Key Learnings:
• Grasp the concept of REDO logs and their significance in InnoDB's transaction management.
• Discover the advantages of dynamic REDO log configuration and how to leverage it for optimal performance.
• Understand the inner workings of instant ADD/DROP columns and their impact on database operations.
• Gain valuable insights into the row versioning mechanism that empowers instant column modifications.
Demystifying Real time Analytics with TiDB
Demystifying Real time Analytics with TiDBMydbops Are you struggling to gain real-time insights from your data?
Mydbops MyWebinar Edition 33 can help you.
Discover how TiDB can revolutionize your analytics game!
Topic: Demystifying Real-Time Analytics with TiDB
Presenter: Kabilesh PR, Founding Partner, Mydbops
In today's data-driven world, real-time analytics is essential for businesses to make quick decisions based on immediate insights. This webinar will explore how TiDB empowers organizations to unlock the full potential of their data. We'll delve into TiDB's powerful capabilities, including:
• Hybrid Transactional/Analytical Processing (HTAP): Run high-speed transactions and complex queries simultaneously without sacrificing performance.
• Real-time Analytics: Gain immediate insights from your data to make informed decisions faster.
• Scalability & Flexibility: Effortlessly scale your database to accommodate growing data volumes.
Download our previous webinar presentations here for free: https://www.mydbops.com/webinars
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top open-source databases: MySQL, MongoDB, MariaDB, PostgreSQL, TiDB and Cassandra.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: [email protected]
Visit: https://www.mydbops.com/
Follow us on LinkedIn: / mydbops
Blogs: https://www.mydbops.com/blog/
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMydbops Mydbops Opensource Database Meetup 16
Topic: Must-Know PostgreSQL Extensions for Developers and DBAs During Migration
Speaker: Deepak Mahto, Founder of DataCloudGaze Consulting
Date & Time: 8th June | 10 AM - 1 PM IST
Venue: Bangalore International Centre, Bangalore
Abstract: Discover how PostgreSQL extensions can be your secret weapon! This talk explores how key extensions enhance database capabilities and streamline the migration process for users moving from other relational databases like Oracle.
Key Takeaways:
* Learn about crucial extensions like oracle_fdw, pgtt, and pg_audit that ease migration complexities.
* Gain valuable strategies for implementing these extensions in PostgreSQL to achieve license freedom.
* Discover how these key extensions can empower both developers and DBAs during the migration process.
* Don't miss this chance to gain practical knowledge from an industry expert and stay updated on the latest open-source database trends.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: [email protected]
Visit: https://www.mydbops.com/
Follow us on LinkedIn: https://in.linkedin.com/company/mydbops
For more details and updates, please follow up the below links.
Meetup Page : https://www.meetup.com/mydbops-databa...
Twitter: https://twitter.com/mydbopsofficial
Blogs: https://www.mydbops.com/blog/
Facebook(Meta): https://www.facebook.com/mydbops/
Efficient MySQL Indexing and what's new in MySQL Explain
Efficient MySQL Indexing and what's new in MySQL ExplainMydbops Efficient MySQL Indexing & What's New in MySQL Explain - Mydbops MyWebinar Edition 32
This session will delve into:
• Strategic indexing techniques: Learn how to optimize your MySQL database by implementing effective indexing strategies, including when to avoid fulltext indexes to prevent wasted resources.
• Demystifying the new MySQL Explain: We'll explore the latest enhancements to the MySQL Explain plan's JSON output format. Discover how to store the output in a variable for further analysis – a valuable addition introduced in MySQL 8.3. You'll also learn about the explain_json_format_version variable, which empowers you to choose between different JSON output versions for greater flexibility.
• Live Chat Engagement: We encourage you to actively participate throughout the webinar! Use the chat functionality to ask questions and share your experiences with indexing and Explain.
This webinar is perfect for:
• Database administrators (DBAs)
• Developers
• Anyone seeking to optimize MySQL performance and streamline database queries
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: [email protected]
Visit: https://www.mydbops.com/
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops Scale your database traffic with Read & Write split using MySQL Router
This webinar recording dives into the world of MySQL Router and its capabilities for effectively managing high database traffic loads.
You'll learn:
• The challenges of scaling database traffic
• How MySQL Router facilitates read/write splitting
• The benefits of implementing read/write splitting
• Step-by-step demonstrations for configuring MySQL Router for:
1. Static read/write routing for standalone servers
2. Dynamic read/write split for InnoDB Cluster & Replica Set
• A comparison of popular load balancers (MySQL Router, ProxySQL, Maxscale)
Mydbops is a trusted database management and consultancy provider, helping businesses achieve optimal database performance and scalability.
Connect with Mydbops!
Website: https://www.mydbops.com/
Email: [email protected]
PostgreSQL Schema Changes with pg-osc - Mydbops @ PGConf India 2024
PostgreSQL Schema Changes with pg-osc - Mydbops @ PGConf India 2024Mydbops Title: PostgreSQL Schema Changes with Minimal Downtime using pg_osc
Speaker: Aakash M, Mydbops
Event: PGConf India, 2024
Description:
This presentation explores pg_osc, a tool that enables efficient schema changes in PostgreSQL tables with minimal downtime and locking. It addresses the challenges of traditional ALTER statements and provides a smoother alternative.
Key points covered:
• Introduction to pg_osc and its benefits.
• Limitations of ALTER statements and how pg_osc overcomes them.
• Step-by-step explanation of the pg_osc process.
• Prominent features and considerations for using pg_osc.
• References and resources for further exploration.
Target Audience:
• Database administrators
• Developers working with PostgreSQL
• Anyone interested in optimizing schema changes
This presentation provides valuable insights for anyone seeking to streamline schema modifications in PostgreSQL while minimizing disruptions.
Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...
Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...Mydbops Choosing the Right Database: Exploring MySQL Alternatives for Modern Applications by Bhanu Jamwal, Head of Solution Engineering, PingCAP at the Mydbops Opensource Database Meetup 14.
This presentation discusses the challenges in choosing the right database for modern applications, focusing on MySQL alternatives. It highlights the growth of new applications, the need to improve infrastructure, and the rise of cloud-native architecture.
The presentation explores alternatives to MySQL, such as MySQL forks, database clustering, and distributed SQL. It introduces TiDB as a distributed SQL database for modern applications, highlighting its features and top use cases.
Case studies of companies benefiting from TiDB are included. The presentation also outlines TiDB's product roadmap, detailing upcoming features and enhancements.
Mastering Aurora PostgreSQL Clusters for Disaster Recovery
Mastering Aurora PostgreSQL Clusters for Disaster RecoveryMydbops The presentation "Mastering Aurora PostgreSQL Clusters for Disaster Recovery" by Bhuvanesh, Co-Founder & CTO of ShellKode, at the Mydbops OpenSource Database Meetup 14 covers advanced topics in managing Aurora PostgreSQL clusters for disaster recovery purposes.
Bhuvanesh discusses key features of Aurora, such as its decoupled storage and compute layers, auto scaling capabilities, and native replication, highlighting its benefits over traditional RDS instances. He also explores Aurora Global Databases, explaining how they enable replication of data across regions for geo-span applications with low latency.
The presentation includes architecture details, such as physical and log replication, and managed failover options for ensuring high availability. Bhuvanesh shares real-world experiences and best practices for managing Aurora clusters, including handling replication lag and TLS certificate management.
Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open So...
Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open So...Mydbops Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open Source Database Meetup 15
Shivji explores the evolution of transactions, implementation challenges, and insights into distributed database environments. Whether you're a database enthusiast or a tech enthusiast, this presentation offers valuable insights into the world of database management.
Contents:
• Historical perspective of transactions
• Implementing transactions
• Challenges and trade-offs in ACID properties
• Distributed transactions in modern databases like Amazon Aurora, DynamoDB, and Google Spanner
Key Takeaways:
• Understanding the evolution of transactions in databases
• Insights into the challenges of implementing ACID properties
• Exploration of distributed transaction models in leading database systems
Ad
Recently uploaded (20)
Linux Support for SMARC: How Toradex Empowers Embedded Developers
Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex Toradex brings robust Linux support to SMARC (Smart Mobility Architecture), ensuring high performance and long-term reliability for embedded applications. Here’s how:
• Optimized Torizon OS & Yocto Support – Toradex provides Torizon OS, a Debian-based easy-to-use platform, and Yocto BSPs for customized Linux images on SMARC modules.
• Seamless Integration with i.MX 8M Plus and i.MX 95 – Toradex SMARC solutions leverage NXP’s i.MX 8 M Plus and i.MX 95 SoCs, delivering power efficiency and AI-ready performance.
• Secure and Reliable – With Secure Boot, over-the-air (OTA) updates, and LTS kernel support, Toradex ensures industrial-grade security and longevity.
• Containerized Workflows for AI & IoT – Support for Docker, ROS, and real-time Linux enables scalable AI, ML, and IoT applications.
• Strong Ecosystem & Developer Support – Toradex offers comprehensive documentation, developer tools, and dedicated support, accelerating time-to-market.
With Toradex’s Linux support for SMARC, developers get a scalable, secure, and high-performance solution for industrial, medical, and AI-driven applications.
Do you have a specific project or application in mind where you're considering SMARC? We can help with Free Compatibility Check and help you with quick time-to-market
For more information: https://www.toradex.com/computer-on-modules/smarc-arm-family
Rusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1 Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark?
At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍
Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency.
This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data.
Attendees will learn:
- Consumer awareness around data brokers and what consumers are doing to limit data collection
- How businesses assess third-party vendors and their consent management operations
- Where business preparedness needs improvement
- What these trends mean for the future of privacy governance and public trust
This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next.
Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/
Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsInData Labs In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy.
This infographic contains:
-AI and data privacy: Key findings
-Statistics on AI data privacy in the today’s world
-Tips on how to overcome data privacy challenges
-Benefits of AI data security investments.
Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.
Mobile App Development Company in Saudi Arabia
Mobile App Development Company in Saudi ArabiaSteve Jonas EmizenTech is a globally recognized software development company, proudly serving businesses since 2013. With over 11+ years of industry experience and a team of 200+ skilled professionals, we have successfully delivered 1200+ projects across various sectors. As a leading Mobile App Development Company In Saudi Arabia we offer end-to-end solutions for iOS, Android, and cross-platform applications. Our apps are known for their user-friendly interfaces, scalability, high performance, and strong security features. We tailor each mobile application to meet the unique needs of different industries, ensuring a seamless user experience. EmizenTech is committed to turning your vision into a powerful digital product that drives growth, innovation, and long-term success in the competitive mobile landscape of Saudi Arabia.
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55 We’re bringing the TDX energy to our community with 2 power-packed sessions:
🛠️ Workshop: MuleSoft for Agentforce
Explore the new version of our hands-on workshop featuring the latest Topic Center and API Catalog updates.
📄 Talk: Power Up Document Processing
Dive into smart automation with MuleSoft IDP, NLP, and Einstein AI for intelligent document workflows.
Quantum Computing Quick Research Guide by Arthur Morgan
Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan This is a Quick Research Guide (QRG).
QRGs include the following:
- A brief, high-level overview of the QRG topic.
- A milestone timeline for the QRG topic.
- Links to various free online resource materials to provide a deeper dive into the QRG topic.
- Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic.
QRGs planned for the series:
- Artificial Intelligence QRG
- Quantum Computing QRG
- Big Data Analytics QRG
- Spacecraft Guidance, Navigation & Control QRG (coming 2026)
- UK Home Computing & The Birth of ARM QRG (coming 2027)
Any questions or comments?
- Please contact Arthur Morgan at [email protected].
100% human made.
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john Analyze the growth of meme coins from mere online jokes to potential assets in the digital economy. Explore the community, culture, and utility as they elevate themselves to a new era in cryptocurrency.
TrsLabs - Fintech Product & Business Consulting
TrsLabs - Fintech Product & Business ConsultingTrs Labs Hybrid Growth Mandate Model with TrsLabs
Strategic Investments, Inorganic Growth, Business Model Pivoting are critical activities that business don't do/change everyday. In cases like this, it may benefit your business to choose a temporary external consultant.
An unbiased plan driven by clearcut deliverables, market dynamics and without the influence of your internal office equations empower business leaders to make right choices.
Getting things done within a budget within a timeframe is key to Growing Business - No matter whether you are a start-up or a big company
Talk to us & Unlock the competitive advantage
HCL Nomad Web – Best Practices and Managing Multiuser Environments
HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/
HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client.
Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience.
In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including
- Accessing the console
- Locating and interpreting log files
- Accessing the data folder within the browser’s cache (using OPFS)
- Understand the difference between single- and multi-user scenarios
- Utilizing Client Clocking
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBrian McKeiver May 2nd, 2025 talk at StirTrek 2025 Conference.
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp This is the keynote of the Into the Box conference, highlighting the release of the BoxLang JVM language, its key enhancements, and its vision for the future.
Andrew Marnell: Transforming Business Strategy Through Data-Driven Insights
Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell With expertise in data architecture, performance tracking, and revenue forecasting, Andrew Marnell plays a vital role in aligning business strategies with data insights. Andrew Marnell’s ability to lead cross-functional teams ensures businesses achieve sustainable growth and operational excellence.
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityriccardosl1 Cyber awareness training educates employees on risk associated with internet and malicious emails
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptx
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock Building 10x Organizations with Modern Productivity Metrics
10x developers may be a myth, but 10x organizations are very real, as proven by the influential study performed in the 1980s, ‘The Coding War Games.’
Right now, here in early 2025, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience. It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them.
But which of these approaches actually work? DORA? SPACE? DevEx? What should we invest in and create urgency behind today, so that we don’t find ourselves having the same discussion again in a decade?
Linux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru Introduction to LPIC-1 Exam - overview, exam details, price and job opportunities
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok In today's fast-paced retail environment, efficiency is key. Every minute counts, and every penny matters. One tool that can significantly boost your store's efficiency is a well-executed planogram. These visual merchandising blueprints not only enhance store layouts but also save time and money in the process.
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat
MongoDB sharded cluster. How to design your topology ?
- 1. © 2019 Percona1 Vinodh Krishnaswamy MongoDB Sharded Cluster - How to design your topology Support Engineer - MongoDB & MySQL Percona
- 2. © 2019 Percona2 ◆ Started as MySQL DB DBA ◆ Support for MongoDB ◆ Trainer ◆ Scripting, Reading, Driving ◆ “Guruji” - who shares knowledge ◆ Now here I am - Perconian!!! Who am I
- 3. © 2019 Percona3 Agenda ▪ Sharded Cluster - components ▪ Ranged Sharding ▪ Hashed Sharding ▪ Zones / Tags
- 4. © 2019 Percona4 Agenda ▪ ReplicaSet - Key points ▪ Topology ▪ Q&A
- 5. © 2019 Percona5 Sharded Cluster
- 6. © 2019 Percona6 Sharded Cluster ▪ Horizontal Scaling • Data across multiple nodes • Reduce stress for larger working set • Ability to add more servers • HA
- 8. © 2019 Percona8 Sharded Cluster ▪ Shard Key ▪ Chunks ▪ Balancer - Chunk distribution
- 9. © 2019 Percona9 POINTS TO NOTE TARGET vs BROADCAST
- 10. © 2019 Percona10 Primary Shard A A
- 11. © 2019 Percona11 Primary Shard sh.status()
- 12. © 2019 Percona12 Primary Shard ▪ Routing - sharded vs unsharded ▪ Keep at right shard db.adminCommand( { movePrimary: <databaseName>, to: <newPrimaryShard> } )
- 13. © 2019 Percona13 Ranged Sharding
- 14. © 2019 Percona14 Strategy ▪ Ranged Sharding
- 15. © 2019 Percona15 Strategy ▪ Ranged Sharding sh.shardCollection( "database.collection", { <shard key> } ) Example: sh.shardCollection( "vinodh.testData", { empNo: 1 } )
- 16. © 2019 Percona16 Monotonically Changing Shard Key
- 17. © 2019 Percona17 Ranged Sharding ▪ Contiguous ranges determined by Shard Key ▪ Default method ▪ Efficient when • High cardinality • Low Shard Key frequency • Non-Monotonically Changing Shard key
- 18. © 2019 Percona18 Hashed Sharding
- 19. © 2019 Percona19 Strategy ▪ Hashed Sharding
- 20. © 2019 Percona20 Strategy ▪ Hashed Sharding uses hashed index sh.shardCollection( "database.collection", { <field> : “hashed” } ) Example: sh.shardCollection( "vinodh.testData", { UUIDfield: “hashed” } )
- 21. © 2019 Percona21 Monotonically Changing Shard Key - Hash
- 22. © 2019 Percona22 Hashed Sharding ▪ Provides equal distribution ▪ Target vs Broadcast operations ▪ Ideal for fields that changes monotonically like ObjectID, timestamp etc
- 24. © 2019 Percona24 Why Zones? ▪ Isolate data set ▪ Geographically closest ▪ Route data to desired Shard h/w ▪ Distribute Writes based on DC
- 25. © 2019 Percona25 How to... ▪ First create tagRange with Shards sh.addShardTag("shard01", "tag1") sh.addShardTag("shard02", "tag2") ▪ Assign tagRanges to collections sh.addTagRange( "<database>.<collection>", { "gender" : "Male", "userid" : MinKey }, { "gender" : "Male", "userid" : MaxKey }, "tag1")
- 28. © 2019 Percona28 ReplicaSet - Key Points
- 31. © 2019 Percona31 Replicaset - ARBITER
- 32. © 2019 Percona32 Replicaset - Election+Priority ELECTION - VOTING & CHOOSING Priority: 5Priority: 1 Priority: 1 Priority: 5
- 33. © 2019 Percona33 Points to Note ▪ Set proper {Priority:n} to elect a member as Primary • Avoid setting Priority for all members ▪ Set {writeConcern:majority} to avoid data loss ▪ Set readPreference for reads
- 34. © 2019 Percona34 Topology- Application Servers
- 35. © 2019 Percona35 Applications and mongos in one server
- 36. © 2019 Percona36 mongos/router in each Application servers
- 37. © 2019 Percona37 Sharded Cluster in Private network
- 38. © 2019 Percona38 Topology- Shards
- 39. © 2019 Percona39 Shard instances deployed in dedicated servers design
- 40. © 2019 Percona40 Multiple Shard members in each Server + HA design
- 41. © 2019 Percona41 Production DC and DR DC + HA design
- 42. © 2019 Percona42 Using Zones / Tags + HA design
- 43. © 2019 Percona43 Points to Note
- 44. © 2019 Percona44 POINTS TO NOTE ▪ SHARD KEY - Pain Point! • Monotonically increased key • Do your tests! ▪ SHARD EARLY! ▪ ZONES
- 45. © 2019 Percona45 POINTS TO NOTE BACKUP + RESTORE https://www.percona.com/blog/2018/12/13/mong odb-backup-how-and-when-to-use-psmdb-hotbac kup-and-mongodb_consistent_backup/
- 47. © 2019 Percona47 THANK YOU