Mongo db v3_deep_dive
Download as pptx, pdf0 likes481 views
The document provides an overview of MongoDB storage engines and new features in version 3.0. It introduces the new WiredTiger storage engine, which offers document-level concurrency, compression, and improved performance over MMAPv1. It also discusses indexing improvements, consistency without journaling in WiredTiger, and how to upgrade MongoDB deployments to leverage the new engine.
1 of 45
Downloaded 15 times
![33
Indexing Improvements
1 4
2 2
3 5
4 7
5 4
6 9
7 3
8 6
9 2
10 1
11 1
12 5
13 4
15 5
14 5
{
_id: 6,
categories: [
“database”,
“distributed”,
“document store”
]
}](https://image.slidesharecdn.com/mongodbv3deepdive-150902215912-lva1-app6891/85/Mongo-db-v3_deep_dive-33-320.jpg)
![34
Indexing Improvements
1 4
2 2
3 5
4 7
5 4
6 9
7 3
8 6
9 2
10 1
11 1
12 5
13 4
15 5
14 5
{$push:
{
categories:
“sharded”
}
}
{
_id: 6,
categories: [
“database”,
“distributed”,
“document store”,
“sharded”
]
}](https://image.slidesharecdn.com/mongodbv3deepdive-150902215912-lva1-app6891/85/Mongo-db-v3_deep_dive-34-320.jpg)
![35
Indexing Improvements
1 4
2 2
3 5
4 7
5 4
6 9
7 3
8 6
9 2
10 1
11 1
12 5
13 4
15 5
14 5
{
_id: 6,
categories: [
“database”,
“distributed”,
“document store”,
“sharded”
]
}](https://image.slidesharecdn.com/mongodbv3deepdive-150902215912-lva1-app6891/85/Mongo-db-v3_deep_dive-35-320.jpg)
Ad
Recommended
User Defined Partitioning on PlazmaDB
User Defined Partitioning on PlazmaDBKai Sasaki User defined partitioning is a new partitioning strategy in Treasure Data that allows users to specify which column to use for partitioning, in addition to the default "time" column. This provides more flexible partitioning that better fits customer data platform workloads. The user can define partitioning rules through Presto or Hive to improve query performance by enabling colocated joins and filtering data by the partitioning column.
Don't change the partition count for kafka topics!
Don't change the partition count for kafka topics!Dainius Jocas The document discusses how increasing the partition count for a Kafka topic caused a "Heisenbug" by changing the hash function used to assign messages to partitions, violating the ordering guarantee within partitions. This led to Elasticsearch indexing messages out of order and failing to delete documents as expected. The bug was fixed by fully reingesting the data into a new Kafka cluster with a consistent partition count. The key lesson is not to change the partition count if an application relies on ordering of messages within a topic.
Presto At Treasure Data
Presto At Treasure DataTaro L. Saito Presto is used to process 15 trillion rows per day for Treasure Data customers. Treasure Data developed tools to manage Presto performance and optimize queries. They collect Presto query logs to analyze performance bottlenecks and classify queries to set implicit service level objectives. Tools like Prestobase Proxy and Presto Stella storage optimizer were created to improve low-latency access and optimize storage partitioning. Workflows using DigDag and a new tabular data format called MessageFrame are being explored to split huge queries and support incremental processing.
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PGConf APAC Devrim Gunduz gives a presentation on Write-Ahead Logging (WAL) in PostgreSQL. WAL logs all transactions to files called write-ahead logs (WAL files) before changes are written to data files. This allows for crash recovery by replaying WAL files. WAL files are used for replication, backup, and point-in-time recovery (PITR) by replaying WAL files to restore the database to a previous state. Checkpoints write all dirty shared buffers to disk and update the pg_control file with the checkpoint location.
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on KubernetesHBaseCon The document discusses building an online HBase cluster for Zhihu using Kubernetes, detailing the architecture, resource management, and challenges faced. It emphasizes the need for environment isolation, SLA definitions, and resource balance while implementing a unified cluster management system. The benefits highlighted include easy isolation, flexible resource configuration, and enhanced performance through customization and container management.
DataEngConf SF16 - High cardinality time series search
DataEngConf SF16 - High cardinality time series searchHakka Labs The document discusses high cardinality time series search and scaling to large datasets. It describes the author's company which collects and analyzes machine data at terabytes per day. General purpose search systems are good for moderate scale but have limitations for high cardinality data with large retention. The author's company built Rocana Search to optimize for their time-oriented event data, with features like parallel ingestion and querying, dynamic partitioning, and keeping all data online without wasted resources. It can handle billions of events per day with low latency and utilizes modern hardware through full distribution.
ClickHouse in Real Life. Case Studies and Best Practices, by Alexander Zaitsev
ClickHouse in Real Life. Case Studies and Best Practices, by Alexander ZaitsevAltinity Ltd The document discusses the real-life applications and best practices of ClickHouse, a high-performance analytics database, highlighting its capabilities for fast analytical queries, low-latency data ingestion, and fault-tolerant data warehousing. It presents various case studies from industries such as adtech, fintech, and blockchain, showcasing ClickHouse's successful deployments and migration from other database systems. Key design decisions for using ClickHouse include efficient data ingestion, star-schema designs, and the use of materialized views and dictionaries to enhance performance.
Lessons PostgreSQL learned from commercial databases, and didn’t
Lessons PostgreSQL learned from commercial databases, and didn’tPGConf APAC PostgreSQL has made significant advancements over the years, establishing itself as a robust open-source alternative to commercial databases. Its strengths include a sound architecture and a close-to-standard SQL design, although it faces challenges including write performance and database size limitations compared to Oracle and DB2. Despite these challenges, PostgreSQL remains a great database option, especially due to its supportive community and continual improvement.
SQL Server In-Memory OLTP: What Every SQL Professional Should Know
SQL Server In-Memory OLTP: What Every SQL Professional Should KnowBob Ward The document discusses the evolution and features of SQL Server's in-memory OLTP, starting from early Hekaton developments in 2007 to the enhancements introduced in SQL Server 2016. It covers key concepts like index types, memory optimization, transaction management, and the architecture of in-memory OLTP, along with practical advice on implementation and performance tuning. Additionally, it includes references to resources for learning more about in-memory OLTP and its applications in Azure SQL Database.
Seattle Cassandra Meetup - HasOffers
Seattle Cassandra Meetup - HasOffersbtoddb This document summarizes Apache Cassandra configuration at HasOffers for supporting high write throughput and fault tolerance across multiple datacenters. Keypoints include Cassandra being configured with two column families, multiple secondary indexes, and a replication factor of 1 across 9 nodes in 3 datacenters. HasOffers sees approximately 13.8 million daily inserts replicated across the datacenters, with 3 million daily queries. Daily insert volumes are shown by datacenter. Additional topics covered include AWS instance types used, disk options like EBS RAID vs provisioned IOPS, and client configuration using Pycassa.
Cassandra Community Webinar: Apache Cassandra Internals
Cassandra Community Webinar: Apache Cassandra InternalsDataStax This document outlines the architecture and internal workings of Apache Cassandra, emphasizing components such as API services, data transport layers, and database operations. It provides insights into the interaction between different layers, including the Dynamo layer and the database layer functionalities. The presentation is led by Aaron Morton, a co-founder of The Last Pickle, highlighting his extensive experience with Cassandra.
Breaking the Sound Barrier with Persistent Memory
Breaking the Sound Barrier with Persistent Memory HBaseCon The document discusses the benefits of persistent memory in overcoming performance limitations related to disk writes and data management. It presents experimental results demonstrating a significant increase in throughput and a reduction in response time when utilizing persistent memory, highlighting its potential advantages for systems like HBase. Additionally, it addresses current software support and outlines potential solutions to achieve higher bandwidth with lower latency in data transactions.
Cassandra Day Atlanta 2015: Diagnosing Problems in Production
Cassandra Day Atlanta 2015: Diagnosing Problems in ProductionDataStax Academy Diagnosing Problems in Production involves first preparing monitoring tools like OpsCenter, Server monitoring, Application metrics, and Log aggregation. Common issues include incorrect server times causing data inconsistencies, tombstone overhead slowing queries, not using the right snitch, version mismatches breaking functionality, and disk space not being reclaimed properly. Diagnostic tools like htop, iostat, vmstat, dstat, strace, jstack, tcpdump and nodetool can help narrow down issues like performance bottlenecks, garbage collection problems, and compaction issues.
Troubleshooting redis
Troubleshooting redisDaeMyung Kang This document discusses troubleshooting Redis. Some key points:
- Redis is single-threaded, so commands like KEYS, FlushAll, and deleting large collections can be slow. It's better to use SCAN instead of KEYS.
- Creating Redis database snapshots (RDB files) and rewriting the append-only file (AOF) can cause high disk I/O and CPU usage. It's best to disable automatic rewrites.
- Monitoring memory usage and fragmentation is important to avoid performance issues. The maxmemory setting also needs monitoring to prevent out-of-memory errors.
- Network and replication failures need solutions like DNS failover or using Zookeeper for coordination to maintain high availability of Redis
Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015N Masahiro This document discusses Treasure Data's data architecture. It describes how Treasure Data collects and imports log data using Fluentd. The data is stored in columnar format in S3 and metadata is stored in PostgreSQL. Treasure Data uses Presto to enable fast analytics on the large datasets. The document provides details on the import process, storage, partitioning, and optimizations to improve query performance.
Presto in Treasure Data (presented at db tech showcase Sapporo 2015)
Presto in Treasure Data (presented at db tech showcase Sapporo 2015)Mitsunori Komatsu Presto is a distributed SQL query engine designed for low latency and high performance without disk I/O, supporting ANSI SQL for ad-hoc querying. The document discusses its architecture, pros and cons, limitations, and provides examples of query plans and usage with various connectors, including those for Cassandra, Hive, and MySQL. Key limitations include challenges with large joins, lack of fault tolerance, and absence of native ODBC support.
Apache Kafka: New Features That You Might Not Know About
Apache Kafka: New Features That You Might Not Know AboutYaroslav Tkachenko The document outlines new features introduced in various versions of Apache Kafka, including message headers in version 0.11 and transactions for atomic writes. It discusses improvements in message delivery guarantees, specifically the 'exactly-once' delivery mechanism, and enhancements in Kafka Streams. Additionally, it highlights controller improvements and shutdown procedures in version 1.1, resulting in reduced downtime.
Introduction to Mongodb execution plan and optimizer
Introduction to Mongodb execution plan and optimizerMydbops The document provides an overview of MongoDB's execution plans and optimizers, including how queries are processed and analyzed during execution. It explains various modes of the `explain()` function, detailing the process of query planning, execution stages, and metrics for analyzing performance. Additionally, the document discusses plan caching and its impacts on efficiency when executing queries in MongoDB.
Hoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkVinoth Chandar The document discusses Uber's development of an analytical datastore using Spark, addressing challenges with large-scale data ingestion and querying. It outlines the design choices for achieving scalability, low latency, and consistency while considering various storage formats and indexing options. Additionally, it introduces Hoodie, an open-source framework for managing and querying large datasets efficiently, and invites collaboration and inquiries on future enhancements.
re:dash is awesome
re:dash is awesomeHiroshi Toyama re:dash is a tool for sharing SQL queries, visualizing results, and scheduling automated refreshes. It supports connecting to various data sources, provides a low-cost option on AWS, and enables caching of query results for improved performance. Key features include sharing queries with team members, running queries on a schedule, connecting to backends like PostgreSQL, and programming visualizations and parameters through the HTTP API. It also focuses on security features such as authentication, authorization, auditing, and SSL encryption.
Building a Complex, Real-Time Data Management Application
Building a Complex, Real-Time Data Management ApplicationJonathan Katz The document discusses the construction of a real-time data management application utilizing PostgreSQL, focusing on managing room availability at a hotel. It highlights various PostgreSQL features such as range types and procedural languages, necessary for building applications that handle CRUD operations efficiently. Additionally, it explores the implementation of logical decoding for replicating changes and emphasizes the importance of optimizing the application's scalability and performance.
Low latency stream processing with jet
Low latency stream processing with jetStreamNative The document discusses low-latency stream processing using Hazelcast Jet, an open-source engine designed for distributed stream and batch processing. It highlights features such as event-time based processing, fault tolerance, and performance benchmarks, emphasizing the importance of immediate data usage and complex event processing. Additionally, it mentions integration with Pulsar and outlines a roadmap for future developments including full SQL support and expanded computational capabilities.
Scaling with sync_replication using Galera and EC2
Scaling with sync_replication using Galera and EC2Marco Tusa The document discusses scaling MySQL using multi-master synchronous replication, specifically through Galera and its suitability for a customer's geographical and operational needs. It details the architecture design, installation, performance testing, and the challenges faced with replication and node management in a cloud environment. The conclusion emphasizes the successful implementation of a flexible and stable solution that meets the customer's requirements for scalability and performance.
Using apache spark for processing trillions of records each day at Datadog
Using apache spark for processing trillions of records each day at DatadogVadim Semenov This document discusses Datadog's use of Apache Spark to process trillions of records daily. It describes their initial Spark setup using AWS EMR with large clusters. It then covers common out of memory errors, measuring memory usage, handling spot instances, and lessons learned around monitoring jobs and ensuring resilience.
Cloud dwh
Cloud dwhAlexander Tokarev The document outlines strategies for optimizing cloud databases, particularly focusing on performance tuning and effective data modeling for various platforms like Redshift, BigQuery, and Snowflake. Key topics include common failures during data migration, pricing models, optimal data structures, and the importance of constraints. The content emphasizes that while cloud solutions offer scalability and cost-effectiveness, they require careful management and understanding of infrastructure to avoid excessive costs and inefficiencies.
Relational databases for BigData
Relational databases for BigDataAlexander Tokarev The document discusses big data and open-source relational database management systems (RDBMSs). It begins by introducing the speaker and their background in cloud architecture and enterprise databases. The rest of the document summarizes various RDBMS options for big data, including PostgreSQL and extensions like Citus and Timescale, as well as columnar databases like Greenplum and Clickhouse. It compares the systems based on their SQL support, integration capabilities, data ingestion styles and performance, support for wide tables versus star schemas, horizontal scaling abilities, and availability as managed cloud services.
Open Policy Agent for governance as a code
Open Policy Agent for governance as a code Alexander Tokarev The document discusses the implementation and promotion of standards in software engineering, focusing on tools like Open Policy Agent (OPA) for policy-based control and validation processes. It outlines compliance maturity levels, key use cases of OPA, and various integrations with Kubernetes and other technologies. The document also highlights challenges, examples of policy implementation, and the importance of automated validations in ensuring software delivery efficiency.
KSQL: Streaming SQL for Kafka
KSQL: Streaming SQL for Kafkaconfluent KSQL is a streaming SQL engine for Apache Kafka that simplifies stream processing, making it accessible for non-developers using a SQL-based interface. It allows for real-time data transformations, anomaly detection, and monitoring through easy-to-use commands and interactive development. The document covers KSQL concepts, deployment examples, and the roadmap for future enhancements.
Performance Enhancements In Postgre Sql 8.4
Performance Enhancements In Postgre Sql 8.4HighLoad2009 PostgreSQL 8.4 introduced several performance enhancements including optimizations to anti-joins, semi-joins, hash aggregation, and new free space map and visibility map features. It also included application-level improvements such as subqueries in LIMIT/OFFSET clauses, window functions, common table expressions, and parallel restore. Many changes provided performance benefits transparently to applications or DBAs while some required application changes to realize gains.
Why PG deserves noSQL fans' respect
Why PG deserves noSQL fans' respectdivarvel The document discusses why PostgreSQL deserves recognition from NoSQL enthusiasts, highlighting its capabilities such as ACID transactions, schemaless design, and native JSON support. It emphasizes the importance of choosing the right database architecture, especially in handling complex data exploration and relationships. The author, Clement Delafargue from Clever Cloud, encourages developers to explore PostgreSQL's features and offers a free development plan.
More Related Content
What's hot (20)
SQL Server In-Memory OLTP: What Every SQL Professional Should Know
SQL Server In-Memory OLTP: What Every SQL Professional Should KnowBob Ward The document discusses the evolution and features of SQL Server's in-memory OLTP, starting from early Hekaton developments in 2007 to the enhancements introduced in SQL Server 2016. It covers key concepts like index types, memory optimization, transaction management, and the architecture of in-memory OLTP, along with practical advice on implementation and performance tuning. Additionally, it includes references to resources for learning more about in-memory OLTP and its applications in Azure SQL Database.
Seattle Cassandra Meetup - HasOffers
Seattle Cassandra Meetup - HasOffersbtoddb This document summarizes Apache Cassandra configuration at HasOffers for supporting high write throughput and fault tolerance across multiple datacenters. Keypoints include Cassandra being configured with two column families, multiple secondary indexes, and a replication factor of 1 across 9 nodes in 3 datacenters. HasOffers sees approximately 13.8 million daily inserts replicated across the datacenters, with 3 million daily queries. Daily insert volumes are shown by datacenter. Additional topics covered include AWS instance types used, disk options like EBS RAID vs provisioned IOPS, and client configuration using Pycassa.
Cassandra Community Webinar: Apache Cassandra Internals
Cassandra Community Webinar: Apache Cassandra InternalsDataStax This document outlines the architecture and internal workings of Apache Cassandra, emphasizing components such as API services, data transport layers, and database operations. It provides insights into the interaction between different layers, including the Dynamo layer and the database layer functionalities. The presentation is led by Aaron Morton, a co-founder of The Last Pickle, highlighting his extensive experience with Cassandra.
Breaking the Sound Barrier with Persistent Memory
Breaking the Sound Barrier with Persistent Memory HBaseCon The document discusses the benefits of persistent memory in overcoming performance limitations related to disk writes and data management. It presents experimental results demonstrating a significant increase in throughput and a reduction in response time when utilizing persistent memory, highlighting its potential advantages for systems like HBase. Additionally, it addresses current software support and outlines potential solutions to achieve higher bandwidth with lower latency in data transactions.
Cassandra Day Atlanta 2015: Diagnosing Problems in Production
Cassandra Day Atlanta 2015: Diagnosing Problems in ProductionDataStax Academy Diagnosing Problems in Production involves first preparing monitoring tools like OpsCenter, Server monitoring, Application metrics, and Log aggregation. Common issues include incorrect server times causing data inconsistencies, tombstone overhead slowing queries, not using the right snitch, version mismatches breaking functionality, and disk space not being reclaimed properly. Diagnostic tools like htop, iostat, vmstat, dstat, strace, jstack, tcpdump and nodetool can help narrow down issues like performance bottlenecks, garbage collection problems, and compaction issues.
Troubleshooting redis
Troubleshooting redisDaeMyung Kang This document discusses troubleshooting Redis. Some key points:
- Redis is single-threaded, so commands like KEYS, FlushAll, and deleting large collections can be slow. It's better to use SCAN instead of KEYS.
- Creating Redis database snapshots (RDB files) and rewriting the append-only file (AOF) can cause high disk I/O and CPU usage. It's best to disable automatic rewrites.
- Monitoring memory usage and fragmentation is important to avoid performance issues. The maxmemory setting also needs monitoring to prevent out-of-memory errors.
- Network and replication failures need solutions like DNS failover or using Zookeeper for coordination to maintain high availability of Redis
Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015N Masahiro This document discusses Treasure Data's data architecture. It describes how Treasure Data collects and imports log data using Fluentd. The data is stored in columnar format in S3 and metadata is stored in PostgreSQL. Treasure Data uses Presto to enable fast analytics on the large datasets. The document provides details on the import process, storage, partitioning, and optimizations to improve query performance.
Presto in Treasure Data (presented at db tech showcase Sapporo 2015)
Presto in Treasure Data (presented at db tech showcase Sapporo 2015)Mitsunori Komatsu Presto is a distributed SQL query engine designed for low latency and high performance without disk I/O, supporting ANSI SQL for ad-hoc querying. The document discusses its architecture, pros and cons, limitations, and provides examples of query plans and usage with various connectors, including those for Cassandra, Hive, and MySQL. Key limitations include challenges with large joins, lack of fault tolerance, and absence of native ODBC support.
Apache Kafka: New Features That You Might Not Know About
Apache Kafka: New Features That You Might Not Know AboutYaroslav Tkachenko The document outlines new features introduced in various versions of Apache Kafka, including message headers in version 0.11 and transactions for atomic writes. It discusses improvements in message delivery guarantees, specifically the 'exactly-once' delivery mechanism, and enhancements in Kafka Streams. Additionally, it highlights controller improvements and shutdown procedures in version 1.1, resulting in reduced downtime.
Introduction to Mongodb execution plan and optimizer
Introduction to Mongodb execution plan and optimizerMydbops The document provides an overview of MongoDB's execution plans and optimizers, including how queries are processed and analyzed during execution. It explains various modes of the `explain()` function, detailing the process of query planning, execution stages, and metrics for analyzing performance. Additionally, the document discusses plan caching and its impacts on efficiency when executing queries in MongoDB.
Hoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkVinoth Chandar The document discusses Uber's development of an analytical datastore using Spark, addressing challenges with large-scale data ingestion and querying. It outlines the design choices for achieving scalability, low latency, and consistency while considering various storage formats and indexing options. Additionally, it introduces Hoodie, an open-source framework for managing and querying large datasets efficiently, and invites collaboration and inquiries on future enhancements.
re:dash is awesome
re:dash is awesomeHiroshi Toyama re:dash is a tool for sharing SQL queries, visualizing results, and scheduling automated refreshes. It supports connecting to various data sources, provides a low-cost option on AWS, and enables caching of query results for improved performance. Key features include sharing queries with team members, running queries on a schedule, connecting to backends like PostgreSQL, and programming visualizations and parameters through the HTTP API. It also focuses on security features such as authentication, authorization, auditing, and SSL encryption.
Building a Complex, Real-Time Data Management Application
Building a Complex, Real-Time Data Management ApplicationJonathan Katz The document discusses the construction of a real-time data management application utilizing PostgreSQL, focusing on managing room availability at a hotel. It highlights various PostgreSQL features such as range types and procedural languages, necessary for building applications that handle CRUD operations efficiently. Additionally, it explores the implementation of logical decoding for replicating changes and emphasizes the importance of optimizing the application's scalability and performance.
Low latency stream processing with jet
Low latency stream processing with jetStreamNative The document discusses low-latency stream processing using Hazelcast Jet, an open-source engine designed for distributed stream and batch processing. It highlights features such as event-time based processing, fault tolerance, and performance benchmarks, emphasizing the importance of immediate data usage and complex event processing. Additionally, it mentions integration with Pulsar and outlines a roadmap for future developments including full SQL support and expanded computational capabilities.
Scaling with sync_replication using Galera and EC2
Scaling with sync_replication using Galera and EC2Marco Tusa The document discusses scaling MySQL using multi-master synchronous replication, specifically through Galera and its suitability for a customer's geographical and operational needs. It details the architecture design, installation, performance testing, and the challenges faced with replication and node management in a cloud environment. The conclusion emphasizes the successful implementation of a flexible and stable solution that meets the customer's requirements for scalability and performance.
Using apache spark for processing trillions of records each day at Datadog
Using apache spark for processing trillions of records each day at DatadogVadim Semenov This document discusses Datadog's use of Apache Spark to process trillions of records daily. It describes their initial Spark setup using AWS EMR with large clusters. It then covers common out of memory errors, measuring memory usage, handling spot instances, and lessons learned around monitoring jobs and ensuring resilience.
Cloud dwh
Cloud dwhAlexander Tokarev The document outlines strategies for optimizing cloud databases, particularly focusing on performance tuning and effective data modeling for various platforms like Redshift, BigQuery, and Snowflake. Key topics include common failures during data migration, pricing models, optimal data structures, and the importance of constraints. The content emphasizes that while cloud solutions offer scalability and cost-effectiveness, they require careful management and understanding of infrastructure to avoid excessive costs and inefficiencies.
Relational databases for BigData
Relational databases for BigDataAlexander Tokarev The document discusses big data and open-source relational database management systems (RDBMSs). It begins by introducing the speaker and their background in cloud architecture and enterprise databases. The rest of the document summarizes various RDBMS options for big data, including PostgreSQL and extensions like Citus and Timescale, as well as columnar databases like Greenplum and Clickhouse. It compares the systems based on their SQL support, integration capabilities, data ingestion styles and performance, support for wide tables versus star schemas, horizontal scaling abilities, and availability as managed cloud services.
Open Policy Agent for governance as a code
Open Policy Agent for governance as a code Alexander Tokarev The document discusses the implementation and promotion of standards in software engineering, focusing on tools like Open Policy Agent (OPA) for policy-based control and validation processes. It outlines compliance maturity levels, key use cases of OPA, and various integrations with Kubernetes and other technologies. The document also highlights challenges, examples of policy implementation, and the importance of automated validations in ensuring software delivery efficiency.
KSQL: Streaming SQL for Kafka
KSQL: Streaming SQL for Kafkaconfluent KSQL is a streaming SQL engine for Apache Kafka that simplifies stream processing, making it accessible for non-developers using a SQL-based interface. It allows for real-time data transformations, anomaly detection, and monitoring through easy-to-use commands and interactive development. The document covers KSQL concepts, deployment examples, and the roadmap for future enhancements.
Viewers also liked (6)
Performance Enhancements In Postgre Sql 8.4
Performance Enhancements In Postgre Sql 8.4HighLoad2009 PostgreSQL 8.4 introduced several performance enhancements including optimizations to anti-joins, semi-joins, hash aggregation, and new free space map and visibility map features. It also included application-level improvements such as subqueries in LIMIT/OFFSET clauses, window functions, common table expressions, and parallel restore. Many changes provided performance benefits transparently to applications or DBAs while some required application changes to realize gains.
Why PG deserves noSQL fans' respect
Why PG deserves noSQL fans' respectdivarvel The document discusses why PostgreSQL deserves recognition from NoSQL enthusiasts, highlighting its capabilities such as ACID transactions, schemaless design, and native JSON support. It emphasizes the importance of choosing the right database architecture, especially in handling complex data exploration and relationships. The author, Clement Delafargue from Clever Cloud, encourages developers to explore PostgreSQL's features and offers a free development plan.
Advanced Replication Internals
Advanced Replication InternalsMongoDB This document discusses the design and goals of advanced replication in MongoDB. The goals are high availability, consistent data, automatic failover across multiple regions/data centers, and dynamic reads. The design includes replicating all databases to each node, using quorum and elections for consistency, smart clients, source selection awareness, recording operations in an oplog, and asynchronous replication with write acknowledgements.
Mongo db3.0 wired_tiger_storage_engine
Mongo db3.0 wired_tiger_storage_engineKenny Gorman The document discusses the features and improvements introduced in MongoDB 3.0, particularly focusing on the WiredTiger storage engine. Key highlights include pluggable storage engine capabilities, enhanced performance through better configuration options, and trade-offs related to data compression techniques. The author emphasizes the importance of understanding the impact of these changes on specific workloads for effective adoption.
Postgre sql intro 0
Postgre sql intro 0March Liu 本文讨论了 PostgreSQL 作为开源关系型数据库的强大功能与广泛应用，包括其数据模型、编程能力及运维技术，适用于不同架构需求。文中介绍了PostgreSQL的独特数据类型、并发性能及全文检索功能，并探讨了语义模型的实现和应用案例。最后，提供了一些安装和运行 PostgreSQL 的实操指导。
Webinar slides: Become a MongoDB DBA - What to Monitor (if you’re really a My...
Webinar slides: Become a MongoDB DBA - What to Monitor (if you’re really a My...Severalnines The document outlines a webinar on MongoDB monitoring led by a support engineer, covering topics like trending and alerting, key metrics, and comparative monitoring between MongoDB and MySQL. It emphasizes the importance of monitoring system health, understanding various metrics (host, operational, storage engine, replication), and utilizing open-source tools such as Nagios and Zabbix for effective database management. The discussion includes practical tips on health checks, capacity planning, and monitoring implementations using solutions like ClusterControl and Percona Monitoring and Management.
Ad
Similar to Mongo db v3_deep_dive (20)
Low Level CPU Performance Profiling Examples
Low Level CPU Performance Profiling ExamplesTanel Poder The document discusses low-level CPU performance profiling and modern CPU efficiency, emphasizing the importance of optimized data structures for improved performance. It covers various techniques and examples, including profiling methodologies for Oracle and Apache Spark environments, highlighting different data access paths and their effects on CPU usage. The content concludes by advocating for future-proof open data formats, such as Apache Parquet and Arrow, to enhance data processing efficiency.
11g R2
11g R2afa reg - Oracle Database 11g Release 2 provides many advanced features to lower IT costs including in-memory processing, automated storage management, database compression, and real application testing capabilities.
- It allows for online application upgrades using edition-based redefinition which allows new code and data changes to be installed without disrupting the existing system.
- Oracle provides multiple upgrade paths from prior database versions to 11g to allow for predictable performance and a safe upgrade process.
Let the Tiger Roar!
Let the Tiger Roar!MongoDB The document promotes MongoDB 3.0, featuring a pluggable storage engine API that includes options like mmapv1 and WiredTiger, offering improved performance and flexibility for different workloads. WiredTiger brings document-level concurrency, compression, and enhanced performance, while also being compatible with existing MongoDB operations. It highlights the benefits of using MongoDB Ops Manager for efficient management, and encourages early registration for an upcoming event with discounts available.
Kafka to the Maxka - (Kafka Performance Tuning)
Kafka to the Maxka - (Kafka Performance Tuning)DataWorks Summit The document provides a comprehensive guide on performance tuning for Kafka, highlighting the importance of load testing and empirical data to inform configuration changes. Key aspects include JVM settings, file system choices, batch management, and monitoring tools, alongside caution about modifying default settings without thorough consideration. It emphasizes that there is no one-size-fits-all solution, urging experimentation and careful testing to optimize Kafka performance.
Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...
Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...javier ramirez The document outlines strategies for designing a high-performance database capable of ingesting over four million events per second, emphasizing the importance of modern hardware, efficient data storage, and managing dependencies. It discusses unique patterns in time-series data, including ingestion and querying strategies, while highlighting technical decisions that support performance, such as using efficient memory management and custom SQL implementations. The author also provides examples of performance benchmarks and future improvements planned for the database system.
Hands-on Lab: How to Unleash Your Storage Performance by Using NVM Express™ B...
Hands-on Lab: How to Unleash Your Storage Performance by Using NVM Express™ B...Odinot Stanislas The document is a comprehensive guide on optimizing storage performance using NVMe-based PCI Express SSDs, discussing essential topics such as NVMe technology, performance measurement, RAID configurations, and monitoring tools. It outlines practical steps for checking the functionality of NVMe drives, measuring their performance with various metrics, and utilizing Intel's SSD Data Center Tool and Fio for testing and troubleshooting. Additional sections address RAID setup with Intel's Rapid Storage Technology and performance monitoring capabilities using visualization tools.
Architecture at Scale
Architecture at ScaleElasticsearch The document outlines best practices for architecting Elastic systems at scale, emphasizing the importance of security, data normalization, and effective index management. It discusses shard management, resource allocation, and automated policies for data lifecycle management, while encouraging the use of templates and aliases for easier index creation. The content also addresses multi-cluster management, cross-cluster replication, and orchestration options for deploying Elastic solutions across various infrastructures.
SQL Server It Just Runs Faster
SQL Server It Just Runs FasterBob Ward The document discusses significant enhancements in SQL Server 2016, focusing on performance improvements such as faster I/O, better scalability, and advanced algorithms including in-memory OLTP and columnstore indexes. Key features include automatic configurations for file setups, improved parallel processing, and enhanced support for large multi-core NUMA systems. Additionally, it highlights advancements in spatial data handling, backup compression, and encryption techniques to optimize efficiency across various workloads.
MongoDB: How We Did It – Reanimating Identity at AOL
MongoDB: How We Did It – Reanimating Identity at AOLMongoDB The document details AOL's implementation of MongoDB to improve data management, overcoming challenges like functional ambiguity and scaling. It discusses their migration strategy, performance testing, deployment details, and various collections, including user identities and relationships. Key lessons learned emphasize optimizing connections, minimizing read delays, and ensuring failovers are handled efficiently.
SUE 2018 - Migrating a 130TB Cluster from Elasticsearch 2 to 5 in 20 Hours Wi...
SUE 2018 - Migrating a 130TB Cluster from Elasticsearch 2 to 5 in 20 Hours Wi...Fred de Villamil The document details the migration of a 130TB Elasticsearch cluster from version 2 to 5 in 20 hours without downtime, showcasing the infrastructure and technology at Ledger and Synthesio. It covers the architecture, node configurations, and challenges faced during the migration, including solutions that utilized segmented data zones and replication factor adjustments. Additionally, it emphasizes the importance of careful planning and resource management in large-scale database operations.
Managing your Black Friday Logs - Antonio Bonuccelli - Codemotion Rome 2018
Managing your Black Friday Logs - Antonio Bonuccelli - Codemotion Rome 2018Codemotion The document outlines Antonio Bonuccelli's presentation on managing logs with the Elastic Stack, focusing on logging architectures, cluster sizing, and optimization strategies. Key topics include the importance of shard management, bulk API usage, and the scaling of data handling solutions. The document emphasizes practical tips for effectively structuring data storage and processing within an Elasticsearch environment.
Introduction to Apache Kafka
Introduction to Apache KafkaShiao-An Yuan This document provides an introduction to Apache Kafka. It describes Kafka as a distributed messaging system with features like durability, scalability, publish-subscribe capabilities, and ordering. It discusses key Kafka concepts like producers, consumers, topics, partitions and brokers. It also summarizes use cases for Kafka and how to implement producers and consumers in code. Finally, it briefly outlines related tools like Kafka Connect and Kafka Streams that build upon the Kafka platform.
https://docs.google.com/presentation/d/1DcL4zK6i3HZRDD4xTGX1VpSOwyu2xBeWLT6a_...
https://docs.google.com/presentation/d/1DcL4zK6i3HZRDD4xTGX1VpSOwyu2xBeWLT6a_...MongoDB The document serves as a technical introduction to the WiredTiger storage engine developed for MongoDB, highlighting its unique features such as document-level concurrency, in-memory performance, and multi-core scalability. It explains the architecture of WiredTiger, including its use of multiversion concurrency control, checksums for data integrity, and various compression techniques. The document also outlines future developments for WiredTiger aimed at enhancing performance and adding new functionalities.
MySQL Manchester TT - 5.7 Whats new
MySQL Manchester TT - 5.7 Whats newMark Swarbrick The document discusses performance improvements and new features in MySQL 5.7. Key points include:
- MySQL 5.7 shows significant performance gains over previous versions, with benchmarks showing 3x faster query performance and 82% faster connection throughput.
- New features include enhanced replication, a new optimizer cost model, performance schema improvements, and native JSON support.
- The optimizer and parser were refactored for improved maintainability, readability and stability. A new cost-based optimizer model provides better performance and tunability.
Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning
Webinar slides: Our Guide to MySQL & MariaDB Performance TuningSeveralnines The document is a comprehensive guide on tuning MySQL and MariaDB performance, detailing the tuning process, configuration settings, and operating system optimizations necessary for optimal database performance. It emphasizes the importance of a test environment, the iterative nature of tuning, and provides practical tips and tools for effective performance management. Additionally, it covers updates in MySQL 8.0 and offers resources for further learning.
Silicon Valley Code Camp 2015 - Advanced MongoDB - The Sequel
Silicon Valley Code Camp 2015 - Advanced MongoDB - The SequelDaniel Coupal The presentation focuses on advanced practices for developing, deploying, and operating applications using MongoDB, targeting developers familiar with the basics. Key topics include the importance of schema design, performance tuning, deployment topologies, and operational best practices to prevent common issues. It also emphasizes using available resources and ongoing support from MongoDB to enhance application success.
MySQL 5.6 - Operations and Diagnostics Improvements
MySQL 5.6 - Operations and Diagnostics ImprovementsMorgan Tocker This document discusses MySQL 5.6 and its improvements to operational and diagnostic capabilities. Key enhancements include online DDL operations that do not block reads or writes, buffer pool dump and restore for faster startup, import/export of partitioned tables, and transportable tablespaces. Diagnostic tools were improved with EXPLAIN showing more details, the ability to EXPLAIN updates and deletes, optimizer tracing, and the performance schema providing detailed query level instrumentation and monitoring by default.
Datadog: a Real-Time Metrics Database for One Quadrillion Points/Day
Datadog: a Real-Time Metrics Database for One Quadrillion Points/DayC4Media The document presents an in-depth overview of Datadog's metrics database, detailing its architecture, data handling processes, and the various performance strategies it employs. It covers the significance of metrics databases, data synchronization, and the use of tagging to enhance data insights, along with performance mantras to optimize system efficiency. Additionally, it describes the challenges and solutions for managing massive data volumes and maintaining quick query responses in a cloud environment.
QuestDB: ingesting a million time series per second on a single instance. Big...
QuestDB: ingesting a million time series per second on a single instance. Big...javier ramirez QuestDB is optimized for time-series data, allowing ingestion of over a million rows per second and querying speeds scanning billions of rows. It focuses solely on time-series problems, leveraging unique storage models and a dedicated architecture for performance. The document outlines its ingestion capabilities, efficiency in handling vast amounts of data, and distinctive features suitable for businesses requiring rapid data processing.
Taking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout SessionSplunk This document provides an overview of scaling a Splunk deployment from an initial use case to a larger enterprise deployment. It discusses growing use cases and data volume over time. The agenda covers use case mapping, simple scaling approaches, indexer and search head clustering, distributed management, and hybrid cloud deployments. Best practices are outlined for sizing storage, tuning indexers, and designing high availability into the forwarding, indexing, and search tiers. Clustering impacts on storage sizing and additional hosts are also addressed.
https://docs.google.com/presentation/d/1DcL4zK6i3HZRDD4xTGX1VpSOwyu2xBeWLT6a_...
https://docs.google.com/presentation/d/1DcL4zK6i3HZRDD4xTGX1VpSOwyu2xBeWLT6a_...MongoDB
Ad
More from Bryan Reinero (9)
Event Sourcing + CQRS
Event Sourcing + CQRSBryan Reinero The document discusses event sourcing as a powerful pattern for managing state changes in systems, illustrating its application in user account balances and transaction logs. It explores concepts such as Command Query Responsibility Segregation (CQRS) versus traditional CRUD operations, highlighting the complexities involved in data modeling and replaying events. Additionally, it touches on external updates and notifications in the context of maintaining data consistency.
MongoDB + Spark
MongoDB + SparkBryan Reinero This document discusses MongoDB and Spark and how they can be used together. It provides an overview of the Spark and Hadoop ecosystems and how MongoDB fits in. It then demonstrates how to connect MongoDB to Spark using the MongoDB Spark connector library, including how to read MongoDB documents into an RDD and run Spark jobs on MongoDB data. Examples of aggregation operations and leveraging data locality with mongos are also briefly mentioned.
MongoDB, Event Sourcing & Spark
MongoDB, Event Sourcing & SparkBryan Reinero The document discusses event sourcing and how it can be used to model state changes as a sequence of events. It provides examples of how event sourcing can track balance changes for users by logging a series of credit and debit events over time. It also discusses how event sourcing allows reconstructing past application states and tracking state evolution by replaying the full sequence of events.
Mongo db &_spark
Mongo db &_sparkBryan Reinero The document describes Spark and how it relates to other big data technologies like HDFS, YARN, and Mesos. It shows how Spark can be used both with these technologies or on its own. Key Spark concepts discussed include Resilient Distributed Datasets (RDDs), transformations, actions, and lineage. It also provides examples of using Spark with MongoDB to read data from a MongoDB collection into an RDD.
Event sourcing
Event sourcingBryan Reinero This document provides an overview of event sourcing, a pattern for managing data changes over time. It discusses how event sourcing records the full history of state changes as a sequence of events, allows for consistent replay of events to derive the current state, and enables building projections from the stream of events. Event sourcing provides benefits like auditability, failure recovery, and the ability to easily derive multiple views of the data for different use cases.
Systems of engagement
Systems of engagementBryan Reinero The document discusses building systems of engagement using MongoDB, highlighting their importance for real-time, context-aware interactions that enhance customer engagement and experience. It includes various examples, such as the application for veterans' health records and social interaction features like notifications and user connections. Additionally, it addresses the complexities of data management, analytics, and the need for a flexible schema to handle diverse data types efficiently.
Internet of things
Internet of thingsBryan Reinero The document discusses using MongoDB for Internet of Things (IoT) applications. It describes data structures for storing IoT sensor data, including embedding data points in documents and using compound unique indexes. It also discusses using MongoDB for building engaging mobile applications, including storing location data, implementing social features like following users, and integrating analytics.
Polyglot Persistence
Polyglot PersistenceBryan Reinero The document discusses polyglot persistence, which involves using multiple database technologies to leverage the right tool for specific application needs. It addresses the transition from traditional relational databases to other data models like key-value, document, and graph databases driven by higher performance requirements and larger datasets. Additionally, it highlights the importance of availability, scalability, and flexible schemas in modern application architecture.
Code instrumentation
Code instrumentationBryan Reinero The document discusses benchmarking tools for databases and introduces a new benchmarking tool called Firehose. It describes some issues with existing tools like Mongoimport and YCSB in that they do not adequately model real-world workloads. Firehose is presented as a new multi-threaded benchmarking tool that aims to have high relevance to real applications. It measures performance metrics like operation durations and supports features like configurable load levels and integration with monitoring systems.
Recently uploaded (20)
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025pcprocore 👉𝗡𝗼𝘁𝗲:𝗖𝗼𝗽𝘆 𝗹𝗶𝗻𝗸 & 𝗽𝗮𝘀𝘁𝗲 𝗶𝗻𝘁𝗼 𝗚𝗼𝗼𝗴𝗹𝗲 𝗻𝗲𝘄 𝘁𝗮𝗯> https://pcprocore.com/ 👈◀
CapCut Pro Crack is a powerful tool that has taken the digital world by storm, offering users a fully unlocked experience that unleashes their creativity. With its user-friendly interface and advanced features, it’s no wonder why aspiring videographers are turning to this software for their projects.
The Future of Technology: 2025-2125 by Saikat Basu.pdf
The Future of Technology: 2025-2125 by Saikat Basu.pdfSaikat Basu A peek into the next 100 years of technology. From Generative AI to Global AI networks to Martian Colonisation to Interstellar exploration to Industrial Nanotechnology to Artificial Consciousness, this is a journey you don't want to miss. Which ones excite you the most? Which ones are you apprehensive about? Feel free to comment! Let the conversation begin!
WebdriverIO & JavaScript: The Perfect Duo for Web Automation
WebdriverIO & JavaScript: The Perfect Duo for Web Automationdigitaljignect In today’s dynamic digital landscape, ensuring the quality and dependability of web applications is essential. While Selenium has been a longstanding solution for automating browser tasks, the integration of WebdriverIO (WDIO) with Selenium and JavaScript marks a significant advancement in automation testing. WDIO enhances the testing process by offering a robust interface that improves test creation, execution, and management. This amalgamation capitalizes on the strengths of both tools, leveraging Selenium’s broad browser support and WDIO’s modern, efficient approach to test automation. As automation testing becomes increasingly vital for faster development cycles and superior software releases, WDIO emerges as a versatile framework, particularly potent when paired with JavaScript, making it a preferred choice for contemporary testing teams.
9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FMESafe Software This session will cover a common use case for local and state/provincial governments who create and/or maintain their 9-1-1 addressing data, particularly address points and road centerlines. In this session, you'll learn how FME has helped Shelby County 9-1-1 (TN) automate the 9-1-1 addressing process; including automatically assigning attributes from disparate sources, on-the-fly QAQC of said data, and reporting. The FME logic that this presentation will cover includes: Table joins using attributes and geometry, Looping in custom transformers, Working with lists and Change detection.
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptxFIDO Alliance Securing Account Lifecycles in the Age of Deepfakes
Quantum AI Discoveries: Fractal Patterns Consciousness and Cyclical Universes
Quantum AI Discoveries: Fractal Patterns Consciousness and Cyclical UniversesSaikat Basu Embark on a cosmic journey exploring the intersection of quantum
computing, consciousness, and ancient wisdom. Together we'll uncover the
recursive patterns that bind our reality.
Security Tips for Enterprise Azure Solutions
Security Tips for Enterprise Azure SolutionsMichele Leroux Bustamante Delivering solutions to Azure may involve a variety of architecture patterns involving your applications, APIs data and associated Azure resources that comprise the solution. This session will use reference architectures to illustrate the security considerations to protect your Azure resources and data, how to achieve Zero Trust, and why it matters. Topics covered will include specific security recommendations for types Azure resources and related network security practices. The goal is to give you a breadth of understanding as to typical security requirements to meet compliance and security controls in an enterprise solution.
Curietech AI in action - Accelerate MuleSoft development
Curietech AI in action - Accelerate MuleSoft developmentshyamraj55 CurieTech AI in Action – Accelerate MuleSoft Development
Overview:
This presentation demonstrates how CurieTech AI’s purpose-built agents empower MuleSoft developers to create integration workflows faster, more accurately, and with less manual effort
linkedin.com
+12
curietech.ai
+12
meetups.mulesoft.com
+12
.
Key Highlights:
Dedicated AI agents for every stage: Coding, Testing (MUnit), Documentation, Code Review, and Migration
curietech.ai
+7
curietech.ai
+7
medium.com
+7
DataWeave automation: Generate mappings from tables or samples—95%+ complete within minutes
linkedin.com
+7
curietech.ai
+7
medium.com
+7
Integration flow generation: Auto-create Mule flows based on specifications—speeds up boilerplate development
curietech.ai
+1
medium.com
+1
Efficient code reviews: Gain intelligent feedback on flows, patterns, and error handling
youtube.com
+8
curietech.ai
+8
curietech.ai
+8
Test & documentation automation: Auto-generate MUnit test cases, sample data, and detailed docs from code
curietech.ai
+5
curietech.ai
+5
medium.com
+5
Why Now?
Achieve 10× productivity gains, slashing development time from hours to minutes
curietech.ai
+3
curietech.ai
+3
medium.com
+3
Maintain high accuracy with code quality matching or exceeding manual efforts
curietech.ai
+2
curietech.ai
+2
curietech.ai
+2
Ideal for developers, architects, and teams wanting to scale MuleSoft projects with AI efficiency
Conclusion:
CurieTech AI transforms MuleSoft development into an AI-accelerated workflow—letting you focus on innovation, not repetition.
Techniques for Automatic Device Identification and Network Assignment.pdf
Techniques for Automatic Device Identification and Network Assignment.pdfPriyanka Aash Techniques for Automatic Device Identification and Network Assignment
OWASP Barcelona 2025 Threat Model Library
OWASP Barcelona 2025 Threat Model LibraryPetraVukmirovic Threat Model Library Launch at OWASP Barcelona 2025
https://owasp.org/www-project-threat-model-library/
cnc-processing-centers-centateq-p-110-en.pdf
cnc-processing-centers-centateq-p-110-en.pdfAmirStern2 מרכז עיבודים תעשייתי בעל 3/4/5 צירים, עד 22 החלפות כלים עם כל אפשרויות העיבוד הדרושות. בעל שטח עבודה גדול ומחשב נוח וקל להפעלה בשפה העברית/רוסית/אנגלית/ספרדית/ערבית ועוד..
מסוגל לבצע פעולות עיבוד שונות המתאימות לענפים שונים: קידוח אנכי, אופקי, ניסור, וכרסום אנכי.
UserCon Belgium: Honey, VMware increased my bill
UserCon Belgium: Honey, VMware increased my billstijn40 VMware’s pricing changes have forced organizations to rethink their datacenter cost management strategies. While FinOps is commonly associated with cloud environments, the FinOps Foundation has recently expanded its framework to include Scopes—and Datacenter is now officially part of the equation. In this session, we’ll map the FinOps Framework to a VMware-based datacenter, focusing on cost visibility, optimization, and automation. You’ll learn how to track costs more effectively, rightsize workloads, optimize licensing, and drive efficiency—all without migrating to the cloud. We’ll also explore how to align IT teams, finance, and leadership around cost-aware decision-making for on-prem environments. If your VMware bill keeps increasing and you need a new approach to cost management, this session is for you!
The Future of Product Management in AI ERA.pdf
The Future of Product Management in AI ERA.pdfAlyona Owens Hi, I’m Aly Owens, I have a special pleasure to stand here as over a decade ago I graduated from CityU as an international student with an MBA program. I enjoyed the diversity of the school, ability to work and study, the network that came with being here, and of course the price tag for students here has always been more affordable than most around.
Since then I have worked for major corporations like T-Mobile and Microsoft and many more, and I have founded a startup. I've also been teaching product management to ensure my students save time and money to get to the same level as me faster avoiding popular mistakes. Today as I’ve transitioned to teaching and focusing on the startup, I hear everybody being concerned about Ai stealing their jobs… We’ll talk about it shortly.
But before that, I want to take you back to 1997. One of my favorite movies is “Fifth Element”. It wowed me with futuristic predictions when I was a kid and I’m impressed by the number of these predictions that have already come true. Self-driving cars, video calls and smart TV, personalized ads and identity scanning. Sci-fi movies and books gave us many ideas and some are being implemented as we speak. But we often get ahead of ourselves:
Flying cars,Colonized planets, Human-like AI: not yet, Time travel, Mind-machine neural interfaces for everyone: Only in experimental stages (e.g. Neuralink).
Cyberpunk dystopias: Some vibes (neon signs + inequality + surveillance), but not total dystopia (thankfully).
On the bright side, we predict that the working hours should drop as Ai becomes our helper and there shouldn’t be a need to work 8 hours/day. Nobody knows for sure but we can require that from legislation. Instead of waiting to see what the government and billionaires come up with, I say we should design our own future.
So, we as humans, when we don’t know something - fear takes over. The same thing happened during the industrial revolution. In the Industrial Era, machines didn’t steal jobs—they transformed them but people were scared about their jobs. The AI era is making similar changes except it feels like robots will take the center stage instead of a human. First off, even when it comes to the hottest space in the military - drones, Ai does a fraction of work. AI algorithms enable real-time decision-making, obstacle avoidance, and mission optimization making drones far more autonomous and capable than traditional remote-controlled aircraft. Key technologies include computer vision for object detection, GPS-enhanced navigation, and neural networks for learning and adaptation. But guess what? There are only 2 companies right now that utilize Ai in drones to make autonomous decisions - Skydio and DJI.
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025OpenACC The OpenACC organization focuses on enhancing parallel computing skills and advancing interoperability in scientific applications through hackathons and training. The upcoming 2025 Open Accelerated Computing Summit (OACS) aims to explore the convergence of AI and HPC in scientific computing and foster knowledge sharing. This year's OACS welcomes talk submissions from a variety of topics, from Using Standard Language Parallelism to Computer Vision Applications. The document also highlights several open hackathons, a call to apply for NVIDIA Academic Grant Program and resources for optimizing scientific applications using OpenACC directives.
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdfPriyanka Aash GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...Fwdays We will explore how e-commerce projects prepare for the busiest time of the year, which key aspects to focus on, and what to expect. We’ll share our experience in setting up auto-scaling, load balancing, and discuss the loads that Silpo handles, as well as the solutions that help us navigate this season without failures.
Mongo db v3_deep_dive
- 1. MongoDB v3.0 Deep Dive { Name: ‘Bryan Reinero’, Title: ‘Developer Advocate’, Twitter: ‘@blimpyacht’, Email: ‘[email protected]’ }
- 2. 2 Agenda • Storage Engine API • MmapV1 • WiredTiger • Document Level Concurrency • Index Improvements • The Future
- 3. 3 Storage Engine API • Allows to "plug-in" different storage engines – Different work sets require different performance characteristics – mmapv1 is not ideal for all workloads – More flexibility • Can mix storage engines on same replica set/sharded cluster • Opportunity to integrate further ( HDFS, native encrypted, hardware optimized …)
- 4. 4 Storage Engine API StorageEngine Top Level Class for creating a Storage Engine RecoveryUnit Durability interface. Ensures data is persisted. On-disk information mutated through this interface DatabaseCatalogEntry MongoDB Logical Database CollectionCatalogEntry MongoDB Collection SortedDataInterface Index implementation. Not all Indexes are B-trees
- 5. 5 MongoDB Storage Engines • <= MongoDB 2.6 – One unique mechanism using Memory Mapped Files – "mmapv1" Storage Engine • MongoDB 3.0 has a few more options – mmapv1 – default – wiredTiger – (in_memory – experimental only)
- 7. 7 MMAPv1
- 9. 9 What is WiredTiger? • Storage engine company founded by BerkeleyDB alums • Recently acquired by MongoDB • Available as a storage engine option in MongoDB 3.0
- 10. 10 Why is WiredTiger Awesome • Document-level concurrency • Disk Compression • Consistency without journaling • Better performance on many workloads – write heavy
- 11. 11 Improving Concurrency • 2.2 – Global Lock • 2.4 – Database-level Locking • 3.0 MMAPv1 – Collection-level Locking • 3.0 WT – Document-level – Writes no longer block all other writes – Higher level of concurrency leads to more CPU usage
- 12. 12 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 http://ses.library.usyd.edu.au/bitstream/2123/5353/1/michael-cahill-2009-thesis.pdf
- 13. 13 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 Read0(8) = 6
- 14. 14 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 Read0(8) = 6 Read1(8) = 6 Read2(8) = 6
- 15. 15 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 write3(8, $inc ) write4(8, $inc )
- 16. 16 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 write3(8, $inc ) write4(8, $inc ) 8 6 8 6
- 17. 17 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 write3(8, $inc ) write4(8, $inc ) 8 6 8 7
- 18. 18 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 7 9 2 10 1 11 1 12 5 13 4 15 5 14 5 write3(8, $inc ) write4(8, $inc ) 8 6 8 7 Compare & Swap
- 19. 19 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 7 9 2 10 1 11 1 12 5 13 4 15 5 14 5 write3(8, $inc ) 8 6
- 20. 20 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 7 9 2 10 1 11 1 12 5 13 4 15 5 14 5 write3(8, $inc ) 8 7
- 21. 21 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 7 9 2 10 1 11 1 12 5 13 4 15 5 14 5 write3(8, $inc ) 8 7Compare & !swap
- 22. 22 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 7 9 2 10 1 11 1 12 5 13 4 15 5 14 5 write3(8, $inc ) 8 7 Re-read & Retry
- 23. 23 Lock Free Algorithms 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 7 9 2 10 1 11 1 12 5 13 4 15 5 14 5 write3(8, $inc ) 8 8
- 24. 24 Document Level Concurrency Control 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 8 9 2 10 1 11 1 12 5 13 4 15 5 14 5 Compare & Swap write3(8, $inc ) 8 8
- 25. 25 Document Level Concurrency Control 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 8 9 2 10 1 11 1 12 5 13 4 15 5 14 5 Compare & Swap write3(8, $inc ) 8 8
- 27. 27 Wired Tiger Concurrency • Fine grained • Lock free • Wait free • Stone cold • Superfly
- 28. 28 Compression • Data is compressed on disk • 2 supported algorithms – snappy: default. Good compression, relatively low overhead – zlib: Better • Indexes are compressed using prefix compression – Allows compression in memory
- 29. 29 Tuning Wired Tiger File System CacheWired Tiger Cache Total RAM Non- mapped
- 30. 30 Tuning Wired Tiger File System CacheWired Tiger Cache Total RAM Default 50% RAM Non- mapped
- 31. 31 Tuning Wired Tiger File System CacheWired Tiger Cache Total RAM Default 50% RAM Non- mapped Knobs • Wired Tiger Cache Size • Compression • Snappy • Zlib • off
- 32. 32 Indexing Improvements MMapV11 4 2 2 3 5 4 7 5 4 6 9 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5
- 33. 33 Indexing Improvements 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 { _id: 6, categories: [ “database”, “distributed”, “document store” ] }
- 34. 34 Indexing Improvements 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 {$push: { categories: “sharded” } } { _id: 6, categories: [ “database”, “distributed”, “document store”, “sharded” ] }
- 35. 35 Indexing Improvements 1 4 2 2 3 5 4 7 5 4 6 9 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 { _id: 6, categories: [ “database”, “distributed”, “document store”, “sharded” ] }
- 36. 36 Indexing Improvements 1 4 2 2 3 5 4 7 5 4 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 6 9 MMapV1 6 9
- 37. 37 Indexing Improvements 1 4 2 2 3 5 4 7 5 4 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 6 9 MMapV1
- 38. 38 Indexing Improvements 1 4 2 2 3 5 4 7 5 4 7 3 8 6 9 2 10 1 11 1 12 5 13 4 15 5 14 5 6 9 WiredTiger The RecordId != DiskLoc
- 39. 39 Consistency without Journaling • MMAPv1 uses write-ahead log (journal) to guarantee consistency • WT doesn't have this need: no in-place updates – Write-ahead log committed at checkpoints • 2GB or 60sec by default – configurable! – No journal commit interval: writes are written to journal as they come in – Better for insert-heavy workloads • Replication guarantees the durability
- 40. 40 7x-10x Performance, 50%-80% Less Storage How: WiredTiger Storage Engine • Same data model, same query language, same ops • Write performance gains driven by document-level concurrency control • Storage savings driven by native compression • 100% backwards compatible • Non-disruptive upgrade MongoDB 3.0MongoDB 2.6 Performance https://www.mongodb.com/blog/post/high-performance- benchmarking-mongodb-and-nosql-systems
- 41. 41 Playing nice together • Can not – Can't copy database files – Can't just restart w/ same dbpath • Yes we can! – Initial sync from replica set works perfectly! – mongodump/restore • Rolling upgrade of replica set to WT: – Shutdown secondary – Delete dbpath – Relaunch w/ --storageEngine=wiredTiger – Rollover
- 42. AND BEYOND THE INFINITE VERSION 3.2
- 43. 43 Storage Engine Storage Engines • WiredTiger (now default) • In-Memory • Encryption at Rest Tools • Schema visualizer Features • $lookup (Enterprise) • Read Committed • Schema Validation Rules • Partial Indexes
- 44. 44 Storage Engine Storage Engines • WiredTiger (now default) • In-Memory • Encryption at Rest Tools • Schema visualizer Features • $lookup (Enterprise) • Read Committed • Schema Validation Rules • Partial Indexes Features now available in v3.1.6 Community release
- 45. Thanks! { name: ‘Bryan Reinero’, title: ‘Developer Advocate’, twitter: ‘@blimpyacht’, code: ‘github.com/breinero’ email: ‘[email protected]’ }
Editor's Notes
- #5: https://github.com/mongodb/mongo/blob/master/src/mongo/db/storage/storage_engine.h https://github.com/mongodb/mongo/blob/master/src/mongo/db/storage/recovery_unit.h https://github.com/mongodb/mongo/blob/master/src/mongo/db/storage/sorted_data_interface.h https://github.com/mongodb/mongo/blob/master/src/mongo/db/catalog/index_catalog.h Mathias Stearn’s https://www.mongodb.com/presentations/write-yourself-storage-engine-40-minutes
- #45: Read Commited