MongoDB Aggregation MongoSF May 2011

May 26, 201112 likes2,199 views

This document discusses MongoDB's new aggregation framework, which provides a more performant and declarative way to perform data aggregation tasks compared to MapReduce. The framework includes pipeline operations like $match, $project, and $group that allow filtering, reshaping, and grouping documents. It also features an expression language for computed fields. The initial release will support aggregation pipelines and sharding, with future plans to add more operations and expressions.

MongoDB’s New Aggregation FeaturesChris Westin© Copyright 2010 10gen Inc.

What problem are we solving?Map/Reduce can be used for aggregation…Currently being used for totaling, averaging, etcMap/Reduce is a big hammerSimpler tasks should be easierShouldn’t need to write JavaScriptAvoid the overhead of JavaScript engineWe’re seeing requests for help in handling complex documentsSelect only subdocuments or arrays

How will we solve the problem?Our new aggregation frameworkDeclarative frameworkNo JavaScript requiredDescribe a chain of operations to applyExpression evaluationReturn computed valuesFramework: we can add new operations easilyC++ implementationHigher performance than JavaScript

Aggregation - PipelinesAggregation requests specify a pipelineA pipeline is a series of operationsConceptually, the members of a collection are passed through a pipeline to produce a resultSimilar to a command-line pipe

Pipeline Operations$matchUses a query predicate (like .find({…})) as a filter$projectUses a sample document to determine the shape of the result (similar to .find()’s optional argument)This can include computed values$groupAggregates items into buckets defined by a key

Computed ExpressionsAvailable in $project operationsPrefix expression languageAdd two fields: $add:[“$field1”, “$field2”]Provide a value for a missing field: $ifnull:[“$field1”, “$field2”]Nesting: $add:[“$field1”, $ifnull:[“$field2”, “$field3”]]Other functions….And we can easily add more as required

Projections$project can reshape results$unwind expression doles out array values one at a timePull fields from nested documents to the topPush fields from the top down into new virtual documents

Grouping$group aggregation expressionsTotal of column values: $sumAverage of column values: $avgCollect column values in an array: $push

Demo(See script at https://gist.github.com/993733)

Usage TipsUse $match in a pipeline as early as possibleThe query optimizer can then be used to choose an index and avoid scanning the entire collection

Driver SupportInitial version is a commandFor any language, build a JSON database object, and execute the command{ aggregate : <collection>, pipeline : {…} }Beware of command result size limit

When is this being released?In final development nowExpect to see this in the near future

Sharding supportInitial release will support shardingMongos analyzes pipeline, and forwards operations up to $group to shards; combines shard server results and continues

Pipeline Operations – Future Plans$sortSorts the document stream according to a key$outSaves the document stream to a collectionSimilar to M/R $out, but with sharded output

Expressions – Future PlansDate field extractionGet year, month, day, hour, etc, from DateDate arithmetic

The document discusses organizing time series metrics data and compares hierarchical and tagged approaches. It argues that a tagged approach is superior as it allows for more flexible querying of metrics by different dimensions and tags. A tagged approach stores tags as metadata alongside metric names and values, allowing filtering and aggregation by any tag combination. This enables more powerful queries and computations across diverse sets of metrics than is possible with a hierarchical organization.

What Reika Taught usUniversity of California, Santa Cruz

- Reika is a domain-specific language for querying time series databases built on ANTLR. It aims to provide a SQL-like syntax that supports multiple backends. - The current implementation includes a lexer, parser, AST generation using ANTLR, and an interpreter. Symbol and type checking are also implemented. - Lessons learned include checking library source code before using, problems can cascade, and deeper understanding comes after initial implementation. Related work includes InfluxQL and other query languages for time series data.

Xephon K A Time series database with multiple backendsUniversity of California, Santa Cruz

Single page application 05Ismaeel Enjreny

This document provides information on filters and the $http service in AngularJS. It discusses built-in filters for formatting data like date, currency, number, as well as custom filters. Array filters like limitTo and orderBy are also covered. The $http service allows communicating with remote servers via HTTP requests. Key aspects of $http configuration like cache, headers, success/error callbacks are summarized.

From Overnight to Always On @ Jfokus 2019Enno Runne

Systems integration is everywhere, not because we want it, but because we need it. It's the download of exchange rates, the list of yesterday's orders and the latest inventory. Not long time ago, we'd pull this kind of information in overnight batches and every system had something to work on. That was the age where we had printed newspapers. Today, data needs to be there. Instantaneously. Or "as fast as possible". We don't want to transfer huge piles of data once every night, but have the updates coming by - just after the change happened. We want streaming data. In this talk, we exemplify the path to move from overnight file exchanges to streaming data by using Alpakka, which is an integration library based on Reactive Streams and Akka.

Let's Compare: A Benchmark review of InfluxDB and ElasticsearchInfluxData

In this webinar, Ivan K will compare the performance and features of InfluxDB and Elasticsearch for common time-series workloads, specifically looking at the rates of data ingestion, on-disk data compression, and query performance. Come hear about how Ivan conducted his tests to determine which time-series db would best fit your needs. We will reserve 15 minutes at the end of the talk for you to ask Ivan directly about his test processes and independent viewpoint.

Introduction tomongodbLee Theobald

This document introduces MongoDB, an open-source document database. It describes MongoDB as schema-less, highly scalable, and capable of replication and auto-sharding. Key features include storing data as JSON documents, a familiar database and collection structure, easy insertion and querying of documents, and aggregation of data using map-reduce functions. The document encourages readers to try MongoDB themselves by downloading it or exploring online tutorials.

Data Processing with Cascading Java API on Apache HadoopHikmat Dhamee

Cascading is an application framework that simplifies developing robust data analytics and management applications on Apache Hadoop. It provides a query API and planner to define, share, and execute data processing workflows on Hadoop clusters. Cascading handles complexities of Hadoop application development, job creation, and scheduling to allow organizations to rapidly develop complex data processing applications.

Introduction to ELKHarshakumar Ummerpillai

This document introduces the ELK stack, which consists of Elasticsearch, Logstash, and Kibana. It provides instructions on setting up each component and using them together. Elasticsearch is a search engine that stores and searches data in JSON format. Logstash is an agent that collects logs from various sources, applies filters, and outputs to Elasticsearch. Kibana visualizes and explores the logs stored in Elasticsearch. The document demonstrates setting up each component and running a proof of concept to analyze sample log data.

Machine Learning in a Twitter ETL using ELK hypto

Updating materialized views and caches using kafkaZach Cox

The document discusses using Apache Kafka to publish data changes and updates to materialized views and caches. It describes building a user information service that currently queries normalized data from multiple tables in a database. This has issues like slow response times and stale data. The document proposes using Kafka streams to read from topics containing normalized data changes. This allows building and storing materialized views locally using RocksDB for low latency access without stale data.

RethinkDB - the open-source database for the realtime webAlex Ivanov

This document summarizes RethinkDB, an open-source database for building realtime web applications. It describes key features of RethinkDB such as its push model that allows applications to receive live updates from the database via changefeeds without polling, its query language, data modeling options using embedded documents or references, and how it can be used to build collaborative, multiplayer and analytics applications.

Building data flows with Celery and SQLAlchemyRoger Barnes

Reporting and analysis systems rely on coherent and reliable data, often from disparate sources. To that end, a series of well established data warehousing practices have emerged to extract data and produce a consistent data store. This talk will look at some options for composing workflows using Python. In particular, we'll explore beyond Celery's asynchronous task processing functionality into its workflow (aka Canvas) system and how it can be used in conjunction with SQLAlchemy's architecture to provide the building blocks for data stream processing.

EG Reports - Delicious DataBenjamin Shum

Delicious Data: Automated Techniques for Complex Reports: Get data into the hands of those that need it most by automating SQL reports, scheduling data extracts using the Evergreen reporter, and extending the reporter with new source definitions. Ben Shum from Bibliomation and Jeff Godin from the Traverse Area District Library will show you how you can meet your advanced or complex reporting needs, both with and without direct database access. Join us in our efforts to eliminate manual, time-consuming reporting workflows! Ben Shum (Biblio), Jeff Godin (TADL)

Apache Spark - Aram MkrtchyanHovhannes Kuloghlyan

Apache Spark is a cluster computing platform designed to be fast and general-purpose. It provides a unified analytics engine for large-scale data processing across SQL, streaming, machine learning, and graph processing. Spark programs can be written in Java, Scala, Python and R. It works by building resilient distributed datasets (RDDs) that can be operated on in parallel. RDDs support transformations like map, filter and join and actions like count, collect and save. Spark also provides caching of RDDs in memory for improved performance.

MongoDBGanesh Kunwar

MongoDB is a document database that provides high performance, high availability, and easy scalability. It uses a document-oriented data model where data is stored in documents that contain field and value pairs similar to JSON objects. Documents can be embedded within other documents to create complex hierarchical relationships between data. MongoDB supports replication and automatic sharding for scalability and high availability.

Replicating application data into materialized viewsZach Cox

9.4jsonAndrew Dunstan

PostgreSQL 9.4 introduces new features for JSON support including: 1) New JSON creation functions like json_build_object() and json_build_array() that make constructing JSON data easier. 2) The jsonb type which stores JSON in a canonical, indexable format and allows more efficient querying and indexing of JSON data compared to the json type. 3) Indexing capabilities for jsonb including GIN indexes that support containment and exists operators as well as expression indexes on JSON subdocuments. 4) Many existing JSON functions now have jsonb variants to allow processing of jsonb data. The jsonb type is generally preferred over json when indexing, querying or modifying JSON data

Sphinx - High performance full-text search for MySQLNguyen Van Vuong

The document discusses Sphinx, an open source full-text search engine. It begins with an overview of full-text search and what Sphinx is - a high performance search engine that integrates well with SQL databases. The document then covers Sphinx's workflow, including indexing data, searching via its API or SphinxQL, and its query syntax. It also discusses how Sphinx scales horizontally across nodes and clusters.

Logstash + Elasticsearch + Kibana Presentation on Startit Tech MeetupStartit

1. Logstash is an open source tool for collecting, processing, and storing logs and other event data. It allows centralized collection and parsing of logs from various sources before sending them to Elasticsearch for storage and indexing. 2. Kibana provides visualization and search capabilities on top of the logs stored in Elasticsearch, allowing users to easily explore and analyze log data. 3. The combination of Logstash, Elasticsearch, and Kibana provides a replacement for commercial log management tools like Splunk, with the ability to collect, parse, store, search, and visualize logs from many different sources in a centralized way.

Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...ForgeRock

Get docs from sp doc librarySudip Sengupta

Mongo db admin_20110316radiocats

Michael DelNegro has been the principal database administrator at AOL for 6 years and currently administers MongoDB installations including applications for MapQuest and Patch. These applications store user profile and activity information with the MapQuest app hosting 300 million users on 26 servers and the Patch app storing around 100GB of data. He outlines his experiences administering MongoDB at AOL including deployment details, best practices, tools, and upcoming plans to upgrade installations and applications.

A Beginner's Guide to Building Data Pipelines with LuigiGrowth Intelligence

In this talk, we provide an introduction to Python Luigi via real life case studies showing you how you can break large, multi-step data processing task into a graph of smaller sub-tasks that are aware of the state of their interdependencies. Growth Intelligence tracks the performance and activity of all the companies in the UK economy using their data ‘footprint’. This involves tracking numerous unstructured data points from multiple sources in a variety of formats and transforming them into a standardised feature set we can use for building predictive models for our clients. In the past, this data was collected by in a somewhat haphazard fashion: combining manual effort, ad hoc scripting and processing which was difficult to maintain. In order to streamline the data flows, we’re using an open-source Python framework from Spotify called Luigi. Luigi was created for managing task dependencies, monitoring the progress of the data pipeline and providing frameworks for common batch processing tasks.

Efficient Data Storage for Analytics with Apache Parquet 2.0Cloudera, Inc.

Apache Parquet is an open-source columnar storage format for efficient data storage and analytics. It provides efficient compression and encoding techniques that enable fast scans and queries of large datasets. Parquet 2.0 improves on these efficiencies through enhancements like delta encoding, binary packing designed for CPU efficiency, and predicate pushdown using statistics. Benchmark results show Parquet provides much better compression and query performance than row-oriented formats on big data workloads. The project is developed as an open-source community with contributions from many organizations.

MongoDB San Francisco DrupalCon 2010Karoly Negyesi

Hadoop meetup : HUGFR Construire le cluster le plus rapide pour l'analyse des...Modern Data Stack France

Construire le cluster le plus rapide pour l'analyse des datas : benchmarks sur un régresseur par Christopher Bourez (Axa Global Direct) Les toutes dernières technologies de calcul parallèle permettent de calculer des modèles de prédiction sur des big datas en des temps records. Avec le cloud est facilité l'accès à des configurations hardware modernes avec la possibilité d'une scalabilité éphémère durant les calculs. Des benchmarks sont réalisés sur plusieurs configuration hardware, allant de 1 instance à un cluster de 100 instances. Christopher Bourez, développeur & manager expert en systèmes d'information modernes chez Axa Global Direct. Alien thinker. Blog : http://christopher5106.github.io/

Практическое применение MongoDB Aggregation FrameworkДенис Кравченко

MongoDB's New Aggregation frameworkChris Westin

The document discusses MongoDB's new aggregation framework, which provides a declarative pipeline for performing data aggregation operations on complex documents. The framework allows users to describe a chain of operations without writing JavaScript. It will offer high-performance operators like $match, $project, $unwind, $group, $sort, and computed expressions to reshape and analyze document data without the overhead of JavaScript. The aggregation framework is nearing release and will support sharding by forwarding pipeline operations to shards and combining results.

More Related Content

What's hot (20)

Data Processing with Cascading Java API on Apache HadoopHikmat Dhamee

Introduction to ELKHarshakumar Ummerpillai

Machine Learning in a Twitter ETL using ELK hypto

Updating materialized views and caches using kafkaZach Cox

RethinkDB - the open-source database for the realtime webAlex Ivanov

Building data flows with Celery and SQLAlchemyRoger Barnes

EG Reports - Delicious DataBenjamin Shum

Apache Spark - Aram MkrtchyanHovhannes Kuloghlyan

MongoDBGanesh Kunwar

Replicating application data into materialized viewsZach Cox

9.4jsonAndrew Dunstan

Sphinx - High performance full-text search for MySQLNguyen Van Vuong

Logstash + Elasticsearch + Kibana Presentation on Startit Tech MeetupStartit

Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...ForgeRock

Get docs from sp doc librarySudip Sengupta

Mongo db admin_20110316radiocats

A Beginner's Guide to Building Data Pipelines with LuigiGrowth Intelligence

Efficient Data Storage for Analytics with Apache Parquet 2.0Cloudera, Inc.

MongoDB San Francisco DrupalCon 2010Karoly Negyesi

Hadoop meetup : HUGFR Construire le cluster le plus rapide pour l'analyse des...Modern Data Stack France

Data Processing with Cascading Java API on Apache HadoopHikmat Dhamee

Introduction to ELKHarshakumar Ummerpillai

Machine Learning in a Twitter ETL using ELK hypto

Updating materialized views and caches using kafkaZach Cox

RethinkDB - the open-source database for the realtime webAlex Ivanov

Building data flows with Celery and SQLAlchemyRoger Barnes

EG Reports - Delicious DataBenjamin Shum

Apache Spark - Aram MkrtchyanHovhannes Kuloghlyan

MongoDBGanesh Kunwar

Replicating application data into materialized viewsZach Cox

9.4jsonAndrew Dunstan

Sphinx - High performance full-text search for MySQLNguyen Van Vuong

Logstash + Elasticsearch + Kibana Presentation on Startit Tech MeetupStartit

Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...ForgeRock

Get docs from sp doc librarySudip Sengupta

Mongo db admin_20110316radiocats

A Beginner's Guide to Building Data Pipelines with LuigiGrowth Intelligence

Efficient Data Storage for Analytics with Apache Parquet 2.0Cloudera, Inc.

MongoDB San Francisco DrupalCon 2010Karoly Negyesi

Hadoop meetup : HUGFR Construire le cluster le plus rapide pour l'analyse des...Modern Data Stack France

Viewers also liked (11)

Практическое применение MongoDB Aggregation FrameworkДенис Кравченко

MongoDB's New Aggregation frameworkChris Westin

MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...MongoDB

This document discusses analyzing flight data using MongoDB aggregation. It provides examples of aggregation pipelines to group, match, project, sort, unwind and other stages. It explores questions about major carriers, airport cancellations, delays by distance and carrier. It also discusses visualizing route data and hub airports. Finally, it proposes a quiz on analyzing NYC flight data by importing data and performing queries on origins, cancellations, delays and weather impacts by month between the three major NYC airports.

Web Design Trends 2011Vitaly Friedman

ShardingMongoDB

How to leverage MongoDB for Big Data Analysis and Operations with MongoDB's A...Gianfranco Palumbo

Agg framework selectgroup feb2015 v2MongoDB

Developers love MongoDB because its flexible document model enhances their productivity. But did you know that MongoDB supports rich queries and lets you accomplish some of the same things you currently do with SQL statements? And that MongoDB's powerful aggregation framework makes it possible to perform real-time analytics for dashboards and reports? Attend this webinar for an introduction to the MongoDB aggregation framework and a walk through of what you can do with it. We'll also demo using it to analyze U.S. census data.

Optimizing MongoDB: Lessons Learned at Localyticsandrew311

The Aggregation FrameworkMongoDB

The document discusses MongoDB's Aggregation Framework, which allows users to perform ad-hoc queries and reshape data in MongoDB. It describes the key components of the aggregation pipeline including $match, $project, $group, $sort operators. It provides examples of how to filter, reshape, and summarize document data using the aggregation framework. The document also covers usage and limitations of aggregation as well as how it can be used to enable more flexible data analysis and reporting compared to MapReduce.

MongoDB for Time Series Data Part 2: Analyzing Time Series Data Using the Agg...MongoDB

The United States will be deploying 16,000 traffic speed monitoring sensors - 1 on every mile of US interstate in urban centers. These sensors update the speed, weather, and pavement conditions once per minute. MongoDB will collect and aggregate live sensor data feeds from roadways around the country, support real-time queries from cars on traffic conditions on their route as well as be the platform for real-time dashboards displaying traffic conditions and more complex analytical queries used to identify traffic trends. In this session, we’ll implement a few different data aggregation techniques to query and dashboard the metrics gathered from the US interstate.

Grid FSChris Powers

Практическое применение MongoDB Aggregation FrameworkДенис Кравченко

MongoDB's New Aggregation frameworkChris Westin

MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...MongoDB

Web Design Trends 2011Vitaly Friedman

ShardingMongoDB

How to leverage MongoDB for Big Data Analysis and Operations with MongoDB's A...Gianfranco Palumbo

Agg framework selectgroup feb2015 v2MongoDB

Optimizing MongoDB: Lessons Learned at Localyticsandrew311

The Aggregation FrameworkMongoDB

MongoDB for Time Series Data Part 2: Analyzing Time Series Data Using the Agg...MongoDB

Grid FSChris Powers

Similar to MongoDB Aggregation MongoSF May 2011 (20)

Introduction to MongoDBRaghunath A

MongoDB is a document-oriented, NoSQL database that stores data in flexible, JSON-like documents. It provides high performance, easy scalability and developer agility. Documents are stored in collections which are analogous to tables in relational databases. MongoDB supports CRUD operations and querying of documents using methods like find, update, remove. Aggregation operations allow grouping, filtering and counting of data in the collections. Some key use cases include storing log data by extracting fields from logs into documents, and modeling TV show data by embedding related data like seasons, episodes etc within a show document.

Experiment no 05Ankit Dubey

MongoDB Aggregation FrameworkCaserta

These are slides from our Big Data Warehouse Meetup in April. We talked about NoSQL databases: What they are, how they’re used and where they fit in existing enterprise data ecosystems. Mike O’Brian from 10gen, introduced the syntax and usage patterns for a new aggregation system in MongoDB and give some demonstrations of aggregation using the new system. The new MongoDB aggregation framework makes it simple to do tasks such as counting, averaging, and finding minima or maxima while grouping by keys in a collection, complementing MongoDB’s built-in map/reduce capabilities. For more information, visit our website at http://casertaconcepts.com/ or email us at [email protected].

Hands on Mahout!OSCON Byrum

1. The document summarizes a presentation about Apache Mahout, an open source machine learning library. It discusses algorithms like clustering, classification, topic modeling and recommendations. 2. It provides an overview of clustering Reuters documents using K-means in Mahout and demonstrates how to generate vectors, run clustering and inspect clusters. 3. It also discusses classification techniques in Mahout like Naive Bayes, logistic regression and support vector machines and shows code examples for generating feature vectors from data.

phoenix-on-calcite-hadoop-summit-2016Maryann Xue

This document summarizes a presentation on using Apache Calcite for cost-based query optimization in Apache Phoenix. Key points include: - Phoenix is adding Calcite's query planning capabilities to improve performance and SQL compliance over its existing query optimizer. - Calcite models queries as relational algebra expressions and uses rules, statistics, and a cost model to choose the most efficient execution plan. - Examples show how Calcite rules like filter pushdown and exploiting sortedness can generate better plans than Phoenix's existing optimizer. - Materialized views and interoperability with other Calcite data sources like Apache Drill are areas for future improvement beyond the initial Phoenix integration.

Cost-based Query Optimization in Apache Phoenix using Apache CalciteJulian Hyde

Cost-based Query Optimization DataWorks Summit/Hadoop Summit

Cost-Based query optimizationDataWorks Summit/Hadoop Summit

The document discusses how Apache Phoenix, an SQL query engine for Apache HBase, is integrating with Apache Calcite, an open source query optimization framework, to improve query optimization capabilities. Key points include: - Phoenix will leverage Calcite's SQL parser, validator, relational algebra, and cost-based query optimization capabilities to improve performance and SQL standard compliance. - Calcite models query optimizations as pluggable rules and uses a cost model and statistics to evaluate optimization alternatives. This will allow Phoenix to make more informed decisions around optimizations like index usage, join algorithms, and predicate pushdown. - Features like materialized views in Calcite will enable new indexing capabilities in Phoenix like defining indexes on query results to better

9-Query Processing-05-06-2023.PPTvenkatapranaykumarGa

The document discusses query processing and optimization. It describes the typical steps in processing a high-level query like SQL, which are translating the query into an internal representation, optimizing the query execution plan, and executing the plan. The key aspects of optimization are enumerating alternative execution plans using translation rules, estimating the cost of each plan, and choosing the lowest cost plan.

Sedna XML Database: Query Parser & Optimizing RewriterIvan Shcheklein

cheat-sheets.pdfFabianaFCordeiro

This document provides summaries of 3 cheat sheets for Java developers: 1. The Java 8 Streams cheat sheet defines streams, terminal and intermediate operations, parallel streams, and common pitfalls. It includes examples of using streams to filter, map, and collect data. 2. The Java Collections cheat sheet compares the performance of common collection classes and lists thread-safe alternatives. It also introduces notable Java collections libraries. 3. The Git cheat sheet outlines commands for creating and working with repositories, branches, tags and commits. It provides shortcuts for fetching updates, pushing changes, and viewing history and diffs.

Software development - the java perspectiveAlin Pandichi

Einführung in MongoDBNETUserGroupBern

This document provides an overview and introduction to NoSQL databases, focusing on MongoDB. It begins with definitions of NoSQL and examples of companies using NoSQL databases. It then discusses the motivations behind NoSQL, including the limitations of SQL and benefits of NoSQL for scalability. The document proceeds to describe MongoDB specifically as a document-oriented database, covering its data model, networking, drivers, collections and indexing. It also covers queries, atomic operations, replication, sharding, map-reduce and GridFS for large files. Well suited use cases include archiving, content management, ecommerce, gaming and mobile applications. The document concludes with a question and contact.

MongoDB .local London 2019: Fast Machine Learning Development with MongoDBMongoDB

Today an increasingly large number of products use machine learning and AI to deliver a great personalized user experience, and workplace software is no exception. Spoke goes beyond traditional ticketing with their friendly, AI-powered chatbot that gives workplace teams hours of time back as it automatically responds to questions on Slack, email, SMS, and web. Learn how Spoke uses MongoDB to do dynamic model training in real time from user interaction data and serves thousands of models, with multiple customized models per client.

MongoDB .local London 2019: Fast Machine Learning Development with MongoDBLisa Roth, PMP

Query optimization to improve performance of the code executionAlexander Decker

This document discusses query optimization techniques to improve the performance of code execution. It describes how object querying provides an abstraction for operations over collections of objects that allows the query evaluator to optimize queries dynamically at runtime. Specifically, it presents an example of using the Java Query Language (JQL) to perform an equi-join on two collections in a more succinct way compared to manually iterating over the collections, and discusses how the JQL query could be optimized using techniques like hash joins.

11.query optimization to improve performance of the code executionAlexander Decker

1. The document discusses query optimization techniques to improve the performance of object querying in Java. 2. It presents the Java Query Language (JQL) which allows programmers to express queries over object collections in Java through a declarative syntax. 3. The key aspects of JQL implementation include a compiler that compiles JQL queries to Java code and a query evaluator that applies optimizations like hash joins and nested loops joins to efficiently evaluate the queries.

MongoDB - Aggregation PipelineJason Terpko

This presentation will demonstrate how you can use the aggregation pipeline with MongoDB similar to how you would use GROUP BY in SQL and the new stage operators coming 3.4. MongoDB’s Aggregation Framework has many operators that give you the ability to get more value out of your data, discover usage patterns within your data, or use the Aggregation Framework to power your application. Considerations regarding version, indexing, operators, and saving the output will be reviewed.

Practical catalystdwm042

Catalyst is a web framework for Perl that allows developers to build dynamic web applications in a modular, reusable way. It utilizes common Perl techniques like Moose, DBIx::Class and Template Toolkit to handle tasks like object modeling, database access and view rendering. Catalyst applications can be built in a model-view-controller style to separate application logic, data access and presentation layers. This framework provides a standard way to write reusable code and build web UIs for tasks like system administration and automation.

Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache BeamFlink Forward

http://flink-forward.org/kb_sessions/no-shard-left-behind-dynamic-work-rebalancing-in-apache-beam/ The Apache Beam (incubating) programming model is designed to support several advanced data processing features such as autoscaling and dynamic work rebalancing. In this talk, we will first explain how dynamic work rebalancing not only provides a general and robust solution to the problem of stragglers in traditional data processing pipelines, but also how it allows autoscaling to be truly effective. We will then present how dynamic work rebalancing works as implemented in Google Cloud Dataflow and which path other Apache Beam runners link Apache Flink can follow to benefit from it.

Introduction to MongoDBRaghunath A

Experiment no 05Ankit Dubey

MongoDB Aggregation FrameworkCaserta

Hands on Mahout!OSCON Byrum

phoenix-on-calcite-hadoop-summit-2016Maryann Xue

Cost-based Query Optimization in Apache Phoenix using Apache CalciteJulian Hyde

Cost-based Query Optimization DataWorks Summit/Hadoop Summit

Cost-Based query optimizationDataWorks Summit/Hadoop Summit

9-Query Processing-05-06-2023.PPTvenkatapranaykumarGa

Sedna XML Database: Query Parser & Optimizing RewriterIvan Shcheklein

cheat-sheets.pdfFabianaFCordeiro

Software development - the java perspectiveAlin Pandichi

Einführung in MongoDBNETUserGroupBern

MongoDB .local London 2019: Fast Machine Learning Development with MongoDBMongoDB

MongoDB .local London 2019: Fast Machine Learning Development with MongoDBLisa Roth, PMP

Query optimization to improve performance of the code executionAlexander Decker

11.query optimization to improve performance of the code executionAlexander Decker

MongoDB - Aggregation PipelineJason Terpko

Practical catalystdwm042

Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache BeamFlink Forward

More from Chris Westin (20)

Data torrent meetup-productionengChris Westin

GripshortChris Westin

Ambari hadoop-ops-meetup-2013-09-19.finalChris Westin

Cluster management and automation with cloudera managerChris Westin

Building low latency java applications with ehcacheChris Westin

SDN/OpenFlow #lspeChris Westin

cfengine3 at #lspeChris Westin

This document discusses new capabilities in CFEngine 3, an advanced configuration management system. Key points include: - CFEngine 3 is declarative, ensures desired state is reached through convergence, is lightweight using 3-6MB of memory, and can run continuously to check configurations every 5 minutes. - It supports both new platforms like ARM boards and older systems like Solaris. - Recent additions allow managing resources like SQL databases, XML files, and virtual machines in a code-free manner using the Design Center. - CFEngine treats all resources like files, processes, and VMs as maintainable and ensures they self-correct through convergence to the desired state.

Nimbula lspe-2012-04-19Chris Westin

mongodb-brief-intro-february-2012Chris Westin

Stingray - Riverbed TechnologyChris Westin

Replication and replica setsChris Westin

Replication in MongoDB allows for high availability and scaling of reads. A replica set consists of at least three mongod servers, with one primary and one or more secondaries that replicate from the primary. Writes go to the primary while reads can be distributed to secondaries for scaling. Replica sets are configured and managed through shell helpers, and maintain consistency through an oplog and elections when the primary is unavailable.

Architecting a Scale Out Cloud Storage SolutionChris Westin

FlashCacheChris Westin

Large Scale CactiChris Westin

MongoDB: An Introduction - July 2011Chris Westin

Practical Replication June-2011Chris Westin

Replication in MongoDB allows for high availability and scaling of reads. A replica set consists of at least three mongod servers, with one primary and one or more secondaries that replicate from the primary. The primary applies all write operations to its oplog, which is then replicated to the secondaries. If the primary fails, a new primary is elected from the remaining secondaries. Administrative commands help monitor and manage the replica set configuration.

MongoDB: An Introduction - june-2011Chris Westin

Ganglia Overview-v2Chris Westin

Mysql Proxy Presentation YahooChris Westin

Mysql proxy presentation_yahooChris Westin

Data torrent meetup-productionengChris Westin

GripshortChris Westin

Ambari hadoop-ops-meetup-2013-09-19.finalChris Westin

Cluster management and automation with cloudera managerChris Westin

Building low latency java applications with ehcacheChris Westin

SDN/OpenFlow #lspeChris Westin

cfengine3 at #lspeChris Westin

Nimbula lspe-2012-04-19Chris Westin

mongodb-brief-intro-february-2012Chris Westin

Stingray - Riverbed TechnologyChris Westin

Replication and replica setsChris Westin

Architecting a Scale Out Cloud Storage SolutionChris Westin

FlashCacheChris Westin

Large Scale CactiChris Westin

MongoDB: An Introduction - July 2011Chris Westin

Practical Replication June-2011Chris Westin

MongoDB: An Introduction - june-2011Chris Westin

Ganglia Overview-v2Chris Westin

Mysql Proxy Presentation YahooChris Westin

Mysql proxy presentation_yahooChris Westin

Recently uploaded (20)

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.

Hands On: Create a Lightning Aura Component with force:RecordDataLynda Kane

#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018Lynda Kane

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

Build Your Own Copilot & Agents For DevsBrian McKeiver

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

"PHP and MySQL CRUD Operations for Student Management System"Jainul Musani

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Building 10x Organizations with Modern Productivity Metrics 10x developers may be a myth, but 10x organizations are very real, as proven by the influential study performed in the 1980s, ‘The Coding War Games.’ Right now, here in early 2025, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience. It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them. But which of these approaches actually work? DORA? SPACE? DevEx? What should we invest in and create urgency behind today, so that we don’t find ourselves having the same discussion again in a decade?

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Datastucture-Unit 4-Linked List Presentation.pptxkaleeswaric3

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

How Can I use the AI Hype in my Business Context?Daniel Lehner

𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨? Everyone’s talking about AI but is anyone really using it to create real value? Most companies want to leverage AI. Few know 𝗵𝗼𝘄. ✅ What exactly should you ask to find real AI opportunities? ✅ Which AI techniques actually fit your business? ✅ Is your data even ready for AI? If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.

Learn the Basics of Agile Development: Your Step-by-Step GuideMarcel David

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...Fwdays

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

Talk at the final event of Data Fusion Dynamics: A Collaborative UK-Saudi Initiative in Cybersecurity and Artificial Intelligence funded by the British Council UK-Saudi Challenge Fund 2024, Cardiff Metropolitan University, 29th April 2025 https://alandix.com/academic/talks/CMet2025-AI-Changes-Everything/ Is AI just another technology, or does it fundamentally change the way we live and think? Every technology has a direct impact with micro-ethical consequences, some good, some bad. However more profound are the ways in which some technologies reshape the very fabric of society with macro-ethical impacts. The invention of the stirrup revolutionised mounted combat, but as a side effect gave rise to the feudal system, which still shapes politics today. The internal combustion engine offers personal freedom and creates pollution, but has also transformed the nature of urban planning and international trade. When we look at AI the micro-ethical issues, such as bias, are most obvious, but the macro-ethical challenges may be greater. At a micro-ethical level AI has the potential to deepen social, ethnic and gender bias, issues I have warned about since the early 1990s! It is also being used increasingly on the battlefield. However, it also offers amazing opportunities in health and educations, as the recent Nobel prizes for the developers of AlphaFold illustrate. More radically, the need to encode ethics acts as a mirror to surface essential ethical problems and conflicts. At the macro-ethical level, by the early 2000s digital technology had already begun to undermine sovereignty (e.g. gambling), market economics (through network effects and emergent monopolies), and the very meaning of money. Modern AI is the child of big data, big computation and ultimately big business, intensifying the inherent tendency of digital technology to concentrate power. AI is already unravelling the fundamentals of the social, political and economic world around us, but this is a world that needs radical reimagining to overcome the global environmental and human challenges that confront us. Our challenge is whether to let the threads fall as they may, or to use them to weave a better future.

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

Network Security. Different aspects of Network Security.gregtap1

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

Hands On: Create a Lightning Aura Component with force:RecordDataLynda Kane

#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018Lynda Kane

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

Build Your Own Copilot & Agents For DevsBrian McKeiver

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

"PHP and MySQL CRUD Operations for Student Management System"Jainul Musani

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Datastucture-Unit 4-Linked List Presentation.pptxkaleeswaric3

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

How Can I use the AI Hype in my Business Context?Daniel Lehner

Learn the Basics of Agile Development: Your Step-by-Step GuideMarcel David

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...Fwdays

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

Network Security. Different aspects of Network Security.gregtap1

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

MongoDB Aggregation MongoSF May 2011

2. What problem are we solving?Map/Reduce can be used for aggregation…Currently being used for totaling, averaging, etcMap/Reduce is a big hammerSimpler tasks should be easierShouldn’t need to write JavaScriptAvoid the overhead of JavaScript engineWe’re seeing requests for help in handling complex documentsSelect only subdocuments or arrays

3. How will we solve the problem?Our new aggregation frameworkDeclarative frameworkNo JavaScript requiredDescribe a chain of operations to applyExpression evaluationReturn computed valuesFramework: we can add new operations easilyC++ implementationHigher performance than JavaScript

4. Aggregation - PipelinesAggregation requests specify a pipelineA pipeline is a series of operationsConceptually, the members of a collection are passed through a pipeline to produce a resultSimilar to a command-line pipe

5. Pipeline Operations$matchUses a query predicate (like .find({…})) as a filter$projectUses a sample document to determine the shape of the result (similar to .find()’s optional argument)This can include computed values$groupAggregates items into buckets defined by a key

6. Computed ExpressionsAvailable in $project operationsPrefix expression languageAdd two fields: $add:[“$field1”, “$field2”]Provide a value for a missing field: $ifnull:[“$field1”, “$field2”]Nesting: $add:[“$field1”, $ifnull:[“$field2”, “$field3”]]Other functions….And we can easily add more as required

7. Projections$project can reshape results$unwind expression doles out array values one at a timePull fields from nested documents to the topPush fields from the top down into new virtual documents

8. Grouping$group aggregation expressionsTotal of column values: $sumAverage of column values: $avgCollect column values in an array: $push

9. Demo(See script at https://gist.github.com/993733)

10. Usage TipsUse $match in a pipeline as early as possibleThe query optimizer can then be used to choose an index and avoid scanning the entire collection

11. Driver SupportInitial version is a commandFor any language, build a JSON database object, and execute the command{ aggregate : <collection>, pipeline : {…} }Beware of command result size limit

12. When is this being released?In final development nowExpect to see this in the near future

13. Sharding supportInitial release will support shardingMongos analyzes pipeline, and forwards operations up to $group to shards; combines shard server results and continues

14. Pipeline Operations – Future Plans$sortSorts the document stream according to a key$outSaves the document stream to a collectionSimilar to M/R $out, but with sharded output

15. Expressions – Future PlansDate field extractionGet year, month, day, hour, etc, from DateDate arithmetic

MongoDB Aggregation MongoSF May 2011

Recommended

More Related Content

What's hot (20)

Viewers also liked (11)

Similar to MongoDB Aggregation MongoSF May 2011 (20)

More from Chris Westin (20)

Recently uploaded (20)

MongoDB Aggregation MongoSF May 2011