Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas

What’s new with MongoDB &
Atlas, for you.
Sigfrido “Sig” Narváez
Executive Solution Architect, MongoDB
@signarvaez
DataConLA 2022

Agenda
MongoDB Developer Data Platform
For Platform Developers
For Data Engineers & Scientists
For Mobile Developers
For DevOps & Security Engineers
What’s new with MongoDB & Atlas

Traditional relational strengths
Expressive query
language and
secondary
indexes
Enterprise
security and
management
ACID
transactions
Additional NoSQL and big
data strengths and innovations
Native data
structures and
idiomatic drivers
Laptop, your data
center, multi-cloud
Scale and
distribution
Mission critical and general purpose

Time Series collections
Clustered indexes
Window functions
Live resharding
Client-Side FLE KMIP & cloud KMS
Atlas Serverless (preview)
Atlas Search fast facets, function
scores, synonyms
Long running snapshot reads
Sharded $lookup & $graphlookup
Majority write concern default
4x faster initial sync
Schema validation diagnostics
New MongoDB Shell GA
Resumable index builds
Rewritten Swift driver
Rewritten C# LINQ provider and
.NET Analyzer
PyMongoArrow API
New accumulator operators
Charts on Data Lake
x509 certiﬁcate rotation
Auditing ++
Atlas K8S controller
Ops Manager migration wizard
Doc-Level Concurrency
RAFT / Fast Failover
$lookup
Ops Manager
Compression
≤50 replicas
Aggregation ++
Encrypted and In-Memory
storage engines
BI Connector
Compass
2015 2016 2017 2018 2019 2020 2021-2022
3.0 & 3.2
WiredTiger
(Acquisition +
Integration) MongoDB Atlas ACID (Transactions)
Stable API & Rapid
Releases
Views Graph Processing
Zones ++Aggregation ++
Auto-balancing ++
Linearizable Reads Decimal
Intra-cluster Compression Log
Redaction Spark Connector
++ BI Connector ++
3.4
Change Streams
Retryable Writes
Schema Validation Expressive
$lookUp
Query Expressivity
Causal Consistency Consistent
Sharded Secondary Reads
Query Advisor
End to End Compression
WiredTiger 1m+ Collections
MongoDB BI Connector ++
R Driver
Charts (post GA)
Atlas X-Region Replication Atlas
Auto Storage Scaling
3.6
Replica set transactions
Atlas global clusters
40% Faster Shard Migrations
Atlas HIPAA
Atlas LDAP
Atlas Audit
Atlas Enc. Storage Engine
Atlas Backup Snapshots
Type Conversions
Snapshot Reads
Non-Blocking Sec. Reads
SHA-2 & TLS 1.1+
Compass Agg Pipeline Builder
Compass Export to Code
Free Monitoring Cloud Service
Ops Manager K8s Beta
4.0
Distributed Transactions
Client-Side Field Level Encryption
Materialized Views
Wildcard Indexes
Global PIT Reads
Large Transactions
Mutable Shard Key Values
Atlas Data Lake (Beta)
Atlas Auto Scaling (Beta)
Atlas Search (Beta)
Multi-CAs
Expressive Updates
Apache Kafka Connector
MongoDB Charts GA
Retryable Reads & Writes
New Index Builds
10x Faster stepDown
Storage Node Watchdog
Zstandard Compression
4.2
Union
Custom Agg Expressions
Reﬁnable Shard Keys
Compound Hashed Shard Keys
Mirrored Reads
Hedged Reads
Resumable Initial Sync
Time-Based Oplog Retention
Simultaneous Indexing
Hidden Indexes
Streaming Replication
Global Read/Write Concerns
Rust & Swift Drivers GA
TLS 1.3 & Faster Client Auth
OCSP Stapling
Kerberos Utility
Atlas Online Archive
Auto-Scaling
Schema Recommendations
AWS IAM Auth & Atlas x509
Federated Queries
Multi-cloud clusters
4.4 5.x
The evolution of MongoDB
Multi-Cloud
2022-2023
6.x
Queryable encryption (preview)
Atlas Serverless GA
Column store indexes
Cluster-to-cluster sync
Independent analytic nodes
scaling
Flexible sync GA
Asymmetric sync preview
Kotlin and Flutter SDKs for Realm
$lookup perf improvements
Atlas CLI GA
Atlas Data API GA
Secondary index on
measurements for time series
Time series read perf +
Atlas Data Lake with fully
managed storage
Atlas Data Federation
Atlas Charts dashboard
embedding
Atlas SQL interface
Encrypted audit log
Change streams improvements
Announcement of relational to
MongoDB migrator
Developer Data
Platform

MongoDB Atlas: global cloud database
Intelligent
performance
optimization
Simpliﬁed data
architecture
Best-in-class security
and operations
Global and
multi-cloud reach
95+ Regions

Application Application Application
Application
RDBMS RDBMS RDBMS RDBMS
Key-Value DB Graph DB
Wide Column
Document DB
Search Engine
Search Engine
Mobile DB
Edge DB
Enterprise Data Warehouse
ETL
ETL
Caching Layer Caching Layer
Sync
ETL
ETL
Search Engine
Time Series DB
ETL
Sync
Data Lake
ETL ETL ETL
ETL
ETL
MongoDB’s Developer Data Platform aims to Reduce Technology Sprawl

Application D
Application C
Application B
Application A
Application D

Accelerate innovation with
One interface.
For any application.
Anywhere.
The MongoDB Atlas
Developer Data Platform

Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas

{
"_id" : ObjectId("5ad88534e3632e1a35a58d00"),
"name" : {
"first" : "John",
"last" : "Doe" },
"address" : [
{ "location" : "work",
"address" : {
"street" : "16 Hatfields",
"city" : "London",
"postal_code" : "SE1 8DJ"},
"geo" : { "type" : "Point", "coord" : [
-0.109081, 51.5065752]}},
+ {...}
],
"dob" : ISODate("1977-04-01T05:00:00Z"),
"retirement_fund" : NumberDecimal("1292815.75")
}
Contrasting
data models
Tabular (Relational) Data
Model
Related data split across multiple
records and tables
Document Data Model
Related data contained in a single, rich document

Basics: Replication & Sharding
Primary
Secondary
Secondary
Replication
Application
Driver
MongoS
Primary
Secondary
Secondary
Shard 1
Primary
Secondary
Secondary
Shard 2
Primary
Secondary
Secondary
Shard N
MongoS •••
•••
•••
Application
High availability
Replica sets
Horizontal scalability
Sharded Clusters
3 Sharding Strategies: Range, Hash & Zone
collection-level
Application
Driver
Application

Tunable Consistency
Highest Data Safety
Financial Transactions, PII, etc.
writeConcern: majority
readConcern: majority
Lowest-latency Reads
eCommerce & Media Catalogs, Content, etc
readPreference: nearest
Fastest Ingestion with tolerance for loss
IoT, Viewer/Gamer Telemetry
writeConcern: 0
readConcern: local

ACID Transactions
try (ClientSession clientSession = client.startSession()) {
clientSession.startTransaction();
collectionA.insertOne(clientSession, docOne);
collectionA.insertOne(clientSession, docTwo);
collectionB.updateOne(clientSession, findQry, updateStmt);
collectionC.deleteOne(clientSession, findDelQry);
clientSession.commitTransaction();
}

Schema Design Patterns: Subset
Subset Pattern
You want to display dependent information, however
only part of it. 2+ strong entities.
•The rest of the data is fetched only if needed
•Examples:
‒ The Cast of a Movie
‒ Last 10 Movies an Actor has starred in
https://www.mongodb.com/blog/post/
building-with-patterns-a-summary

Schema Design Patterns: Subset
Subset Pattern
You want to display dependent information, however
only part of it. 2+ strong entities.
•The rest of the data is fetched only if needed
•Examples:
‒ The Cast of a Movie
‒ Last 10 Movies an Actor has starred in
https://github.com/snarvaez/
mdb-distributedtx-subset
https://www.mongodb.com/blog/post/
building-with-patterns-a-summary
Multi-Cloud Distributed ACID Transaction

Database driver and
query language
Search driver and
query language
Architectural complexity
Difficult to keep data in sync between
two separate systems. Requires its
own systems and skills
Operational overhead
More to provision, secure, upgrade,
patch, back up, monitor, scale, etc.
Lower developer productivity
Developers need to use different
query APIs for database and search
and coordinate schema changes
Application
Database Cluster Data Sync Search Cluster
Pain of bolting on a search engine to a database

Application
Same driver and
query API
Higher developer productivity
Build database and search features
using the same query API
Fully managed platform
Get the security, performance, and
reliability of Atlas
Simpliﬁed data architecture
Automatic data synchronization, even
as your data and schema changes
Atlas Database Atlas Search
MongoDB Atlas
MongoDB Atlas: integrated database and search

Atlas Search Architecture Atlas Search offers the features needed
to ﬁne-tune your search results to help
users ﬁnd what they need, including:
• Fuzzy search
• Autocomplete
• Filters and facets
• Synonyms
• Multi-data type support
• Multi-language support
• Highlighting
• Custom scoring
• Rich query language
• More-Like-This
• Dense Vector Search (Q3’22)

Just like that! Local Atlas Search Query:
db.getMongo().setReadPref("nearest");
db.movies.aggregate([
{ "$search": {
"compound": {
"must": [{
"text": {
"query": arg,
"path": ["plot","fullplot"],
"fuzzy": {
"maxEdits": 1,
"prefixLength": 1,
"maxExpansions": 256
}}}],
"should": [{
"text": {
"query": arg,
"path": "title",
"fuzzy": {
"maxEdits": 1,
"prefixLength": 1,
"maxExpansions": 256
}}}]
}}
}]);

At the end of the day, building
applications boils down to
accessing your data from the
database from your app in a
scalable and seamless manner

The core Atlas Application Services…
Database and
Scheduled Events can
be processed by
serverless functions
Data Triggers
Out-of-the-box
schema-based API
with ability to
customize with
custom resolvers
GraphQL API
Out-of-the-box
ﬂexible-based API
with ability to
customize with HTTPS
Endpoints
Data API
Seamless mobile
client sync without the
REST APIs or conﬂict
resolution
Device Sync

…are supercharged by the supporting services
Email Password, API
Key, Custom Function,
JWT Authentication -
help provide identities
to the client and IP
Access list
Auth/Security
Deﬁne what data can
and cannot be
accessed
Rules/Permissions
Access the core
services from native
client libraries (React
Native, Web Browser,
iOS, Android, .NET,
Flutter)
SDKs
Can be run in the
context of Triggers,
APIs, or from SDKs
Serverless Functions

Integrate all of Atlas into your apps via a single endpoint
Atlas GraphQL API
GraphQL API
Serverless Compute
Layer over HTTPS
Database
Data Lake
Frontends
Federate into a
larger supergraph
Microservices and
3rd party services
Search

The best way to visualize MongoDB data
Atlas
Charts
Fast
Easy
Powerful
Built for the
Document Model
No Data Movement
or Duplication
Integrated into
MongoDB Atlas
Share
& Collaborate
Drive Actionable
Insights
Create Engaging
Data Experiences

Atlas Charts in 2022
Dashboard
Embedding
Allow users to embed
a full dashboard
rather than a single
chart
Q1 2022 Q1 2022 H1 2022 2022 2022
Query Library
Reuse queries in
multiple charts
Streamlined Data
Sources
Create charts without
explicitly creating a
data source
Charts API Scheduled and
Async Queries
2022
Auto suggest
Charts for new
users
2022
Enterprise
Sharing

● Model their application
data as Documents
● Use MQL and the
Aggregation
Framework
● Want to query multiple
operational databases
with SQL-based BI Tools
● No time to learn or
switch to MQL
RESULT: Productivity RESULT: Friction
Developers SQL Users
Challenge: MongoDB + SQL Users

Atlas SQL Interface
Visualize data using Named
Connectors for BI tools
Use ODBC/JDBC drivers
for other SQL connections
Use the Atlas SQL interface to unlock access to
Atlas data from SQL based tools and SQL
centric users so that we can embrace the
ecosystem of people that want access to data
in our platform.
BI Tools ODBC/JDBC Drivers
Atlas SQL provides a ﬁrst class SQL
experience for querying Atlas data
Atlas SQL Interface is currently in preview
Atlas SQL
interface
Connect to the Atlas SQL interface from the most popular BI tools

Step 3:
Add Named Connector
-or-
Connect via
JDBC/ODBC Driver
Atlas SQL
Drivers
Atlas Atlas SQL Interface BI or
Analytics
Tool
Atlas SQL
Connectors
Step 1:
Enable SQL Interface
Step 2:
Download Driver
Atlas SQL Connectors & Drivers
Atlas SQL Interface
Leverages Atlas Data Federation
as the query engine
Use MongoSQL, a MongoDB-Native SQL-92 compatible dialect
How does it work?
MongoDB
ODBC
MongoDB
JDBC
Tableau
Connector
PowerBI
Connector

Atlas Data
Federation • Federate queries across Atlas and AWS S3
without data movement or transformation to
power real-time applications.
• Seamlessly aggregate data across 1+
sources. Gather insight, and enrich data
without complex data pipelines.
• Convert MongoDB data to a columnar ﬁle
format and output to S3 to be consumed
by downstream teams for enabling
operational insights and modern app
experiences.

Control Plane
MongoDB Atlas Control
{Data Federation
Endpoint}
Data Plane
Atlas Data Federation architecture
Application & Services
Apps
• Analytic Engines
• Data Products
MongoDB Compass
• Data Exploration
MongoDB SQL Support
• Business Analytics
• Reporting
Data Science Tools
• Statistics
• Machine Learning
DRIVERS:
Javascript, Perl,
Python, C, C++,
Java, Ruby, Go,
Scala, R
MONGODB SHELL
Compute Plane
Data Federation
computer node
Data Federation
computer node
Data Federation
computer node
Data Federation
computer node
Data Federation
computer node
Data Federation
computer node
Data Federation
computer node
Data Federation
computer node
Data Federation
computer node
Region 1
Region 2
Atlas Clusters
HTTP Store
BYO AWS S3
or
Atlas Data Lake
1+
Any Atlas
Cluster Type
1+

Atlas Data Lake is a combination of
several pieces of new functionality:
● Automated Data Extraction
● Partition Level Indexing
● MongoDB Native Analytic File
Format
● Object Storage
This set of functionality provides you a
workload isolated view of your cluster
data in economical object storage with
high performance for analytic type
queries.
Atlas Data Lake
A fully managed, high
performance, analytic
storage option in Atlas,
powered by cost-effective
cloud object storage.

PARTITION
INDEXES
Write
Metadata
Write
Data
Atlas
CLOUD DATABASE
AWS
S3
(Future)
A B C A B C A B C
Enriched Extraction, Partitioning & Auto-Balancing
ANALYTIC
FILE FORMAT

What is in a Partition Index?
● Unique Values or Ranges
● Sum
● Count
● Average
● ...
Improve query performance
● Improve point query
performance by ﬁnding
partitions faster.
● Additionally, certain aggregate
queries like Counts and Sums are
“covered” by the index.
FEDERATED
ENDPOINT
PARTITION INDEXES
COLUMNAR PARTITIONS
Note: All components of Atlas Data Lake are fully managed solutions that exist in the Atlas Infrastructure.
{
name: [‘ben’,
‘john’, ‘james’]
age: 34-75
}
{
name: [‘prem’,
‘sally’]
age: 23-75
}
{
name: [‘tifani’,
‘joe’]
age: 23-75
}
A B C A B C A B C
Partition Indexes

MongoDB
Spark Connector
• Massively parallel processing, machine learning
and streaming at scale
• Process data “in place” avoiding the latency of ETL
• Aggregation pre-ﬁltering with secondary indexing
to only select data that’s required
• Read from secondaries isolate analytics workload
from business-critical operations
• Shard aware for data locality
SQL
Machine
Learning
Streaming Graph
Scala, Java, Python, R APIs
MongoDB Spark Connector
MongoDB Spark Connector

MongoDB Kafka Connector
Sink:
Source:
MongoDB Database
MongoDB
Connector
topicA
topicB
topicC
Kafka Cluster
Writes documents
to DB collection
Receives events from
Kafka Topic(s)
MongoDB Database
MongoDB
Connector
Kafka Cluster
Receives documents
from DB collection
Writes events to
Kafka Topics(s)
topicA
topicB
topicC
Change
Streams
Built in partnership with Conﬂuent

Analytical Nodes
Analytics
Consumers
Primary Secondary Secondary Secondary
{Analytics Node}
Transactional
Applications
Secondary
{Analytics Node}
Rich MongoDB Query API and distributed architecture
allows you to run both Transactions and Analytics on
the same cluster with no resource contention
Higher Tier
(RAM / CPU) for
demanding
analytical & ML

Atlas → Atlas
On-prem → Atlas On-prem → On-prem
On-prem
cluster
Atlas cluster
Atlas cluster Atlas cluster
On-prem
cluster
On-prem
cluster
Atlas → On-Prem
Atlas cluster On-prem
cluster
Cluster-to-Cluster Sync

Real-time experiences at scale require
complex sync & conflict resolution
Data
synchronization
JSON
NATIVE OBJECT
ORM
MSG QUEUE
BACKEND OBJECT
REST CALL
TO API
PUSH SERVICE
PUSH
NOTIFICATION
COMPLEX RETRY &
ERROR HANDLING
POTENTIAL APP CRASH
OFFLINE?
WRITE CONFLICTS?
INVALID DATA?
SERVER
ERROR?
TIMED OUT?
Backend database
WRITE CONFLICTS?
Mobile devices
with local data

The easiest way to build and scale mobile apps
Atlas Device Sync
Out-of-the-box device-cloud
synchronization
Atlas
The leading
Developer Data Platform
on the cloud
Realm
Mobile-optimized
data store
Lightning fast
Super lightweight
No ORM needed
No DAO needed Dynamic, ﬂexible sync options
Granular permissions
Built-in conﬂict resolution
Built-in error handling

Supported
frameworks
and languages
Kotlin/Kotlin Multiplatform
Swift/SwiftUI
.NET
React Native
Node.js / Javascript
Dart/Flutter (beta)
Xamarin
Unity
Java
Ionic
MAUI

Provides a single data layer across all platforms
WatchOS
Mobile devices
(iOS, Android)
Tablets
(iOS, Android, Windows)
Laptops, Desktops POS Systems
IoT (connected cars,
wearables,
equipment, etc.)
Atlas Device Sync
Atlas

How Device Sync works: dynamic queries
Customer
Store
Warehouse
Delivery.date == 01-01-2022
Inventory.store == 101
category == 'jeans'
AND size == 8
AND price < 40
Atlas Device Sync
MongoDB Atlas
Atlas Device Sync
Atlas Device Sync

How Device Sync works: hierarchical permissions
VP
Manager
Sales
Atlas Device Sync
MongoDB Atlas
Atlas Device Sync
Atlas Device Sync
Sales.team == 'NA-US'
Sales.team == 'US-WEST'
Sales.owner ==
'Bill Buckner'

How Device Sync works: document-level permissions
Banker
READ: ALL FIELDS
WRITE: ALL FIELDS
MongoDB Atlas
Atlas Device Sync
Trades
Support
Bank
User
Support
READ: ALL FIELDS
WRITE: SOME FIELDS
End User
READ: ALL FIELDS
WRITE: PROFILE ONLY

{
"_id": {
"$oid": "60e51a64d0ff67f998bc1732"
},
"name": "Jane Leaf",
"address": {
"street": "Bedford Ave"
"city": "New York"
}
"billing": {
"creditCard": "1234123412341234"
"balance": "20"
}
"vitals": [{
"visitDate": "May62022”
"bpm": "80”
"bloodPressure": "120/80"
]}
"prescriptions":[{
"name": "tylenol”
"quantity": "20”}
]}
}
How Device Sync works: field-level permissions
Patient
{
"_id": {
"$oid": "60e51a64d0ff67f998bc1732"
},
"address": {
"street": "Bedford Ave"
"city": "New York"
}
"billing": {
"creditCard": "1234123412341234"
"balance": "20"
}
}
Billing
{
"_id": {
"$oid": "60e51a64d0ff67f998bc1732"
},
"vitals": [{
"visitDate": "May62022”
"bpm": "80”
"bloodPressure": "120/80"
]}
"prescriptions":[{
"name": "tylenol”
"quantity": "20”}
]}
}
Doctor
ID
Address
Billing
Vitals
Prescription

Business Continuity
● 3 AZ’s default
● 3 Clouds: AWS, Google, Azure
● Cross-Region, Cross-Cloud &
Multi-Cloud
● Automatic-failover
● Continuous Backup
○ Second-resolution RPO <<<
● Auto Scale Compute & Storage
● Atlas Serverless

Automation, APM &
Integrations
● K8S Operator
● Hashicorp Terraform & Vault
● AWS CloudFormation
● REST API & CLI
● Built-in Metrics, Alerts,
Performance Advisor
● DataDog, Prometheus,
PagerDuty, etc.

DEMO
Migration to Atlas
● Live Migrate (hosted by Atlas)
● Push Migrate (from OpsMgr or Cloud Mgr)
-Enterprise customers - NEW!!
● Cluster to Cluster Replication - NEW!!!
● mongomirror (hosted by you)

MongoDB Security Assurance
HITRUST
https://www.mongodb.com/cloud/trust

What’s so hard about
“right to erasure”?
Deleting user data from the
database is easy
• But what about user data stored in
backups and logs... how do you delete
that?
• Risk of archived data being restored
and used
With Client-side FLE, you simply
destroy the encryption key for the
user’s data, and their PII is
unusable and unrecoverable:
in the DB, in backups, in logs…
...everywhere

Network (In-Flight) Storage (At-Rest) Memory (In-Use) Encryption Scope
TLS ❌ ❌
All data transmitted
over the network
Volume and
Storage Engine
❌ ❌
All data stored in the
database cluster
Client-Side FLE
Speciﬁc ﬁelds
containing
sensitive data
Addressing different threat models
End-to-end encryption for defense in depth

db.runCommand(“Thank you!!”);
db.QnA.find({Q:“answer”}).explain();
Sig Narváez
Executive Solution Architect
Aug 13 2022
Sigfrido “Sig” Narváez
Executive Solution Architect, MongoDB
@signarvaez

Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas

Recommended

More Related Content

What's hot (20)

Similar to Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas (20)

More from Data Con LA (20)

Recently uploaded (20)

Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas