An Enterprise Architect's View of MongoDB

An Enterprise Architect’s View of
MongoDB
Matt Kalan
Business Architect
matt.kalan@mongodb.com
@matthewkalan

Agenda
• Modern drivers of change on enterprises
• Requirements these create
• How traditional databases are handling changes
• New capabilities needed
• How MongoDB provides these capabilities
• Case studies
• Enterprise adoption
2

More Technologies and Requirements
Than Ever
NoSQL
Datawarehouse
Hadoop
Internet of Things
4
Document Data Stores
Big Data
Key-value
JSON
Wide-column
MongoDB
Cloud Computing
Mobile
Gamification
Social networking
Graph
Agile Development
ODS
Analytics
Consumerization

More Technologies and Requirements
Than Ever
NoSQL
Datawarehouse
Hadoop
Internet of Things
5
Document Data Stores
Big Data
Key-value
JSON
Wide-column
MongoDB
Cloud Computing
Mobile
Gamification
Social networking
Graph
Agile Development
ODS
Analytics
Consumerization
Opportunity cost
Globalization
Customer 360
Cross-channel
New Revenue Streams
Faster Competition
Emerging markets
Regulation
Data Monetization
More with less
Common Services
Empowered customers
Lowering TCO

Questions for Enterprise Architects
• What current and future requirements does all
6
this raise?
• How to prepare my enterprise to handle these?
• Which technologies and products will help me?
• How to bring them into my enterprise
successfully?
• How does old and new technology work together?
• What does the future state architecture look like?

Modern Application Requirements
Data Types & OOP
• Object-oriented
• Variably structured
• Unstructured (not tabular)
7
Volume of Data
• Petabytes of data
• Trillions of records
• Millions of queries per
second
Agile Development
• Iterative
• Short development
cycles
• Fast time-to-market
New Architectures
• Horizontal scaling
• Commodity
servers
• Cloud computing
RDBMS
Single Views
• Disparate data
• Intraday
• Cross-channel/silo
• Global

Modern Application Requirements
Data Types & OOP
• Object-oriented
• Variably structured
• Unstructured (not tabular)
8
Volume of Data
• Petabytes of data
• Trillions of records
• Millions of queries per
second
Agile Development
• Iterative
• Short development
cycles
• Fast time-to-market
New Architectures
• Horizontal scaling
• Commodity
servers
• Cloud computing
RDBMS
Single Views
• Disparate data
• Intraday
• Cross-channel/silo
• Global

Impact of New Requirements Handled
with 40-year old Technology
• Customfield1…100 or separate tables
• Caching & ORMs
• Expensive hardware and storage
• Schema migration project
• One canonical schema
• Application-specific partitioning
• Use files instead of databases
• Schema change takes 6 months
9

Impact of New Requirements Handled
with 40-year old Technology
• Customfield1…100 or separate tables
• Caching & ORMs
• Expensive hardware and storage
• Schema migration project
• One canonical schema
• Application-specific partitioning
• Use files instead of databases
• Schema change takes 6 months
10
Slow time-to-market
Agility lost
High cost
Failed projects
Business frustrated

An Enterprise Architect's View of MongoDB

How Do I Prepare My Enterprise
for Modern Requirements?

What would we need to make it
easier?
13
New capabilities
• Dynamic and variable
schemas
• Richly-structured [object]
data
• Higher performance
• Easy horizontal scaling
• Low TCO

What would we need to make it
easier?
• Dynamic and variable
schemas
• Richly-structured [object]
data
14
New capabilities
• Higher performance
• Easy horizontal scaling
• Low TCO
Traditional capabilities
• Rich querying
• Strongly consistently data
• High availability
• Security

Documents Support Modern
Requirements
15
Relational Document Data Structure
{ customer_id : 1,
first_name : "Mark",
last_name : "Smith",
city : "San Francisco",
location : [40.74, -73.97],
image : <binary>,
phones: [ {
number : “1-212-777-1212”,
dnc : true,
type : “home”
},
{
number : “1-212-777-1213”,
type : “cell”
}]
}

Modern DB Architecture
16
Application
Driver
Query Router
Shard 1
Primary
Secondary
Secondary
Shard 2
Primary
Secondary
Secondary
…
Shard N
Primary
Secondary
Secondary
db.customer.insert({…})
db.customer.find({
name: ”John Smith”})
1.Dynamic Document
Schema
{ name: “John Smith”,
date: “2013-08-01”),
address: “10 3rd St.”,
phone: [
{ home: 1234567890},
{ mobile: 1234568138} ]
}
2. Native language drivers
4. High performance
- Data locality
- Rich Indexes
- RAM
3. High availability
- Replica sets
5. Horizontal scalability
- Sharding

MongoDB Built Exactly For These
Requirements
17
Application
Driver
Query Router
Shard 1
Primary
Secondary
Secondary
Shard 2
Primary
Secondary
Secondary
…
Shard N
Primary
Secondary
Secondary
db.customer.insert({…})
db.customer.find({
name: ”John Smith”})
1.Dynamic Document
Schema
{ name: “John Smith”,
date: “2013-08-01”),
address: “10 3rd St.”,
phone: [
{ home: 1234567890},
{ mobile: 1234568138} ]
}
2. Native language drivers
4. High performance
- Data locality
- Rich Indexes
- RAM
3. High availability
- Replica sets
5. Horizontal scalability
- Sharding

Performance
18
Better Data
Locality
In-Memory
Caching
In-Place
Updates

No SQL But Still Flexible Querying
19
MongoDB
Rich Queries
• Find anyone with phone # “212…”
• Check if the person with number
“555…” is on the “do not call” list
Geospatial
• Find the best offer for the customer at
geo coordinates of 42nd St. and 6th Ave
Text Search
• Find all tweets that mention the firm
within the last 2 days
Aggregation
• Count and sort number of customers by
city
Map Reduce
• For customers in each zip code, what
are the top 5 most common products
{ customer_id : 1,
first_name : "Mark",
last_name : "Smith",
city : ”New York",
phones: [ {
number : “1-212-777-1212”,
dnc : true,
type : “home”
},
{
number : “1-212-777-1213”,
type : “cell”
}]
}

Security Capabilities
• Kerberos
• LDAP
• x.509 certificates
20
• User-Defined Roles
• Field Level Security
• Admin Actions
• CRUD operations
• Partner support
• SSL support on wire
• Disk encryption
support by partners

Global Deployment with Local
Read/Writes
21
Primary:NYC
Primary:LON
Secondary:NYC
Primary:SYD
Secondary:LON
Secondary:NYC
Secondary:SYD
Secondary:LON
Secondary:SYD

MongoDB Business Value
22
Faster Time to Market Lower TCO
Enabling New Apps Faster Response Time
& Scalability

Database Landscape
24
2010
RDBMS
NoSQL
Key-Value/
Wide-column
Hadoop
OLAP/DW
2000
RDBMS
OLAP/DW
1990
RDBMS
Operational
Database
Datawarehousing
Document DB

Database Landscape
25
2010
RDBMS
NoSQL
Key-Value/
Wide-column
Hadoop
OLAP/DW
2000
RDBMS
OLAP/DW
1990
RDBMS
Operational
Database
Datawarehousing
Document DB

MongoDB Hadoop Connector
26
Operational
Database
• Low latency
• Rich fast querying
• Flexible indexing
• Aggregations in database
• Known data relationships
• Great for any subset of data
Analytics
• Longer jobs
• Batch analytics
• Highly parallel processing
• Unknown data relationships
• Great for looking at all data
MongoDB-Hadoop
Connector

Operational Database Use Cases
27
RDBMSs

28
RDBMSs
Key/Value or
Wide Column Stores

29
RDBMSs
MongoDB
Key/Value or
Wide Column Stores

MongoDB 5th Most Popular Database
30

Leading Organizations Rely on MongoDB
31

Architecture Patterns
1. Operational Data Store (ODS)
2. Enterprise Data Service
3. Datamart/Cache
4. Master Data Distribution
5. Single Operational View
33

Architecture Patterns
1. Operational Data Store (ODS)
2. Enterprise Data Service
3. Datamart/Cache
4. Master Data Distribution
5. Single Operational View
34
System of Record
System of Engagement

Architectural Pattern –
Operational Data Store (ODS)

Challenge: Applications not agile nor
scalable enough
36
Requirement changes
Change

Solution: Match dynamic data model
to the application
37

Criteria for benefitting most from
MongoDB instead of RDBMS
Data
 Variably or
unstructured
 Hierarchical
 Geo-coordinates
 Disparate sources
 Schema changes
often
38
Querying
 Real-time analytics &
aggregations
 Location-based
 Lowest latency
 Performance affects
user experience
Requirements
 Agile development &
fastest time-to-market
 Data will grow quickly
 Best performance for
request/response
 Lowest TCO
 Multiple sources
aggregated
 Challenges today with
RDBMS

ADP’s Global Mobile Platform
One of the world's largest providers of payments solutions
constructs a completely reliable and robust mobile
experience
39
Problem Why MongoDB Results
• Needed a signature
mobile app for customers
• Must support millions of
users
• Needed to quickly change
features & functionality
• High availability was
critically important
• Built-in high availability
architecture optimized for
global, multi-data center
distribution
• Dynamic schema & rich
querying – deep
functionality from launch &
new features easily added
• Much lower TCO,
especially with commodity
hardware
• iTunes App Store “Top 15”
business app since 2012
launch
• Over 1 million active users, 17
countries, 23 languages
• Extremely high performance
through predictive caching
• Maintenance much easier =>
simple codebase, less
hardware
• New functionality easy and
quick to add

Enterprise Data Service

Challenge: Siloed operational
applications
41
Silo 1 Data
Silo 2 Data
…
Silo N Data
Impact
• Views are siloed
• Duplicate management
and data access layer
• Need another layer to
aggregate
Silo 1 systems
Silo 2 Systems
…
Silo N
Systems
Reporting Reporting Reporting

Solution: Unified data services
42
…
Benefit
• Each application can still
save its own data
• Data is already aggregated
for cross-silo reporting
• One cluster and data access
layer to manage
Silo 1 Systems
Silo 2 Systems
…
Silo N Systems
Reporting
……

Case Study: Global Broker Dealer
Trade Mart for all OTC Trades
Distribute reference data globally in real-time for
fast local accessing and querying
43
• Each application had its
own persistence and
audit trail
• Wanted one unified
framework and
persistence for all
trades and products
• Needed to handle many
variable structures
across all securities
• Dynamic schema: can
save trade for all products
in one data service
• Easy scaling: can easily
keep trades as long as
required with high
performance
• Rich querying: can query
on any fields each
business requires
• Fast time-to-market using
the persistence framework
• Store any structure of
products/trades without
changing a schema
• One consolidated trade
store for auditing and
reporting

Datamart/Cache

Challenge: Response From Data
Warehouse or Other System is Slow
45
Cards
Loans
…
Deposits
Data
Warehouse
Issues
• Data stored normalized
• Reports slow to generate
• Data updated daily but user
response must be fast
Impact
• Lost productivity
• Dissatisfied users and
business
Reporting
Cards
Silo 1
Loans
Silo 2
Deposits
Silo 3

Solution: Optimize Data Structure as a
Datamart In-memory or On-disk
Cards
Loans
Deposits
46
…
Data
Warehouse
Solution
• Data stored in optimal
structure for reports
• Optionally in memory
Impact
• Response times is as fast
as possible
• Users and business
satisfied
Fast Reporting
Cards
Silo 1
Loans
Silo 2
Deposits
Silo 3
Datamart/Cache
…

Case Study: Global Bank -
Personalized In-memory Datamart
Needed fast reporting for finance on global
banking transaction data (about 2 petabytes)
47
• Data warehouse was
too slow for reporting
• No visibility into how
long reports took
• Could not generate
multiple ad hoc reports
• Users included
regulators so even
more demanding
• Dynamic schema: store
data in optimal structure
• Performance: storing
report results optimally
• In-memory caching of
results
• Rich querying: can query
on any field
• Easy scaling: results
spread across shards to
generate report in parallel
• Create a personalized in-memory
data mart
• Reports configured and
notified when results ready
• Data all in memory so fast
to manipulate
• Data spread across shards
for ultra-fast reporting

Master Data Distribution

Challenge: Master data can be hard
to change and distribute
49
Golden
Copy
Batch
Batch
Batch
Batch
Batch
Batch
Batch
Batch
Common issues
• Hard to change schema
of master data
• Data copied everywhere
and gets out of sync
Impact
• Process breaks from out
of sync data
• Business doesn’t have
data it needs
• Many copies creates
more management

Solution: Persistent dynamic cache
replicated globally
50
Real-time
Real-time Real-time
Real-time
Real-time
Real-time
Real-time
Real-time
Solution:
• Load into primary with
any schema
• Replicate to and read
from secondaries
Benefits
• Easy & fast change at
speed of business
• Easy scale out for one
stop shop for data
• Low TCO

Case Study: Global bank
Reference Data Distribution
Distribute reference data globally in real-time for
fast local accessing and querying
51
• Delays up to 36 hours in
distributing data by batch
• Charged multiple times
globally for same data
• Incurring regulatory
penalties from missing
SLAs
• Had to manage 20
distributed systems with
same data
• Dynamic schema: easy to
load initially & over time
• Auto-replication: data
distributed in real-time,
read locally
• Both cache and database:
cache always up-to-date
• Simple data modeling &
analysis: easy changes
and understanding
• Will save about
$40,000,000 in costs and
penalties over 5 years
• Only charged once for data
• Data in sync globally and
read locally
• Capacity to move to one
global shared data service

Single Operational View

Challenge: Aggregation of disparate
data is difficult
53
Cards
Loans
…
Deposits
Data
Warehouse
Batch
Cross-Silo
applications
Issues
• Yesterday’s data
• Details lost
• Inflexible schema
• Slow performance
Datamar
t
Datamar
t
Datamar
t
Batch
Impact
• What happened today?
• Worse customer
satisfaction
• Missed opportunities
• Lost revenue
Batch
Batch
Reporting
Cards
Silo 1
Loans
Silo 2
Deposits
Silo 3

Solution: Using dynamic schema and
easy scaling
54
Operational Single View Benefits
Data
Warehouse
Real-time or
Batch
…
Customer-facing
Applications
Regulatory
applications
• Real-time
• Complete details
• Agile
• Higher customer
retention
• Increase wallet share
• Proactive exception
handling
Strategic
Reporting
Operational
Reporting
Cards
Loans
…
Deposits
Customer
Accounts
Cards
Silo 1
Loans
Silo 2
Deposits
Silo N

Case Study
Insurance leader generates coveted 360-degree view of
customers in 90 days – “The Wall”
55
• No single view of
customer
• 145 yrs of policy data,
70+ systems, 15+ apps
• 2 years, $25M in failing
to aggregate in RDBMS
• Poor customer
experience
• Agility – prototype in 5
days; production in 90
days
• Dynamic schema:
Imperative to combine
disparate data
• Rich querying: necessary
for match data across silos
• Hot tech to attract top
talent
• Unified customer view
available to all channels
• Increased call center
productivity
• Better customer
experience, reduced
churn, more upsell opps
• Dozens more projects
on same data platform

Single [Operational] View of ….
Cards
Silo 1
Loans
Silo 2
56
Operational
Reporting
Real-time
or Batch
…
Single CSR
Application
Unified
Customer Portal
Operational Data Layer
Cards
Loans
…
Deposits
Deposits
Silo N
Strategic
Reporting
…
• Millisecond latency
• Request/response
• Easily scalable
• Flexible schema
• Low TCO
• Rich querying
• Globally distributed
DW/Analytic Data Layer
• Analytical/Offline processing
• 10s seconds to hours latency
• Also scalable, low TCO, and
flexible
• Pre-defined slices of data
(few indexes)
Analytics/Batch
processing
MongoDB
Hadoop Connector
…

Processing + Data Access Paradigm
Processing
model
Data access
model
57
Request/response
Map-reduce
Batch, ETL, etc.
Analytical Jobs
Latency important (e.g.
user waiting)
Milliseconds to seconds
Small to large subsets
of data
Indexes valuable
Multiple seconds to hours
Processing all or large sets
of data
Indexes not used

Processing
model
Data access
model
58
Request/response
Map-reduce
Batch, ETL, etc.
Analytical Jobs
user waiting)
of data
Indexes valuable
of data
Indexes not used
Typical MongoDB
Use Case

Processing
model
Data access
model
59
Request/response
Map-reduce
Batch, ETL, etc.
Analytical Jobs
user waiting)
of data
Indexes valuable
of data
Indexes not used
Typical MongoDB
Use Case
Typical Hadoop
Use Case

Processing
model
Data access
model
60
Request/response
Map-reduce
Batch, ETL, etc.
Analytical Jobs
user waiting)
of data
Indexes valuable
of data
Indexes not used
Typical MongoDB
Use Case
Typical Hadoop
Use Case

Processing
model
Data access
model
61
Request/response
Map-reduce
Batch, ETL, etc.
Analytical Jobs
user waiting)
of data
Indexes valuable
of data
Indexes not used
Typical MongoDB
Use Case
Typical Hadoop
Use Case
Data
Discovery

Example Adoption Path
Use of MongoDB
63
One Project
MongoDB CoE
A Few Projects
Certified
Widespread
Adoption
Operationally
Supported
Time
Defined

Traditional Data Integrity Enforcement
64
RDBMS
• Apps access DB directly
• Data Integrity must be in the RDBMS
• Schema implemented by a DBA
Application 1
Application 2
Application 3

Modern Apps (SOA) - Data Access
Layer Should Enforce Data Integrity
Application 1
65
MongoDB Cluster
Application 2
• Data Integrity and validations done in
• Implemented in code
Data
Access
Layer
…
Application N
…
Data Access Layer
REST/API/WS API on TCP/IP

Data Governance Benefits
• Greater adoption from natural developer
66
framework on common data models
• Easier for master data or upstream changes to
flow into MongoDB-backed apps
• MongoDB useful for distributing master data
• ETL providers support MongoDB most in NoSQL

Factors to Consider in Adoption
• SDLC and data governance for an application
• Enterprise-wide data governance (inter-app)
• Enterprise-wide security
• Roles and responsibilities
• Training requirements
• Operations/production support
• Center of Excellence (COE)
• Process for choosing which DB to use
• How to work with other technologies in-house
68

Recommended Center of Excellence
69
Database Engineering & CoE
Operational Database CoE
Datawarehousing CoE

70
Database
Advisory
Services
Datawarehousing CoE

71
RDBMS
Engineering
Database
Advisory
Services
Datawarehousing CoE

72
RDBMS
Engineering
Database
Advisory
Services
Database
SMEs
Datawarehousing CoE

73
Database
Advisory
Services
MongoDB
Engineering
RDBMS
Engineering
Database
SMEs
Datawarehousing CoE

74
Database
Advisory
Services
MongoDB
Engineering
RDBMS
Engineering
MongoDB
Incubator
(& cluster)
Database
SMEs
Datawarehousing CoE

75
Database
Advisory
Services
MongoDB
Engineering
RDBMS
Engineering
MongoDB
Incubator
(& cluster)
Database
SMEs
RDBMS PaaS
Engineering
MongoDBaaS
Engineering
Datawarehousing CoE

76
MongoDB
Engineering
RDBMS
Engineering
MongoDB
Incubator
(& cluster)
Database
SMEs
RDBMS PaaS
Engineering
MongoDBaaS
Engineering
DW Product
Engineering
Datawarehousing CoE
Hadoop
Incubator
Clusters
Database
SMEs
DW PaaS
Engineering
Hadoop PaaS
Engineering
Database
Advisory
Services

Summary
• Enormous technology and business change today
• Old technologies not suited for many of them
• MongoDB is purpose built for today and future applications
• And can help solve common architectural challenges
• Bring MongoDB, Inc. in to learn how to adopt it more widely
77
when appropriate
• Firms using MongoDB benefit from 50% time-to-market,
70% lower TCO, lower operating costs, and making the
infeasible possible

For More Information
79
Resource Location
Resource Location
MongoDB Downloads mongodb.com/download
Free Online Training university.mongodb.com
Webinars and Events mongodb.com/events
White Papers mongodb.com/white-papers
Case Studies mongodb.com/customers
Presentations mongodb.com/presentations
Documentation docs.mongodb.org
Additional Info info@mongodb.com

An Enterprise Architect's View of MongoDB

Recommended

More Related Content

What's hot (20)

Similar to An Enterprise Architect's View of MongoDB (20)

More from MongoDB (20)

Recently uploaded (20)

An Enterprise Architect's View of MongoDB

Editor's Notes