Schema Design by Chad Tindel, Solution Architect, 10gen

Real Time Analytics
Chad Tindel
chad.tindel@10gen.com

The goal
Real Time
Analytics Engine
Real Time
Analytics Engine
Data
Sourc
e
Data
Sourc
e
Data
Sourc
e

Simple log storage
Design Pattern

Aggregation - PipelinesAggregation - Pipelines
• Aggregation requests specify a pipeline
• A pipeline is a series of operations
• Conceptually, the members of a collection
are passed through a pipeline to produce
a result
– Similar to a Unix command-line pipe

Aggregation PipelineAggregation Pipeline

Aggregation - PipelinesAggregation - Pipelines
db.collection.aggregate(
[ {$match: … },
{$group: … },
{$limit: …}, etc
]

Pipeline OperationsPipeline Operations
• $match
– Uses a query predicate (like .find({…})) as a
filter
{ $match : { author : "dave" } }
{ $match : { score : { $gt : 50, $lte : 90 } } }

• $project
– Uses a sample document to determine the
shape of the result (similar to .find()’s 2nd
optional argument)
• Include or exclude fields
• Compute new fields
– Arithmetic expressions, including built-in functions
– Pull fields from nested documents to the top
– Push fields from the top down into new virtual documents

• $unwind
– Hands out array elements one at a time
{ $unwind : {"$myarray" } }
• $unwind “streams” arrays
– Array values are doled out one at time in the
context of their surrounding document
– Makes it possible to filter out elements before
returning

• $group
– Aggregates items into buckets defined by a
key

GroupingGrouping
• $group aggregation expressions
– Define a grouping key as the _id of the result
– Total grouped column values: $sum
– Average grouped column values: $avg
– Collect grouped column values in an array or
set: $push, $addToSet
– Other functions
• $min, $max, $first, $last

• $sort
– Sort documents
– Sort specifications are the same as today,
e.g., $sort:{ key1: 1, key2: -1, …}
{ $sort : {“total”:-1} }

• $limit
– Only allow the specified number of documents
to pass
{ $limit : 20 }

• $skip
– Skip over the specified number of documents
{ $skip : 10 }

Computed ExpressionsComputed Expressions
• Available in $project operations
• Prefix expression language
– Add two fields: $add:[“$field1”, “$field2”]
– Provide a value for a missing field: $ifNull:
[“$field1”, “$field2”]
– Nesting: $add:[“$field1”, $ifNull:[“$field2”,
“$field3”]]
(continued)

Computed ExpressionsComputed Expressions
(continued)(continued)
• String functions
– toUpper, toLower, substr
• Date field extraction
– Get year, month, day, hour, etc, from ISODate
• Date arithmetic
• Null value substitution (like MySQL ifnull(),
Oracle nvl())
• Ternary conditional
– Return one of two values based on a predicate
• Other functions….
– And we can easily add more as required

Sample data
Original
Event
Data
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326 “http://www.example.com/start.html" "Mozilla/4.08
[en] (Win98; I ;Nav)”
As JSON doc = {
_id: ObjectId('4f442120eb03305789000000'),
host: "127.0.0.1",
time: ISODate("2000-10-10T20:55:36Z"),
path: "/apache_pb.gif",
referer: “http://www.example.com/start.html",
user_agent: "Mozilla/4.08 [en] (Win98; I ;Nav)”
}
Insert to
MongoDB
db.logs.insert( doc )

Dynamic Queries
Find all
logs for
a URL
db.logs.find( { ‘path’ : ‘/index.html’ } )
Find all
logs for
a time
range
db.logs.find( { ‘time’ :
{ ‘$gte’ : new Date(2012,0),
‘$lt’ : new Date(2012,1) } } );
Find all
logs for
a host
over a
range of
dates
db.logs.find( {
‘host’ : ‘127.0.0.1’,
‘time’ : { ‘$gte’ : new Date(2012,0),
‘$lt’ : new Date(2012, 1) } } );

Aggregation Framework
Request
s per
day by
URL
db.logs.aggregate( [
{ '$match': {
'time': {
'$gte': new Date(2012,0),
'$lt': new Date(2012,1) } } },
{ '$project': {
'path': 1,
'date': {
'y': { '$year': '$time' },
'm': { '$month': '$time' },
'd': { '$dayOfMonth': '$time' } } } },
{ '$group': {
'_id': {
'p':'$path’,
'y': '$date.y',
'm': '$date.m',
'd': '$date.d' },
'hits': { '$sum': 1 } } },
])

Aggregation Framework
{
‘ok’: 1,
‘result’: [
{ '_id': {'p':’/index.html’,'y': 2012,'m': 1,'d': 1 },'hits’: 124 } },
{ '_id': {'p':’/index.html’,'y': 2012,'m': 1,'d': 2 },'hits’: 245} },
{ '_id': {'p':’/index.html’,'y': 2012,'m': 1,'d': 5 },'hits’: 94} }
]
}

Roll-ups with map-
reduce
Design Pattern

Map Reduce – Map Phase
Generat
e hourly
rollups
from log
data
var map = function() {
var key = {
p: this.path,
d: new Date(
this.ts.getFullYear(),
this.ts.getMonth(),
this.ts.getDate(),
this.ts.getHours(),
0, 0, 0) };
emit( key, { hits: 1 } );
}

Map Reduce – Reduce Phase
Generat
e hourly
rollups
from log
data
var reduce = function(key, values) {
var r = { hits: 0 };
values.forEach(function(v) {
r.hits += v.hits;
});
return r;
}
)

Map Reduce
Generat
e hourly
rollups
from log
data
cutoff = new Date(2012,0,1)
query = { 'ts': { '$gt': last_run, '$lt': cutoff } }
db.logs.mapReduce( map, reduce, {
‘query’: query,
‘out’: { ‘reduce’ : ‘stats.hourly’ } } )
last_run = cutoff

Map Reduce Output
> db.stats.hourly.find()
{ '_id': {'p':’/index.html’,’d’:ISODate(“2012-0-1 00:00:00”) },
’value': { ’hits’: 124 } },
’value': { ’hits’: 245} },
’value': { ’hits’: 322} },
’value': { ’hits’: 175} },
... More ...

Chained Map Reduce
Collection 1 :
Raw Logs
Collection 1 :
Raw Logs
Map
Reduce
Map
Reduce
Collection 2:
Hourly Stats
Collection 2:
Hourly Stats
Collection 3:
Daily Stats
Collection 3:
Daily Stats
Map
Reduce
Map
Reduce
Runs
every hour
Runs
every day

Pre-aggregated
documents
Design Pattern

Pre-Aggregation
Data for
URL /
Date
{
_id: "20101010/site-1/apache_pb.gif",
metadata: {
date: ISODate("2000-10-10T00:00:00Z"),
site: "site-1",
page: "/apache_pb.gif" },
daily: 5468426,
hourly: {
"0": 227850,
"1": 210231,
...
"23": 20457 },
minute: {
"0": 3612,
"1": 3241,
...
"1439": 2819 }
}

Pre-Aggregation
Data for
URL /
Date
id_daily = dt_utc.strftime('%Y%m%d/') + site + page
hour = dt_utc.hour
minute = dt_utc.minute
# Get a datetime that only includes date info
d = datetime.combine(dt_utc.date(), time.min)
query = {
'_id': id_daily,
'metadata': { 'date': d, 'site': site, 'page': page } }
update = { '$inc': {
‘daily’ : 1,
'hourly.%d' % (hour,): 1,
'minute.%d.%d' % (hour,minute): 1 } }
db.stats.daily.update(query, update, upsert=True)

Pre-Aggregation
Data for
URL /
Date
db.stats.daily.findOne(
{'metadata': {'date':dt,
'site':'site-1',
'page':'/index.html'}},
{ 'minute': 1 }
);

Schema Design by Chad Tindel, Solution Architect, 10gen

Recommended

More Related Content

What's hot (20)

Similar to Schema Design by Chad Tindel, Solution Architect, 10gen (20)

More from MongoDB (20)

Recently uploaded (20)

Schema Design by Chad Tindel, Solution Architect, 10gen