JSLT: JSON querying and transformation

JSLT: JSON query & transform
Lars Marius Garshol, lars.marius.garshol@schibsted.com
http://twitter.com/larsga 2018–09–12, JavaZone 2018

Data Platform
2
Data Platform
Batch
Streaming
Pulse

Routing
• We send data to ~210 different destinations
• Filters on the data specify which data should go
where
• often very detailed conditions on many ﬁelds
• Full routing tree has ~600 ﬁlter/transform/sink
nodes
4

Transforms
• Because GDPR we need to anonymize most incoming data
formats
• Some data has data quality issues that cannot be ﬁxed at
source, requires transforms to solve
• In many cases data needs to be transformed from one format to
another
• Pulse to Amplitude
• Pulse to Adobe Analytics
• ClickMeter to Pulse
• Convert data to match database structures
• …
5

Who configures?
• Schibsted has >100 business units
• for Data Platform to do detailed configuration for all of
these isn’t going to scale
• for sites to do it themselves saves lots of time
• Configuration requires domain knowledge
• each site has its own specialities in Pulse tracking
• to transform and route these correctly requires knowing all
this
6

Batch config: 1 sink
{
"driver": "anyoffilter",
"name": "image-classification",
"rules": [
{ "name": "ImageClassification", "key": "provider.component", "value": "ImageClassification" },
{ "name": "ImageSimilarity", "key": "provider.component", "value": "ImageSimilarity" }
],
"onmatch": [
{
"driver": "cache",
"level": "memory+disk"
},
{
"driver": "demuxer",
"rules": "${pulseSdrnFilterUri}",
"parallel": true,
"onmatch": {
"driver": "textfilewriter",
"uri": "${imageSiteUri}",
"numFiles": {
"eventsPerFile": 500000,
"max": ${numExecutors}
}
}
}
],
"consume": true
7
Early config was 1838 lines
} JSON matching

What if?
• We had an expression language for JSON
• something like, say, XPath for JSON
• could write routing ﬁlters using that
• We had a tranformation language for JSON
• write as JSON template, using expression language to
compute values to insert
• A custom routing language for both batch and
streaming, based on this language
• designed for easy expressivity & deploy
9

• Already existing query language for JSON
• https://stedolan.github.io/jq/
• Originally implemented in C
• there is a Java implementation, too
• Can do things like
• .foo
• .foo.bar
• .foo.bar > 25
• …
10

First, fumbling attempt
{
“event_type” : “View”,
“insert_id” : {“__expr__” : “.object.id”},
“source” : {“__if__” : {
“test” : “.source”,
“then” : “.source.id”,
“else” : “.src”
}
}
11

Second, fumbling attempt
{
“insert_id” : “${.object.id}”,
“source” : “${if (.source) .source.id else .src}”
}
12

Third attempt
{
“insert_id” : ${ .object.id },
“source” : if ${ .source } ${ .source.id } else ${ .src }
}
13

Proof-of-concept
• Implement real-world transforms in this language
• before it was implemented
• Helped improve and solidify the design
• Veriﬁed that the language could do what we
needed
• Transforms looked quite reasonable.
14

A simple language
• JSON is written in JSON syntax
• evaluates to itself
• if <expr> <expr> else <expr>
• [for <expr> <expr>]
• let <name> = <expr>
• ${ … jq … }
15

Matchers
{
* : ${ . }
}
16
{
* - “object” : ${ . }
}

Stunt prototype
• Most of it implemented in two days
• Implemented in Scala
• using Antlr 3 to generate the parser
• jackson-jq for jq
• jackson for JSON
• A simple object tree interpreter
• Constructor.construct(Context, JsonNode) => JsonNode
17

Object tree{
“source” : if ${ .source } ${ .source.id } else ${ .src }
}
18
ObjectConstructor
PairConstructor(LiteralConstructor)
PairConstructor(JqConstructor)
PairConstructor(IfConstructor(Jq, Jq, Jq))

Object expression
20
Create object
Construct value
Add to object

If
21
Evaluate
condition
Construct then
Construct else

The parser
• Code that takes a character stream and builds the
expression tree
• Use a parser generator to handle the difﬁcult part
• requires writing a grammar
• Parser generator produces Abstract Syntax Tree
• basically corresponds to the grammar structure
22

Antlr Grammar
grammar Jstl;
WS : [ trn]+ -> skip ; // ignore whitespace
COMMENT : '//' ~[rn]* [rn] -> skip ; // ignore comments
STRING : '"' ((~["]) | ('"'))+ '"' ;
INT: '-'? [0-9]+ ;
FLOAT: '-'? [0-9]+ '.' [0-9]+ ;
NULL: 'null';
BOOL: 'true' | 'false';
IDENT: [A-Za-z] [_A-Za-z]* ;
JQ : '$' '{' (~[}"] | '"' (~["] | '' .)* '"')+ '}' ;
jstl : let* expr EOF ;
expr : object | array | STRING | INT | FLOAT | NULL | BOOL |
JQ | ifvalue | forexpr | parenthesis;
23

Language in use
• Implemented Data Quality Tooling using jq
• ﬁlters done in jq
• Implemented routing using jq ﬁlters
• and transforms in JSLT
• Wrote some transforms using the language
• anonymization of tracking data
• cleanup transforms to handle bad data
• …
25

The good
• The language works
• proven by DQT, routing, and transforms
• Minimal implementation effort required
• Users approved of the language
• general agreement it was a major improvement
• people started writing their own transforms
26

The bad
• Performance could be better
• not horrible, but not great, either
• The ${ … } wrappers are really ugly
• jq
• does not handle missing data well
• has dangerous features
• has weird and difﬁcult syntax for some things
• Too many dependencies
• Scala runtime (with versioning issues)
• Antlr runtime
27

2.0
• Implement the complete language ourselves
• goodbye ${ … }
• Get rid of the jq strangeness
• Add some new functionality
• Implement in pure Java with JavaCC
• JavaCC has no runtime dependencies
• only dependency is Jackson
28

JSLT expressions
29
.foo Get “foo” key from input object
.foo.bar Get “foo”, then “.bar” on that
.foo == 231 Comparison
.foo and .bar < 12 Boolean operator
$baz.foo Variable reference
test(.foo, “^[a-z0-9]+$”) Functions (& regexps)

Sinks
VG-FrontExperimentsEngagement-1:
eventType: PulseAnonymized
ﬁlter: get-client(.) == "vg" and ."@type" == "Engagement" and
contains(.object."@type", ["Article", "SalesPoster"]) and
(contains("df-86-", .origin.terms) or
contains("df-86-", .object."spt:custom".terms))
transform: transforms/vg-article-views.jslt
kinesis:
arn: arn:aws:kinesis:eu-west-1:070941167498:stream/
vg_front_experiments_engagement
role: arn:aws:iam::070941167498:role/data-platform-kinesis-write
32

Expressions
• + - / *
• and or
• > < <= != == >=
• literals
• variable references
• function calls (+ function library)
33

Dealing with missing data
• 2 + null => null
• size(null) => null
• number(“12”) => 12
• number(null) => null
34

Operators
35
Evaluate left & right
Do the operation

Numeric operators
36
Null handling
Convert to numbers
or error
Decide if int or ﬂoat
The actual operations

Object for expressions
38
{
“insert_id” : .object.id,
“source” : if (.source) .source.id else .src
} + {
for (.custom)
“custom_” + .key : .value
}
{
“object” : {“id” : 21323, … },
“src” : “App123”,
“custom” : {
“time” : 4592593492,
“event”: 3433
}
}
{
“insert_id” : 21323,
“source” : “App123”,
“custom_time”: 4592593492,
“custom_event” : 3433
}

Function declarations
def <name>(<param1>, <param2>)
<let>*
<expr>
Very easy to implement
Very useful
But means going Turing-complete…
39

Turing-complete?
• Means that the language can express any
computation
• It’s known that all that’s required is
• conditionals (we have if tests)
• recursion (our functions can call themselves)
• But can this really be true?
42

N-queens
• Write a function that takes the size of the
chessboard and returns it with queens
• queens(4) =>
[
[ 0, 1, 0, 0 ],
[ 0, 0, 0, 1 ],
[ 1, 0, 0, 0 ],
[ 0, 0, 1, 0 ]
]
43https://github.com/schibsted/jslt/blob/master/examples/queens.jslt

Danger?
• It’s possible to implement operations that run
forever
• But in practice the stack quickly gets too deep
• The JVM will then terminate the transform
45

Performance
• 5-10 times faster than 1.0
• The main difference: no more jackson-jq
• jackson-jq is not very efﬁcient
• internal model is List<JsonNode>
• creates many unnecessary objects during evaluation
• does work at run-time that should be done at compile-time
46

JSLT improvements
• Value model is JsonNode
• can usually just return data from input object or from code
• Efﬁcient internal structures
• all collections are arrays
• very fast to traverse
• Boolean short-circuiting
• once we know the result, stop evaluating
• Cache regular expressions to avoid recompiling
47

The optimizer
• An optimizer is a function that takes an expression
and outputs an expression such that
• the new expression is at least as fast, and
• always outputs the same value
• Improves performance quite substantially even with
very simple techniques
48

Constant folding
contains(get-client(.), [“vg”, “aftenposten”, “bt”])
49
FunctionExpression
FunctionExpression
DotExpression
ArrayExpression
LiteralExpression
LiteralExpression
LiteralExpression
FunctionExpression
FunctionExpression
DotExpression
LiteralExpression

Performance
• Test case: pulse-cleanup.jslt, real data, my laptop
• a complicated transform: 165 lines
• Transforms 132,000 events/second in one thread
• 1.0 did about 20,000 events/second
51

Three strategies
• Syntax tree interpreter
• known to be the slowest approach
• Bytecode compiler with virtual machine
• C version of jq does this
• Java does that (until the JIT kicks in)
• Python does this
• Native compilation
• what JIT compiler in Java does
52

Designing a VM
53
Opcode Param
DUP
MKOBJ
CALL <func>
int[] bytecode;
JsonNode[] stack;
int top;
switch (opcode) {
case DUP:
stack[++top] = stack[top-1];
break;
case MKOBJ:
stack[++top] = mapper.createObj…
break;
// …
}

Compiler
• Traverse down the object tree
• emit bytecode as you go
• Stack nature of the VM matches object tree
structure
• each Expression produces code that leaves the value of
that expression on the stack
• Example:
• MKARR, <ﬁrst value>, ARRADD, <second>, ARRADD, …
54

Prototype
• Stunt implemented over a couple of days
• Depressing result: object tree interpreter ~40%
faster
• Anthony Sparks tried the same thing
• original VM implementation 5x slower
• eventually managed to achieve performance parity
• So far: performance does not justify complexity
55

Java bytecode?
• The JVM is actually a stack-based VM
• can simply compile to Java bytecode instead
• Tricky to learn tools for generating bytecode
• no examples, very little documentation
• In the end decided to use the Asm library
• not very nice to use
• very primitive API
• crashes with NullPointerException on bad bytecode
56

Results
• Hard work to build
• many surprising issues in Java bytecode
• Performance boost of 15-25%
• code lives on jvm-bytecode branch in Github
• Ideas for how it could be even faster…
• through type inference
59

Type inference beneﬁts
"sdrn:" + $namespace + ":" + $rType + ":" + $rId
Plus
Plus(“sdrn” $namespace)
Plus(
Plus(“:” $rType)
Plus(“:” $rId)
)
60
Plus:
JsonNode -> String
JsonNode -> String
String + String
new String -> new JsonNode
Will make 4 unnecessary TextNode objects
Will wrap and unwrap String repeatedly
Will check types unnecessarily

Solution
• + operator can ask both sides: what type will you
produce?
• If one side says “string” then the result will be a string
• When compiling, do compile(generator, String)
• will compile code that produces a Java String object
• + operator will make a new String if that’s what’s
wanted
• or turn it into a TextNode if the context wants Any
61

Freedom from Jackson
• The current codebase is bound to Jackson
• JVM bytecode compilation might be a way to
escape that
• Could build compilers that can interface with
different JSON representations
• Have ideas for a more efﬁcient JSON representation
• basically encode everything as arrays of ints
• should save memory, GC, and produce faster code
62

Freedom from JSON
• If we aren’t bound to Jackson, why should we be
bound to JSON?
• Could support Avro, too
• Perhaps also other formats
63

Internal status
• JSLT now used in
• Data Quality Tooling (to express tests on data)
• routing ﬁlters
• transforms
• In Schibsted we have
• 52 transforms, 2370 lines of code
• written by many people in different parts of the company
• Data Platform runs ~11 billion transforms/day
65

Open source status
• Released in June
• People are using it for real
• one certain case, several more examples
• details unknown
• Useful contributions from outsiders
• several bug ﬁxes to datetime/number handling
• Two alternative implementations being worked on
• one in .NET
• one is virtual machine-based in Java
66

Lessons learned
• A custom language can make life much simpler
• if it ﬁts the use case well
• Implementing a language is easier than it seems
• basically doable in a week
• Designing a language is not easy
• unfortunately
67

https://github.com/schibsted/jslt
https://www.slideshare.net/larsga
Slides at

JSLT: JSON querying and transformation

Recommended

More Related Content

What's hot (20)

Similar to JSLT: JSON querying and transformation (20)

More from Lars Marius Garshol (20)

Recently uploaded (20)

JSLT: JSON querying and transformation