8th TUC Meeting - David Meibusch, Nathan Hawes (Oracle Labs Australia). Frappé: Querying and managing evolving code dependency graphs.

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
David Meibusch & Nathan Hawes
Oracle Labs Australia
June 2016
Frappé: a code comprehension tool for large C/C++ codebases
Querying and managing
evolving code dependency graphs

Safe Harbour
The following is intended to provide some insight into a line of research in Oracle
Labs. It is intended for information purposes only, and may not be incorporated
into any contract.
It is not a commitment to deliver any material, code, or functionality, and should
not be relied upon in making purchasing decisions. Oracle reserves the right to
alter its development plans and practices at any time, and the development,
release, and timing of any features or functionality described in connection with
any Oracle product or service remains at the sole discretion of Oracle.
Any views expressed in this presentation are my own and do not necessarily reflect
the views of Oracle.
Frappé: querying the Linux kernel dependency graph 2

The Truth is in the Source!
But the source is often complicated, multi-language and really really big
Frappé: a code comprehension tool for large C/C++ codebases 3

Frappé: code as a property graph
• Graph natural in this domain
– Call graph, directory hierarchy, type hierarchy, data/control flow graphs
• Overlay data from different spaces
– File system, build, preprocessor, AST, cross-language
• Lets users specify queries in terms of this graph

How it works
EXTRACTOR IMPORTER
EDITOR PLUGINS
CLI SCRIPTS
Source Code .fo Server User
WEB UI

Graph model example
prog
module
main.c
file
foo.o
module
foo.c
source_file
foo.h
source_file
bar
function_decl
bar
function
main
function
argc
parameter
argv
parameter
input
parameter
int
primitive
char
primitive
compiled_from
compiled_from
includes
linked_from
includes
file_contains
file_contains
calls
has_param
has_param
is_type is_type
compiled_from
file_contains
has_param
is_type
declares
int bar(int); foo.h
#include “foo.h”
int bar(int *input) {
return *input * 2;
}
foo.c
int main(int argc, char **argv) {
return bar(&argc);
}
main.c
gcc foo.c –c –o foo.o
gcc main.c foo.o –o prog
build

How it works
EXTRACTOR IMPORTER
EDITOR PLUGINS
CLI SCRIPTS
Source Code IR Server User
WEB UI

Use cases
• Code search
• Code navigation
– Go to definition
– Find references
• Code comprehension
– Visualization
– Transitive closure calls, includes, etc.
– Shortest path queries

How it looks
• dddd • Web UI Xref • Code Map with impact
overlay

Use cases
• Code search
• Code navigation
– Find references
– Visualization
– Reachability queries
Queries and Neo4j performance
detailed in GRADES’15 paper:
”Frappé: Querying the Linux Kernel
dependency graph”

Code search
foo.c

Code search
foo.c
(n with name=‘foo.c’)

Code search
foo.c
struct foo {
int c;
}

Code search
foo.c
(c with name=‘foo’)
-[:contains]->
(n with name=‘c’)
UNION
struct foo {
int c;
}

Code search
foo.c
(c with name=‘foo’)
-[:contains]->
UNION
struct bar {
int c;
}
typedef struct bar foo;

Code search
foo.c
(c with name ‘foo’)
-[:aliases*]->()-[:contains]->
UNION
struct bar {
int c;
}

Use cases
• Code search
• Code navigation
– Find references
– Visualization
dependency graph”

Use cases
• Code search
• Code navigation
– Find references
– Visualization
• Custom user queries
– (Anti) pattern detection
– Expose query language
dependency graph”

Dependency cycle
source_file
file_contains
source_file
file_contains
source_file
file_contains file_contains file_contains file_contains
references references
references

BUT
Developers working off of different
versions of the code

Target Scenario
• 1000s of developers working from
main branch
• Changes merged regularly
• Most developers working off of
versions from the past 30 days

Current Deployment
• Separate Neo4j instance for the most
recent 5 versions of the code
• New graph generated in nightly
regression
• Script on the client to determine which
server to connect to
• Deployment effort and complexity
• Inefficient use of resources
– Redundant data
– Memory requirements

Current Deployment
• Separate Neo4j instance for the most
recent 5 versions of the code
• New graph generated in nightly
regression
• Script on the client to determine which
server to connect to
• Deployment effort and complexity
• Inefficient use of resources
– Redundant data
– Memory requirements
Single logical server
with multiple versions
and efficient storage

Use cases for multiple versions
• Single version use cases per version
• Code review: 2 versions
– Are there any architectural constraints being newly violated?
– Are there any new usages of deprecated methods?
– Are any methods now unused?
Select version

Version selection
SELECT path
FROM freebsd@1013
WHERE path =
(:function WITH name='source') -/:calls*/-> (:function WITH name='sink')
PATH vcalls := () -[:call WITH 1013 between fromV and toV]-> ()
SELECT path
FROM freebsd
WHERE path =
(:function WITH name='source’, 1013 between fromV and toV)
-/:vcalls*/->
(:function WITH name='sink’, 1013 between fromV and toV)
VS

Use cases for multiple versions
• Single version use cases per version
• Code review: 2 versions
– Are there any architectural constraints being newly violated?
– Are there any new usages of deprecated methods?
– Are any methods now unused?
Match in each version
+
Compare results

Compare results
SELECT DIFFERENCE path
FROM freebsd@1013, freebsd@1014
WHERE path =
SELECT path
FROM freebsd@1013
WHERE path =
DIFFERENCE
SELECT path
FROM freebsd@1014
WHERE path =
VS

Open Questions
• Node/Edge identity
– Supplied or derived using graph information
• Query language expressiveness
– Regular path expressions
– Multiple edge labels
• Efficient storage and querying for multiple graph versions:
– List or ideally DAG of versions
– Union, intersection, difference of results from different versions
– Query language that abstracts away versioning

FreeBSD dataset available on OTN
• Includes graphs of 10.1, 10.2, 10.3
kernel + documentation
• Each graph
– Extracted from 10M LOC
– 2 million vertices
– 10 million edges
• Try it out and get in touch:
– nathan.hawes@oracle.com
david meibusch@oracle.com
ben.barham@oracle.com
http://www.oracle.com/technetwork/oracle-
labs/datasets/downloads/index.html

8th TUC Meeting - David Meibusch, Nathan Hawes (Oracle Labs Australia). Frappé: Querying and managing evolving code dependency graphs.

Graph model example
main.c
file
bar
function
main
function
name_start_line
name_start_column
use_start_line
use_end_line
name_start_line
name_start_column
use_start_line
use_end_line
return *input * 2;
}
foo.c
return bar(&argc);
}
main.c
file_containscalls
name location
use location
name location
use location

Graph model example
main.c
file
bar
function
main
function
name_start_line
name_start_column
use_start_line
use_end_line
name_start_line
name_start_column
use_start_line
use_end_line
return BAR(&argc);
}
main.c
file_containscalls
#define BAR(A) bar(A)
int bar(int);
foo.h
return *input * 2;
}
foo.c

Graph model example
main.c
file
bar
function
main
function
name_file_id
name_start_line
name_start_column
use_file_id
use_start_line
use_end_line
name_file_id
name_start_line
name_start_column
use_file_id
use_start_line
use_end_line
return BAR(&argc);
}
main.c
file_containscalls
#define BAR(A) bar(A)
int bar(int);
foo.h
return *input * 2;
}
foo.c
name location
use location

8th TUC Meeting - David Meibusch, Nathan Hawes (Oracle Labs Australia). Frappé: Querying and managing evolving code dependency graphs.

More Related Content

Similar to 8th TUC Meeting - David Meibusch, Nathan Hawes (Oracle Labs Australia). Frappé: Querying and managing evolving code dependency graphs. (20)

More from LDBC council (20)

Recently uploaded (20)

8th TUC Meeting - David Meibusch, Nathan Hawes (Oracle Labs Australia). Frappé: Querying and managing evolving code dependency graphs.