EclipseCon France 2017 - Xtending Our Vhdl Xtext Formatter With The Formatter2 API

redefine.digital.design: Helping you deal with complexity in VHDL and Verilog.
Xtending our VHDL Xtext
formatter with the formatter2 API
ir. Titouan Vervack
EclipseCon France
2017-06-21

redefine.digital.design: Helping you deal with complexity in VHDL and Verilog.
Xtending our VHDL Xtext
formatter with the formatter2 API
ir. Titouan Vervack
EclipseCon France, Toulouse
2017-06-21

What do we do?
VHDL & SV IDE
3

What do we do?
Advanced analysis and linting
4

What do we do?
Visualisation
5

What do we do?
Visualisation
6

What do we do?
And cookies...
8

What do we do?
And occasionally… formatting
9

Our formatter problems
● Big grammar
○ ~1.2k LoC
○ ~200 rules
10

● Big grammar
○ ~1.2k LoC
○ ~200 rules
● Hard to fix bugs
10

● Big grammar
○ ~1.2k LoC
○ ~200 rules
● Hard to fix bugs
● Near impossible to add new features
10

Formatter 1.0 vs Formatter 2.0
11
Formatter 1.0 Formatter 2.0
● No access to
○ Node model
○ AST
○ Text
● => Specify everything purely on the grammar
● => Lots of hacks
● Access to
○ Node model
○ AST
○ Grammar
● Can’t take context into account
○ Can’t adjust to the user
● Context aware formatting
○ Conditional formatting
○ Table formatting
● Not customizable ● Very customizable

Formatter 2.0 region example
variable _ foo _ : _ bit _ ; _-- barn
variable foo : bit; -- bar
= ISemanticRegion
= IHiddenRegion
_ -- bar n
= IHiddenRegionPart
12
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api

Formatting 2.0 usage example
13

class AwesomO4000 extends AbstractFormatter2 {
}
13

def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) {
}
}
13

interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent]
}
}
13

fd.declarations.forEach[format]
}
}
13

fd.regionFor.keyword(“;”)
}
}
13

.prepend[noSpace highPriority]
}
}
13

.prepend[noSpace highPriority]
.append[setNewlines(1, 1, 2)]
}
}
13

Time for a plan
1. Make tests compatible
14

Time for a plan
2. Start out easy: only indentation
14

Time for a plan
3. Fix tests one by one
14

Time for a plan
4. Fix old bugs
14

Time for a plan
4. Fix old bugs
5. Do not introduce new bugs
14

Time for a plan
4. Fix old bugs
6. Feature parity & extend with new features
14

Time for a plan
4. Fix old bugs
6. Feature parity & extend with new features
7. Performance tweaking to same or better level than old
14

Actual timeline
formatter.format(node, node.offset, node.length).formattedText
15

Actual timeline
formatter.format(node, node.offset, node.length).formattedText
val request = new FormatterRequest()
val parsed = new XtextResource()
parsed.contents.add(parseResult.rootASTElement)
parsed.setParseResult(parseResult)
val access = builder.forNodeModel(resource).create()
request.setTextRegionAccess(access)
val replacements = formatter.format(request)
access.rewriter.renderToString(replacements)
15

interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent]
Actual timeline
16

Actual timeline
● Indentation requires newline definitions
16

Actual timeline
● Indentation requires newline definitions
● => replace all newlines with… the same amount of newlines
16

Actual timeline
4. Fix old bugs
17

Actual timeline
18

Actual timeline
6. Feature parity & extend formatter with new features
19

Actual timeline
● Parity:
19

Actual timeline
● Parity:
○ Keyword casing
19

Actual timeline
● Parity:
○ Keyword casing
○ Comment alignment to columns
19

Actual timeline
● Parity:
○ Keyword casing
○ Vertical alignment on keywords “:”, “:=”,...
19

Actual timeline
● Parity:
○ Keyword casing
○ Preserve newline
19

Actual timeline
● Parity:
○ Keyword casing
● New features:
19

Actual timeline
● Parity:
○ Keyword casing
● New features:
○ Correct indentation
19

Actual timeline
● Parity:
○ Keyword casing
● New features:
○ Formatter tags
19

Actual timeline
● Parity:
○ Keyword casing
● New features:
○ Formatter tags
Easy once we have alignment
19

Alignment?
● Replacements are added in random order, applied in correct order
21

Alignment?
● Alignment needs to be done on post-formatted input
21

Alignment?
● Alignment needs to be done on post-formatted input
● How do I know which region to format?
21

Alignment!
Find smallest encapsulating object that
formats region between start and end
start end
22

Alignment!
clk : in std_logic := ‘X’;
23

Alignment!
start end
23

PortDeclaration
Alignment!
start end
name direction type value
end.getSemanticElement()
PortDeclaration
PortList
23

PortDeclaration
Alignment!
start end
name direction type value
offset x
x + 4
end.getSemanticElement()
PortDeclaration
PortList
23

val rule = portDecl.getParserRule()
val oldAccess = createTextRegionAccess(rule, portDecl.text)
val formatted = portDecl.format()
Alignment!
Format smallest encapsulating object
24

val rule = portDecl.getParserRule()
val oldAccess = createTextRegionAccess(rule, portDecl.text)
val formatted = portDecl.format()
Alignment!
val newAccess = createTextRegionAccess(rule, formatted.text)
val oldIndices = oldAccess.getIndices(start, end)
val length = newAccess.countBetween(oldIndices)
Format smallest encapsulating object
Map unformatted string to formatted string
24

Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
25

→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
25

→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
→return foo((5,
→············6),
→···········3));
25

→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
→return foo((5,
→············6),
→···········3));
→return foo(
→→(5,
→→·6),
→→3));
25

→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
→return foo((5,
→············6),
→···········3));
→return foo(
→→(5,
→→·6),
→→3));
Find largest encapsulating element on same line
25

Actual timeline
Correct indentation
26

Actual timeline
Correct indentation
replacements
.filter[ r|
]
26

Actual timeline
Correct indentation
replacements
.filter[ r|
r.lineCount > 1
]
26

Actual timeline
Correct indentation
replacements
.filter[ r|
r.lineCount > 1 &&
r.replacementText.newlines + 1 == r.lineCount
]
26

Actual timeline
Formatter tags
27

Actual timeline
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
27

Actual timeline
Formatter tags
val originalRegions = request.getRegions() ?: document.region
27

Actual timeline
Formatter tags
val betweenRegions = document.findRegionsBetween(off, on)
27

Actual timeline
Formatter tags
.addAll(document.findRegionsBetween(off, EOF))
27

Actual timeline
Formatter tags
val newRegions = originalRegions.removeRegions(betweenRegions)
27

Actual timeline
Formatter tags
val newRegions = originalRegions.removeRegions(betweenRegions)
request.setRegions(newRegions)
27

Performance
28

Performance
● Cache ITextRegionAccess in subformats
28

Performance
● Cache IParser.createAllRules(Grammar)
28

Performance
● Cache IParser.createAllRules(Grammar)
● Merge all replacements into one big replacement
28

Performance
emacs/xtend > 25m
29

Performance
emacs/xtend > 25m
30

Biggest problems in hindsight
31

● Tests pass, close enough?
31

○ Nope, old testset isn’t nearly enough
31

● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
31

● Alignment interactions with normal and first param alignments and
comments
31

● Alignment interactions with normal and first param alignments and
comments
3031

Warning to Xtext users
Default ExceptionHandler logs DSL source code to stderr
32

Sigasi future work
● Smooth out remaining bugs
33

Sigasi future work
● Some more context awareness
33

Sigasi future work
● Autowrap
33

Sigasi future work
● Autowrap
● Performance
33

Sigasi future work
● Autowrap
● Performance
● Upstream features
33

Sigasi future work
● Autowrap
● Performance
○ Formatter tags
33

Sigasi future work
● Autowrap
● Performance
○ Formatter tags
○ Performance optimizations
33

Xtext wish list
● Accept my PRs (tags, correct indent, performance)
34

Xtext wish list
● Fix/discuss my issues
34

Xtext wish list
● Fix Xtend formatter issues
34

Xtext wish list
● Very slow Xtend formatter source file
34

Xtext wish list
● Very slow Xtend formatter source file
● Add pre- and post format hooks in format(FormatterRequest)
34

Conclusion
● Context aware
35

Conclusion
● Context aware
● Debugable
35

Conclusion
● Context aware
● Debugable
● Good performance
35

Conclusion
● Context aware
● Debugable
● Very customizable
35

Conclusion
● Context aware
● Debugable
● User friendly
35

Conclusion
● Context aware
● Debugable
● User friendly
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
35

Conclusion
● Context aware
● Debugable
● User friendly
● Happy with the switch
35

Conclusion
● Context aware
● Debugable
● User friendly
○ New features
35

Conclusion
● Context aware
● Debugable
● User friendly
○ New features
○ Very readable
35

Conclusion
● Context aware
● Debugable
● User friendly
○ New features
○ Very readable
○ Conditional formatting
35

Evaluate the Sessions
-1 0 +1
Sign in and vote at eclipsecon.org

EclipseCon France 2017 - Xtending Our Vhdl Xtext Formatter With The Formatter2 API

More Related Content

What's hot (10)

Similar to EclipseCon France 2017 - Xtending Our Vhdl Xtext Formatter With The Formatter2 API (20)

Recently uploaded (20)

EclipseCon France 2017 - Xtending Our Vhdl Xtext Formatter With The Formatter2 API