Building DSLs with the Spoofax Language Workbench
3 likes1,226 views
This document discusses the Spoofax language workbench, which is a tool for building domain-specific languages (DSLs). It addresses key concerns in developing DSLs such as the tradeoffs between internal and external DSLs, textual vs graphical syntax, code generation vs interpretation, and static vs dynamic typing. The Spoofax workbench uses declarative syntax definitions and rewrite rules to provide features like syntax checking, error recovery, and incremental processing for integrated development environments.
![Full Text Search
define page search() {
var query : String
action show() {
var pubs : List<Publication> := searchPublication(query);
if(pubs.length > 0) {
replace(results, results(query,pubs));
} else {
replace(results, noresults()); entity Publication {
title :: String (name,searchable)
}
journal :: String (searchable)
} year :: Int (searchable)
main{ month :: String (searchable)
section{ ...
}
header{"Search"}
form{ input(query)[onkeyup:=show()] }
placeholder results{}
}
}
}](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-14-320.jpg)
![Access Control Rules
define showReview(review : Review) {
action delete() { ... }
section{ ...
navigate(editReview(review)){"[Edit]"}
submitlink delete() {"[X]"}
}
}](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-21-320.jpg)
![Access Control Rules
define showReview(review : Review) {
action delete() { ... }
section{ ...
navigate(editReview(review)){"[Edit]"}
submitlink delete() {"[X]"}
}
}
access control rules
principal is User with credentials username, password
predicate isReviewer(review : Review) {
review.reviewer == securityContext.principal
}
rule template showReview(review : Review, pubLink : Bool) {
review.public || isReviewer(review)
rule action delete() {
loggedIn() && isReviewer(review)
}
}
rule page editReview(review : Review) {
loggedIn() && isReviewer(review)
}](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-22-320.jpg)
![Data Validation: Form Input Validation
label("Acronym (without year)"){
input(acronym){
validate(!isEmptyString(acronym),
"Provide acronym")
validate(!(/[ ]/.find(acronym)),
"Acronym should not contain spaces")
validate(!(/[0-9][0-9][0-9][0-9]/.find(acronym)),
"Acronym should not contain year")
}
}](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-24-320.jpg)
![Terms to Represent Abstract Syntax
Entity(“Blog”,
[Property(“name”,SimpleType(“String”)),
Property(“author”, SimpleType(“User”))]
)](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-52-320.jpg)
![Embedding HQL
module expressions
imports Common
imports HQL
exports
context-free syntax
SelectStatement -> Exp
define page index() {
list{
for(p : Post in select p from Post as p order by (p.title)) {
listitem{ output(p) }
}
}
}
ForElem("p", SimpleType("Post")
, QueryRule(
SelectFrom(
Some(Select(None(), [AliasedExpression(Path(["p"]), None())]))
, FromClause([FromRangeJoin(FromClass(Path(["Post"]), Some(AsAlias(Alias("p"
)
, Some(OrderByClause([OrderElement(Paren([Path(["p", "title"])]), None())]))
)
, [CallElems("listitem", [CallArgs("output", [Var("p")])])]
)](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-65-320.jpg)
![Syntax Properties
module nwl-Syntax.generated
language Syntax properties (static defaults)
// Comment constructs:
line comment : "//"
block comment : "/*" * "*/"
// Fences (used for matching,
// inserting, indenting brackets):
fences : [ ]
( )
{ }
// Automatic indent hints
// (indent after these tokens):
indent after : "="
":"
// Regular expression for identifiers:
identifier lexical : "[A-Za-z0-9_]+"](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-74-320.jpg)
![Term Rewrite Rule
left-hand side pattern
label/name
desugar :
Property(x, t) -> Property(x, t, [])
variable
right-hand side pattern](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-81-320.jpg)
![Rewriting with Concrete Object Syntax
elem-to-java-servlet :
tc@elem|[ x(e)[passign*]{elem*} ]| ->
<wrap-input-render-java-code>
bstm* |[
String x_temp = ident+"~inputident"+uniqueid;
~*<render-error-messages-with-error-template(|java:expr|[ x_temp ]|,
<ErrorTemplateInput>)> bstm*|[
bstm_call*
]|
]|
where <is-input-template> tc
with inputident := <get-inputnumber> tc
; x_temp := <newname> "temp"
; bstm_call* := <control-flow-tcall-helper(|"render",expr|[ x_temp ]|)> tc
Stratego + embedded Java](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-87-320.jpg)
![Error Checking: Binary Operators
check :
e@BinOp(e1, op, e2) ->
(e, $[operator [op] not defined for [<pp>t1] and [<pp>t2]])
where t1 := <type-of> e1
where t2 := <type-of> e2
where require(<type-of> e)](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-97-320.jpg)
![Pretty-Printing with String Interpolation
pp : Entity(x, prop*) ->
$[entity [x] {
[<map(pp)>prop*]
}]
pp : Property(x,t) ->
$[[x] : [<pp>t]
]
pp : SimpleType(x) -> x
pp : SetType(t) -> $[Set<[<pp> t]>]
pp : [] -> $[]
pp : [t] -> <pp>t
pp : [t1,t2|ts] -> $[[<pp>t1],[<pp>[t2|ts]]]](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-98-320.jpg)
![Type Analysis: Binary Operators
type-of :
BinOp(e1, op, e2) -> t
where t := <function-type>(op, [<type-of>e1, <type-of>e2])
function-type :
("+", [SimpleType("String"), SimpleType("String")]) -> SimpleType("String")
function-type :
("+", [SimpleType("Int"), SimpleType("Int")]) -> SimpleType("Int")
function-type :
("-", [SimpleType("Int"), SimpleType("Int")]) -> SimpleType("Int")](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-101-320.jpg)
![Variables: Check
check :
e@Var(x) -> (e, $[Variable '[x]' not declared])
where require(<type-of>e)](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-107-320.jpg)
![Rewrite Rules are Context-free
desugar :
Property(x, t) -> Property(x, t, [])](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-124-320.jpg)
![Rewrite in Context
Inline :
[def@TemplateDef(f,[],elem1*) | def1*] -> [def | def2*]
where def2* := <alltd((Call(f) -> Call("container", elem1*)))> def1*
local traversal bound in context
container needed to replace single call with multiple elements](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-126-320.jpg)
![Rewrite in Context
Inline :
[def@TemplateDef(f,[],elem1*) | def1*] -> [def | def2*]
where def2* := <alltd((Call(f) -> Call("container", elem1*)))> def1*
bound in context](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-133-320.jpg)
![Inline :
[def@TemplateDef(f,[],elem*) | def1*] -> [def | def2*]
where def2* := <alltd((Call(f) -> Call("container", elem*)))> def1*
f elem*
[ TemplateDef(
"footer"
, []
, [Navigate(PageRef("webdsl", []), [String("WebDSL")])]
)
, TemplateDef(
"blog"
, [Param("b", SimpleType("Blog"))]
, [Call("footer")]
)
]
Call(f)
def1*](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-134-320.jpg)
![Rules with Parameters
Transform all elements of a list
map(s) : [] -> []
map(s) : [x|xs] -> [<s>x | <map(s)> xs]
Invert order of elements of a list
inverse(|ys) : [] -> ys
inverse(|ys) : [x|xs] -> <inverse(|[x|ys])> xs
Pair elements of two lists
zip(s) : ([],[]) -> []
zip(s) : ([x|xs],[y|ys]) -> [<s>(x,y) | <zip(s)>(xs,ys)]](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-136-320.jpg)
![Rewrite in Context: Local Traversal
Inline :
[def@TemplateDef(f,[],elem1*) | def1*] -> [def | def2*]
where def2* := <alltd((Call(f) -> Call("container", elem1*)))> def1*
local traversal](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-139-320.jpg)
![Rewrite in Context: Not Optimal
Inline :
[def@TemplateDef(f,[],elem1*) | def1*] -> [def | def2*]
where def2* := <alltd((Call(f) -> Call("container", elem1*)))> def1*
requires def before use
local traversal for each declaration](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-141-320.jpg)
![Dynamic Rewrite Rules
Inline :
[def@TemplateDef(f,[],elem1*) | def1*] -> [def | def2*]
where def2* := <alltd((Call(f) -> Call("container", elem1*)))> def1*
declare-inline :
TemplateDef(f,[],elem1*) -> TemplateDef(f,[],elem1*)
where rules(
InlineTemplate : Call(f) -> Call("container", elem1*)
)
inline =
alltd(declare-inline);
topdown(try(InlineTemplate))
separate traversal from rule definition (binding closures)](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-142-320.jpg)
![Inlining as Rewrite Problem (Revisited)
outputPost(p) -> navigate post(p) { output(p.name) }
where define outputPost(pst : Post) {
navigate post(pst) { output(pst.name) }
}
(informal)
declare-inline :
TemplateDef(f,[],elem1*) -> TemplateDef(f,[],elem1*)
where rules(
InlineTemplate : Call(f) -> Call("container", elem1*)
)
(formal; but not yet complete)](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-143-320.jpg)
![Inlining Templates with Parameters
declare-inline :
def@TemplateDef(mod*,f,param*,elem1*) -> def
where rules(
InlineTemplate :
Call(f, e*, []) -> Call("container", [], elem3*)
where elem3* := <substitute> (param*, e*, elem1*)
)
substitute :
(param*, e*, elem1*) -> elem2*
where {| Subst
: <zip(bind-arg)> (param*, e*)
; elem2* := <alltd(Subst)> elem1*
|}
bind-arg :
(Param(x, t), e) -> (Param(x, t), e)
where rules( Subst : Var(x) -> e )](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-145-320.jpg)
![Element Parameters
define list() { <ul> elements </ul> }
declare-inline :
def@TemplateDef(mod*,f,param*,elem1*) -> def
where rules(
InlineTemplate :
Call(f, e*, elem2*) -> Call("container", [], elem3*)
where {| Subst
: rules(
Subst : Elements() -> Call("container",[],elem2*)
)
; elem3* := <substitute> (param*, e*, elem1*)
|}
)](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-146-320.jpg)
![Removing Intermediate Structures
rules // remove containers
desugar-container :
[Call("container",[], elem1*) | elem2*] -> [elem1*, elem2*]
desugar :
elem1* -> elem2*
where elem2* := <at-suffix(desugar-container)> elem1*
container needed to replace single call with multiple elements](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-147-320.jpg)
![module template-inlining
imports libstratego-lib
imports include/nwl
imports desugar
strategies
inline-all =
desugar-all;
alltd(declare-inline);
innermost(desugar <+ InlineTemplate)
rules
declare-inline :
TemplateDef(mod*,f,param*,elem1*) -> TemplateDef(mod*,f,param*,elem1*)
where rules(
InlineTemplate :
Call(f, e*, elem2*) -> Call("container", [], elem3*)
Template Inlining where {| Subst
: rules( Subst : Elements() -> Call("container", [], elem2*) )
; elem3* := <substitute> (param*, e*, elem1*)
Transformation )
|}
substitute :
(param*, e*, elem1*) -> elem2*
where {| Subst
: <zip(bind-arg)> (param*, e*)
; elem2* := <alltd(Subst)> elem1*
|}
bind-arg :
(Param(x, t), e) -> (Param(x, t), e)
where rules( Subst : Var(x) -> e )
rules // remove containers
desugar-container :
[Call("container",[], elem1*) | elem2*] -> [elem1*, elem2*]
desugar :
elem1* -> elem2*
where elem2* := <at-suffix(desugar-container)> elem1*](https://image.slidesharecdn.com/codegen10-100618054627-phpapp01/85/Building-DSLs-with-the-Spoofax-Language-Workbench-149-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (14)
Ad
Similar to Building DSLs with the Spoofax Language Workbench (20)
Ad
More from Eelco Visser (20)
Recently uploaded (20)
Building DSLs with the Spoofax Language Workbench
- 1. Building DSLs with The Spoofax Language Workbench Eelco Visser Delft University of Technology http://eelcovisser.org Joint work with Lennart Kats, Karl-Trygve Kalleberg, Maartje de Jonge, and many others
- 2. The Meta-Languages Wars Internal DSLs vs External DSLs Textual vs Graphical Partial vs Complete Code Generation Project-Specific vs Multi-Project DSLs Code Generation vs Interpretation Dynamically vs Statically Typed
- 3. The Meta-Languages Wars Internal DSLs vs External DSLs Textual vs Graphical Partial vs Complete Code Generation Project-Specific vs Multi-Project DSLs Code Generation vs Interpretation Dynamically vs Statically Typed
- 4. WebDSL http://webdsl.org separation of concerns & linguistic integration Eelco Visser. WebDSL: A Case Study in Domain-Specific Language Engineering. GTTSE 2007: 291-373
- 6. publication records correct & extend
- 8. bibliographies tagging reputation system access control rules user groups conference calendar community engineering etc.
- 9. 18,000 lines of WebDSL code 138 (generated) tables in mysql
- 10. Concerns in web applications - Data model - User interface templates - Actions - Access control rules - Data validation WebDSL - Linguistic integration - Cross concern, static consistency checking
- 11. WebLib: a model of researchr
- 12. Data Model entity Publication { key :: String (id) title :: String (name,searchable) authors -> List<Author> journal :: String (searchable) volume :: Int issue :: Int year :: Int (searchable) month :: String (searchable) firstpage :: Int lastpage :: Int cites -> Set<Publication> citedBy -> Set<Publication> (inverse=Publication.cites) } entity Author { key :: String (id) name :: String publications -> Set<Publication> (inverse=Publication.authors) created :: DateTime modified :: DateTime }
- 13. User Interface
- 14. Full Text Search define page search() { var query : String action show() { var pubs : List<Publication> := searchPublication(query); if(pubs.length > 0) { replace(results, results(query,pubs)); } else { replace(results, noresults()); entity Publication { title :: String (name,searchable) } journal :: String (searchable) } year :: Int (searchable) main{ month :: String (searchable) section{ ... } header{"Search"} form{ input(query)[onkeyup:=show()] } placeholder results{} } } }
- 15. Page & Template Definitions define page root() { main{ section{ header{"Welcome to WebLib: Your Digital Library"} aboutWeblib() recent() tips() } } } define aboutWeblib() { par{ "WebLib is a digital library ..." } }
- 16. XML Embedding define no-span main() { <div class="page-wrap"> top() <div class="body"> messages() elements() </div> <div class="clear"></div> </div> footer() }
- 17. Page Navigation define tips() { section{ header{"What Can You Do Here?"} list{ listitem{ navigate(newauthor()){ "Add an author" } } listitem{ navigate(newpublication()){ "Add a publication" } } listitem{ navigate(search()){ "Search" } } } } } define page search() { ... }
- 18. HQL Embedding extend entity Person { publications -> List<Publication> := select p from Publication as p, Author as a where (a.person = ~this) and (p = a.publication) }
- 19. Forms & Data Binding define page editpublication(pub : Publication) { main{ section{ header{"Edit Publication '" output(pub.title) "'"} form{ group{ label("Key:" ){ input(pub.key) } label("Title:" ){ input(pub.title) } label("Authors:"){ input(pub.authors) } ... } submit action{ pub.modified := now(); return publication(pub); } { "Save" } } } } }
- 20. Access Control Danny M. Groenewegen, Eelco Visser. Declarative Access Control for WebDSL: Combining Language Integration and Separation of Concerns. ICWE 2008: 175-188
- 21. Access Control Rules define showReview(review : Review) { action delete() { ... } section{ ... navigate(editReview(review)){"[Edit]"} submitlink delete() {"[X]"} } }
- 22. Access Control Rules define showReview(review : Review) { action delete() { ... } section{ ... navigate(editReview(review)){"[Edit]"} submitlink delete() {"[X]"} } } access control rules principal is User with credentials username, password predicate isReviewer(review : Review) { review.reviewer == securityContext.principal } rule template showReview(review : Review, pubLink : Bool) { review.public || isReviewer(review) rule action delete() { loggedIn() && isReviewer(review) } } rule page editReview(review : Review) { loggedIn() && isReviewer(review) }
- 23. Data Validation Danny M. Groenewegen, Eelco Visser. Integration of Data Validation and User Interface Concerns in a DSL for Web Applications. SLE 2009: 164-173
- 24. Data Validation: Form Input Validation label("Acronym (without year)"){ input(acronym){ validate(!isEmptyString(acronym), "Provide acronym") validate(!(/[ ]/.find(acronym)), "Acronym should not contain spaces") validate(!(/[0-9][0-9][0-9][0-9]/.find(acronym)), "Acronym should not contain year") } }
- 25. Data Validation: Data Invariants entity Publication { key :: String (id, index(25), validate(isUniquePublication(this), "that key is already in use"), validate(isValidKeyForURL(key), "Key should consist of letters, digits, : and -")) ... }
- 26. Data Validation: Assertions extend entity Person { invitation -> Invitation (inverse=Invitation.invitee) function invite(email : Email, invitation : WikiText) : Invitation { validate(findUser(email) == null && findInvitationByEmail(email).length == 0, "That email address is already in use."); ... } }
- 27. Concerns in web applications - Data model - User interface templates - Actions - Access control rules - Data validation WebDSL - Linguistic integration - Cross concern, static consistency checking
- 28. IDEs for DSLs
- 29. parse check Model AST Errors desugar Traditional Compiler core Architecture generate Code
- 30. Compiler Ingredients Syntax definition ★ concrete syntax ★ abstract syntax Static semantics ★ error checking ★ name resolution ★ type analysis Model-to-model transformation ★ express constructs in core language Code generation ★ translate core language models to implementation
- 31. IDEs Required
- 32. Editor Services for Modern IDEs Syntactic Editor Services ★ syntax checking ★ bracket matching ★ syntax highlighting ★ code folding ★ outline view Semantic Editor Services ★ error checking ★ reference resolving ★ hover help ★ code completion ★ refactoring
- 33. parse Editor AST feedback core Integrated Development Environment Code
- 34. Architectural Requirements for IDEs Editor services are based on AST ★ services analyze structure, not text of source Error recovery ★ continue services in presence of syntactic errors Incremental processing ★ effort of applying analysis, transformation, generation should be proportional to change in source Separate compilation (analysis, transformation) ★ keep track of dependencies
- 35. Holy Grail of Software Language Definition Automatically derive efficient, scalable, incremental compiler + usable IDE from high-level, declarative language definition
- 37. The Spoofax/IMP Language Workbench Syntax definition: SDF ★ declarative, modular syntax definition Transformation, analysis, generation: Stratego ★ rewrite rules, strategies, dynamic rules Editor services ★ domain-specific languages for configuration Based on IMP framework by IBM Research
- 38. Syntax Definition
- 39. Syntax Definition Pretty-Print Parse Table Signature Table Parse Transform Pretty-Print @Entity entity User { class User { name :: String String _user; pw :: Secret public User } getUser() { def output(u : syntax definition is basis of language definition return _user; User) { }
- 40. Lexical & Context-Free Syntax
- 41. Structure of Sentences composed from
- 42. Context-Free Grammar sort regular expression symbol Definition grammar production Property* {Stat “;”}* “entity”
- 43. Lexical Syntax regular expression character class characters in lexical syntax not separated by layout
- 44. Syntax of Data Models
- 46. Lexical Syntax: Identifiers & Literals follow restriction: symbol may not reserved words be followed by character from this class: BlogEntry is one ID
- 47. Lexical Syntax: Strings escape complement escape escape
- 48. Lexical Syntax: Whitespace LAYOUT? is inserted between symbols in CF productions normalize
- 50. Parsing & Abstract Syntax building structured representation of sentence
- 51. Linking Concrete & Abstract Syntax
- 52. Terms to Represent Abstract Syntax Entity(“Blog”, [Property(“name”,SimpleType(“String”)), Property(“author”, SimpleType(“User”))] )
- 54. Signatures defining the structure of abstract syntax terms
- 55. Syntax Definition Pretty-Print Parse Table Signature Table Parse Transform Pretty-Print @Entity entity User { class User { name :: String String _user; pw :: Secret public User } getUser() { def output(u : syntax definition is basis of language definition return _user; User) { }
- 56. Signatures declare arguments and types of constructors signature is generated from grammar
- 57. Disambiguation
- 58. Arithmetic Expressions note: this style produces good abstract syntax Plus(Times(Var(“x”),Int(2)),Var(“z”))
- 60. Associativity
- 61. Priority
- 63. Brackets
- 65. Embedding HQL module expressions imports Common imports HQL exports context-free syntax SelectStatement -> Exp define page index() { list{ for(p : Post in select p from Post as p order by (p.title)) { listitem{ output(p) } } } } ForElem("p", SimpleType("Post") , QueryRule( SelectFrom( Some(Select(None(), [AliasedExpression(Path(["p"]), None())])) , FromClause([FromRangeJoin(FromClass(Path(["Post"]), Some(AsAlias(Alias("p" ) , Some(OrderByClause([OrderElement(Paren([Path(["p", "title"])]), None())])) ) , [CallElems("listitem", [CallArgs("output", [Var("p")])])] )
- 67. Editor Services Syntax highlighting ★ token coloring Code folding ★ folding & unfolding code fragments Outline view ★ table of contents Syntax properties ★ bracket matching, indentation Syntax completion ★ match on syntactic triggers & replace with template
- 68. Editor Services Composition module nwl.main imports nwl-Builders nwl-Colorer nwl-Folding nwl-Outliner nwl-References nwl-Syntax nwl-Completions language General properties name : nwl id : nwl extends : Root description : "Spoofax/IMP-generated editor for the nwl language" url : http://strategoxt.org extensions : nwl table : include/nwl.tbl start symbols : Start
- 69. Syntax Definition Default Syntax Default Default Highlighting Code Folding Outline View Custom Syntax Custom Custom + Highlighting + Code Folding + Outline View Editor
- 70. Code Folding module nwl-Folding.generated folding Default folding Start.Module Definition.Entity Property.Property Definition.TemplateDef Element.CallArgs PageRef.PageRef Element.Action module nwl-Folding imports nwl-Folding.generated folding Element.CallElems Element.Call Element.ForElem Element.ForAllElem
- 71. Outline View module nwl-Outliner.generated outliner Default outliner Start.Module Definition.Entity Property.Property Definition.TemplateDef Element.CallArgs Exp.MethodCall Element.CallElems Element.Call Element.ForElem Element.ForAllElem PageRef.PageRef Element.Action Statement.For Statement.While Statement.If Element.Submit Element.XmlElem
- 72. module nwl-Colorer.generated colorer Default, token-based highlighting Default Syntax Highlighting keyword : 127 0 85 bold identifier : default string : blue number : darkgreen var : 255 0 100 italic operator : 0 0 128 layout : 100 100 0 italic colorer System colors darkred = 128 0 0 red = 255 0 0 darkgreen = 0 128 0 green = 0 255 0 darkblue = 0 0 128 blue = 0 0 255 cyan = 0 255 255 magenta = 255 0 255 yellow = 255 255 0 white = 255 255 255 black = 0 0 0 gray = 128 128 128 grey = gray orange = 255 165 0 pink = 255 105 180 brown = 139 69 19 default = _
- 73. Custom Syntax Highlighting module nwl-Colorer imports nwl-Colorer.generated colorer Element : darkgreen
- 74. Syntax Properties module nwl-Syntax.generated language Syntax properties (static defaults) // Comment constructs: line comment : "//" block comment : "/*" * "*/" // Fences (used for matching, // inserting, indenting brackets): fences : [ ] ( ) { } // Automatic indent hints // (indent after these tokens): indent after : "=" ":" // Regular expression for identifiers: identifier lexical : "[A-Za-z0-9_]+"
- 76. Parse Error Recovery also for composite languages
- 78. Builders: Analysis & Transformation Services module nwl-Builders builders provider: include/nwl.ctree observer: editor-analyze builder : "Generate Java code" = generate-java (openeditor) (realtime) builder : "Show ATerm (selection)" = generate-aterm (openeditor) (realtime) (meta) builder : "Normalize (selection)" = show-normalized (openeditor) (realtime) (meta) builder : "Normalize Pretty (selection)" = show-normalized-pp (openeditor) (realtime) (meta)
- 79. Term Rewriting
- 80. Term Rewriting Term rewrite rules ★ transform term to term ★ pattern matching ★ variable binding ★ substitution Rewriting strategy ★ algorithm for applying rewrite rules
- 81. Term Rewrite Rule left-hand side pattern label/name desugar : Property(x, t) -> Property(x, t, []) variable right-hand side pattern
- 82. Rewrite Strategy generic strategy strategy definition desugar-all = innermost(desugar) strategy instantiation apply transformation s exhaustively to all sub- innermost(s) terms of subject term in bottom-up order
- 83. Constant Folding y := a + (3 * 5 + 2); Assign( Var("y") , Plus( Var("a") , Plus(Times(IntLit("3"), IntLit("5")), IntLit("2")) parse ) ) Assign( Var("y") , BinOp(Var("a"), "+", IntLit("17")) desugar + eval ) y := (a + 17); pretty-print
- 84. Constant Folding Rules strategies eval-all = innermost(desugar + eval) rules eval : BinOp(IntLit(x), "+", IntLit(y)) -> IntLit(<addS>(x, y)) eval : BinOp(IntLit(x), "-", IntLit(y)) -> IntLit(<subtS>(x, y)) eval : BinOp(IntLit(x), "*", IntLit(y)) -> IntLit(<mulS>(x, y)) eval : BinOp(IntLit(x), "/", IntLit(y)) -> IntLit(<divS>(x, y))
- 85. Conditional Rewrite Rules bound in condition eval : BinOp(IntLit(x), "+", IntLit(y)) -> IntLit(z) condition where z := <addS>(x, y) match apply transformation eval : BinOp(IntLit(x), "+", IntLit(y)) -> IntLit(<addS>(x, y))
- 86. Code Generation by Model Transformation
- 87. Rewriting with Concrete Object Syntax elem-to-java-servlet : tc@elem|[ x(e)[passign*]{elem*} ]| -> <wrap-input-render-java-code> bstm* |[ String x_temp = ident+"~inputident"+uniqueid; ~*<render-error-messages-with-error-template(|java:expr|[ x_temp ]|, <ErrorTemplateInput>)> bstm*|[ bstm_call* ]| ]| where <is-input-template> tc with inputident := <get-inputnumber> tc ; x_temp := <newname> "temp" ; bstm_call* := <control-flow-tcall-helper(|"render",expr|[ x_temp ]|)> tc Stratego + embedded Java
- 89. Consistency Checking Syntax definition ★ what are well-formed sentences? Static analysis ★ not all ‘well-formedness’ properties are context-free ★ consistency of compositions ★ consistency of expressions wrt declarations Error reporting ★ indicate errors in editor ★ use sensible error message
- 92. Consistency Checking: Ingredients Editor Interface ★ collecting and displaying errors, warnings Error checking ★ checking static constraints and reporting errors Type analysis ★ computing types of expressions Name resolution ★ disambiguation of names Reference resolving ★ linking identifiers to declarations
- 93. Consistency Checking: Generic Approach Rename ★ make identifiers unique Map ★ map identifiers to declarations Project ★ compute properties of declarations, expressions Check ★ check constraints
- 94. Editor Interface module nwl-Builders imports nwl-Builders.generated builders provider: include/nwl.ctree observer: editor-analyze editor/nwl-Builders.esv trans/static-analysis.str module static-analysis imports include/nwl imports entities imports utils rules // static analysis editor-analyze: (ast, path, fullpath) -> (errors, warnings, notes) with ...
- 95. Editor Interface editor-analyze: (ast, path, fullpath) -> (errors, warnings, notes) with errors := <collect-all(check, conc)> ast; warnings := <collect-all(constraint-warning, conc)> ast; notes := <collect-all(constraint-note, conc)> ast
- 96. Error Checking Rules check : context -> (target, error) where assumption where require(constraint) require(s) = not(s) – Context: identifying points in the code to check – Assumptions: only report an error if certain assumptions hold (validating the context and avoiding spurious errors) – Constraints: checking for constraints at the context – Formulating an error message – Attribution of the error to a particular character range in the source text (usually, only part of the context
- 97. Error Checking: Binary Operators check : e@BinOp(e1, op, e2) -> (e, $[operator [op] not defined for [<pp>t1] and [<pp>t2]]) where t1 := <type-of> e1 where t2 := <type-of> e2 where require(<type-of> e)
- 98. Pretty-Printing with String Interpolation pp : Entity(x, prop*) -> $[entity [x] { [<map(pp)>prop*] }] pp : Property(x,t) -> $[[x] : [<pp>t] ] pp : SimpleType(x) -> x pp : SetType(t) -> $[Set<[<pp> t]>] pp : [] -> $[] pp : [t] -> <pp>t pp : [t1,t2|ts] -> $[[<pp>t1],[<pp>[t2|ts]]]
- 99. Type Analysis type-of : e -> t compute type of expression
- 100. Type Analysis: Literals type-of : StringLit(x) -> SimpleType("String") type-of : IntLit(x) -> SimpleType("Int")
- 101. Type Analysis: Binary Operators type-of : BinOp(e1, op, e2) -> t where t := <function-type>(op, [<type-of>e1, <type-of>e2]) function-type : ("+", [SimpleType("String"), SimpleType("String")]) -> SimpleType("String") function-type : ("+", [SimpleType("Int"), SimpleType("Int")]) -> SimpleType("Int") function-type : ("-", [SimpleType("Int"), SimpleType("Int")]) -> SimpleType("Int")
- 102. Type Analysis: What is Type of Variable? define page root(x : Int) { action exptest() { for(y : Int in {1,2,x}) { x := x + y; } } type-of : } Var(x) -> t where t := ??? Assign( Var("x") , BinOp(Var("x"), "+", Var("y")) ) type of variable not part of variable use
- 103. Variables: Map declare-all = alltd(declare) declare : Param(x, t) -> Param(x, t) with rules( TypeOf : x -> t ) type-of : Var(x) -> t where t := <TypeOf> x
- 104. Scope define page root(x : Int) { action exptest() { for(x : Int in {1,2,x}) { print(x); } } } multiple occurrences of same identifier corresponding to different declarations
- 105. Variables: Map + Rename rename-all = alltd(rename) rename : Param(x, t) -> Param(y, t) with y := <rename-var>(x, t) unique annotation rename-var : (x, t) -> y with y := x{<new>}; map variable to type rules( TypeOf : y -> t RenameId : x -> y ) rename occurrences rename : Var(x) -> Var(y) where y := <RenameId> x type-of : Var(x) -> t where t := <TypeOf> x
- 106. Term Annotations t{t1,...,tn} add additional information to term without affecting signature
- 107. Variables: Check check : e@Var(x) -> (e, $[Variable '[x]' not declared]) where require(<type-of>e)
- 108. Variable Binding Constructs rename : For(x, t, e1, stat1*) -> For(y, t, e2, stat2*) with e2 := <rename-all> e1 with {| RenameId : y := <rename-var>(x, t) ; stat2* := <rename-all> stat1* |} For defines local variable x in body stat1*’
- 109. Editor Interface with Analysis editor-analyze: (ast, path, fullpath) -> (errors, warnings, notes) with ast2 := <analyze> ast; errors := <collect-all(check, conc)> ast2; warnings := <collect-all(constraint-warning, conc)> ast2; notes := <collect-all(constraint-note, conc)> ast2 analyze = rename-all
- 110. Rename, Map, Project, Check Rename ★ make local variables unique Map ★ variables to their type Project ★ compute type of expressions Check ★ check constraints using types
- 111. Reference Resolution
- 112. Reference Resolution Aiding program navigation ★ Hover-click on identifier to jump to declaration Reuse name resolution infrastructure
- 113. Reference Resolution module nwl-References imports nwl-References.generated references reference _ : editor-resolve editor-resolve: (source, position, ast, path, fullpath) -> target where target := <compute-target> source
- 114. From Use to Declaration editor-resolve: (SimpleType(type), position, ast, path, fullpath) -> target where Entity(target,_) := <declaration-of> type editor-resolve: (ref@PageRef(x,e*), position, ast, path, fullpath) -> target where TemplateDef(_,target,_,_) := <declaration-of> ref
- 115. Experience
- 116. Experience Bootstrapped ★ SDF, Stratego, ATerm, PP ★ Editor service language Spoofax ★ WebDSL ★ Mobl: mobile web applications ★ Acoda: migration of data models ★ Aster: attribute grammars ★ PIL: platform independent language ★ Student languages SDF+Stratego ★ Java, AspectJ, XML, PHP, SQL, Jimple, Octave, Dot, ...
- 117. Future Work (in progress) Higher-level definition of scope & type system ★ cover scope systems of real languages Refactoring ★ layout preservation, generic refactoring strategies Systematic language design ★ how do we decide that a language design is good? Interaction design ★ higher-level specifications of interaction Combining textual and visual Textual IDE in the browser
- 118. - the end - Lennart C. L. Kats, Eelco Visser. The Spoofax Language Workbench. Rules for Declarative Specification of Languages and IDEs. OOPSLA 2010 http://spoofax.org http://webdsl.org http://mobl.org
- 120. define page blog(b : Blog) { Template Inlining header{output(b.name)} list{ for(p : Post in b.posts) { listitem{ outputPost(p) } } } } define outputPost(pst : Post) { navigate post(pst) { output(pst.name) } } define list() { <ul> elements </ul> } define listitem() { <li> elements </li> } define header() { <h1> elements </h1> } define page blog ( b : Blog ) { <h1> output(b.name) </h1> <ul> for ( p : Post in b.posts ) { <li> navigate post(p) { output(p.name) } </li> } </ul> }
- 121. Context-sensitive Transformation Local-to-local ★ replace term by another term Local-to-global ★ local term influence terms in other parts of model Global-to-local ★ global information influences transformation of local terms
- 122. define page blog(b : Blog) { header{output(b.name)} list{ for(p : Post in b.posts) { listitem{ outputPost(p) } } } Inlining is Global-to-Local } define outputPost(pst : Post) { navigate post(pst) { output(pst.name) } } define list() { <ul> elements </ul> } define listitem() { <li> elements </li> } define header() { <h1> elements </h1> } define page blog ( b : Blog ) { <h1> output(b.name) </h1> <ul> for ( p : Post in b.posts ) { <li> navigate post(p) { output(p.name) } </li> } </ul> }
- 123. Inlining as Rewrite Problem outputPost(p) -> navigate post(p) { output(p.name) } where define outputPost(pst : Post) { navigate post(pst) { output(pst.name) } } (this is not a valid Stratego rewrite rule)
- 124. Rewrite Rules are Context-free desugar : Property(x, t) -> Property(x, t, [])
- 125. Simplification: Parameterless Templates define page blog(b : Blog) { ... footer() } define footer() { navigate url(http://webdsl.org) { “WebDSL” } } define page blog ( b : Blog ) { ... container() { navigate url(http://webdsl.org) { “WebDSL” } } }
- 126. Rewrite in Context Inline : [def@TemplateDef(f,[],elem1*) | def1*] -> [def | def2*] where def2* := <alltd((Call(f) -> Call("container", elem1*)))> def1* local traversal bound in context container needed to replace single call with multiple elements
- 128. transformation is partial* function from terms to terms * transformation may fail to apply to a term
- 129. Defining Transformations: Rules & Strategies Rewrite rules are basic transformations ★ transform term matching lhs to instantiation of rhs ★ evaluate condition (where clause) Strategies combine rules into complex transformations ★ select rules to apply ★ select algorithm to apply rules Strategy combinators ★ composition of custom transformations
- 130. Strategy Combinators id ★ Identity fail ★ failure s1 ; s2 ★ sequential composition s1 <+ s2 ★ choice
- 131. Variables {x, y : s} ★ term variable scope ?t ★ match term pattern t !t ★ build term pattern t t1 := t2 ★ match term t2 to term (pattern) t1 <s> t ★ apply strategy s to term t
- 132. Rewrite Rules l : t1 -> t2 where s ★ named, scoped rewrite rule ★ all variables in t1, t2, s are in scope of the rule (t1 -> t2 where s) ★ unscoped rewrite rule ★ variables are in scope of enclosing scope
- 133. Rewrite in Context Inline : [def@TemplateDef(f,[],elem1*) | def1*] -> [def | def2*] where def2* := <alltd((Call(f) -> Call("container", elem1*)))> def1* bound in context
- 134. Inline : [def@TemplateDef(f,[],elem*) | def1*] -> [def | def2*] where def2* := <alltd((Call(f) -> Call("container", elem*)))> def1* f elem* [ TemplateDef( "footer" , [] , [Navigate(PageRef("webdsl", []), [String("WebDSL")])] ) , TemplateDef( "blog" , [Param("b", SimpleType("Blog"))] , [Call("footer")] ) ] Call(f) def1*
- 135. Strategy Definitions f(x,y|a,b) = s ★ x, y are strategy parameters ★ a, b are term parameters ★ s uses all parameters f(x) = s f = s ★ term parameters are optional ★ all parameters are optional Examples ★ try(s) = s <+ id ★ repeat(s) = try(s; repeat(s))
- 136. Rules with Parameters Transform all elements of a list map(s) : [] -> [] map(s) : [x|xs] -> [<s>x | <map(s)> xs] Invert order of elements of a list inverse(|ys) : [] -> ys inverse(|ys) : [x|xs] -> <inverse(|[x|ys])> xs Pair elements of two lists zip(s) : ([],[]) -> [] zip(s) : ([x|xs],[y|ys]) -> [<s>(x,y) | <zip(s)>(xs,ys)]
- 137. Traversal Combinators all(s) ★ apply s to all direct sub-terms (children) one(s) ★ apply s to exactly one sub-term some(s) ★ apply s to at least one sub-term
- 138. Traversal Strategies topdown(s) = s; all(topdown(s)) ★ apply s to all sub-terms in top-down order bottomup(s) = all(bottomup(s)); s ★ apply s to all sub-terms in bottom-up order oncetd(s) = s <+ one(oncetd(s)) ★ apply s to one sub-term alltd(s) = s <+ all(alltd(s)) ★ apply s to frontier
- 139. Rewrite in Context: Local Traversal Inline : [def@TemplateDef(f,[],elem1*) | def1*] -> [def | def2*] where def2* := <alltd((Call(f) -> Call("container", elem1*)))> def1* local traversal
- 141. Rewrite in Context: Not Optimal Inline : [def@TemplateDef(f,[],elem1*) | def1*] -> [def | def2*] where def2* := <alltd((Call(f) -> Call("container", elem1*)))> def1* requires def before use local traversal for each declaration
- 142. Dynamic Rewrite Rules Inline : [def@TemplateDef(f,[],elem1*) | def1*] -> [def | def2*] where def2* := <alltd((Call(f) -> Call("container", elem1*)))> def1* declare-inline : TemplateDef(f,[],elem1*) -> TemplateDef(f,[],elem1*) where rules( InlineTemplate : Call(f) -> Call("container", elem1*) ) inline = alltd(declare-inline); topdown(try(InlineTemplate)) separate traversal from rule definition (binding closures)
- 143. Inlining as Rewrite Problem (Revisited) outputPost(p) -> navigate post(p) { output(p.name) } where define outputPost(pst : Post) { navigate post(pst) { output(pst.name) } } (informal) declare-inline : TemplateDef(f,[],elem1*) -> TemplateDef(f,[],elem1*) where rules( InlineTemplate : Call(f) -> Call("container", elem1*) ) (formal; but not yet complete)
- 144. define page blog(b : Blog) { Template Parameter header{output(b.name)} list{ for(p : Post in b.posts) { listitem{ outputPost(p) } } } } define outputPost(pst : Post) { navigate post(pst) { output(pst.name) } } define list() { <ul> elements </ul> } define listitem() { <li> elements </li> } define header() { <h1> elements </h1> } define page blog ( b : Blog ) { <h1> output(b.name) </h1> <ul> for ( p : Post in b.posts ) { <li> navigate post(p) { output(p.name) } </li> } </ul> }
- 145. Inlining Templates with Parameters declare-inline : def@TemplateDef(mod*,f,param*,elem1*) -> def where rules( InlineTemplate : Call(f, e*, []) -> Call("container", [], elem3*) where elem3* := <substitute> (param*, e*, elem1*) ) substitute : (param*, e*, elem1*) -> elem2* where {| Subst : <zip(bind-arg)> (param*, e*) ; elem2* := <alltd(Subst)> elem1* |} bind-arg : (Param(x, t), e) -> (Param(x, t), e) where rules( Subst : Var(x) -> e )
- 146. Element Parameters define list() { <ul> elements </ul> } declare-inline : def@TemplateDef(mod*,f,param*,elem1*) -> def where rules( InlineTemplate : Call(f, e*, elem2*) -> Call("container", [], elem3*) where {| Subst : rules( Subst : Elements() -> Call("container",[],elem2*) ) ; elem3* := <substitute> (param*, e*, elem1*) |} )
- 147. Removing Intermediate Structures rules // remove containers desugar-container : [Call("container",[], elem1*) | elem2*] -> [elem1*, elem2*] desugar : elem1* -> elem2* where elem2* := <at-suffix(desugar-container)> elem1* container needed to replace single call with multiple elements
- 148. Inlining Strategy module template-inlining imports libstratego-lib imports include/nwl imports desugar strategies inline-all = desugar-all; alltd(declare-inline); innermost(desugar <+ InlineTemplate) rules declare-inline : ...
- 149. module template-inlining imports libstratego-lib imports include/nwl imports desugar strategies inline-all = desugar-all; alltd(declare-inline); innermost(desugar <+ InlineTemplate) rules declare-inline : TemplateDef(mod*,f,param*,elem1*) -> TemplateDef(mod*,f,param*,elem1*) where rules( InlineTemplate : Call(f, e*, elem2*) -> Call("container", [], elem3*) Template Inlining where {| Subst : rules( Subst : Elements() -> Call("container", [], elem2*) ) ; elem3* := <substitute> (param*, e*, elem1*) Transformation ) |} substitute : (param*, e*, elem1*) -> elem2* where {| Subst : <zip(bind-arg)> (param*, e*) ; elem2* := <alltd(Subst)> elem1* |} bind-arg : (Param(x, t), e) -> (Param(x, t), e) where rules( Subst : Var(x) -> e ) rules // remove containers desugar-container : [Call("container",[], elem1*) | elem2*] -> [elem1*, elem2*] desugar : elem1* -> elem2* where elem2* := <at-suffix(desugar-container)> elem1*