RepreZen DSL: Pushing the limits of language usability with XText
- 1. RAPID – Resource API Design Language Pushing the limits of language usability with XText TED EPSTEIN, FOUNDER AND CEO TANYA FESENKO, LEAD DEVELOPER EclipseCon NA 2015, Xtext Day 1COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. ™ ™
- 2. Overview • RepreZen: solving API interoperability • Challenge: a better API description language • RAPID Language Features ◦ Optional Fluency: accommodating coding styles ◦ Keyword Sequence: encourage, don’t Enforce ◦ Indent-Based Block Scoping: life without curly braces • Implementation Overview • Summary • Q&A 2COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED.
- 3. COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. 3 / Unify Data Models / Federate APIs / Integrate Faster The Elevator Pitch
- 4. The Elevator Pitch COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. 4 Make Canonical Data Models Practical Enable Highly Interoperable APIs Significantly Lower Long- Term Integration Cost • Frequently recommended, hard to implement • Current reality: much easier to build from scratch than to reuse existing data models • We want to change that!
- 5. COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. 5 Make Canonical Data Models Practical Enable Highly Interoperable APIs Significantly Lower Long- Term Integration Cost • Frequently recommended, hard to implement • Current reality: much easier to build from scratch than to reuse existing data models • We want to change that! • Reusable data models: business-oriented, technology-independent, semantically rich • Allow variability through realization • Message payloads still conform to canonical model Property Subsets Perspective <party dataSource=“MSDB”> <partyId>123</partyId> <partyName xsi:nil=“true” nullValue=“Not Available” /> <accounts> <account dataSource=“A2” transType=“insert”> <accountId>XYZ</accountId> <balance xsi:nil=“true” isRestricted=“true” /> </account> </accounts> </party> Metadata Contextual Constraints The Elevator Pitch
- 6. COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. 6 Make Canonical Data Models Practical Enable Highly Interoperable APIs Significantly Lower Long- Term Integration Cost • Frequently recommended, hard to implement • Current reality: much easier to build from scratch than to reuse existing data models • We want to change that! • Reusable data models: business-oriented, technology- independent, semantically rich • Allow variability through realization • Message payloads still conform to canonical model • Service APIs speak the same language • Clients can work with multiple APIs, without multiple DTOs, SDKs, etc. • Stop runaway spending on remedial integration technology
- 7. Changing the way we build APIs Message Schema COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. 7 Business Domain Model Realization • Subset • Perspective • Constraints • Metadata Resource Data Model Representation • Media Type • Format • Conventions • Hyperlinks This needs to be built into the API modeling language.
- 8. Yet Another API Description Language? YAADL? YAAPIDL? There are at least four of these... “The nice thing about standards is that you have so many to chose from.” - Andrew Tanenbaum COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. 8 … and they’re pretty good. ◦ Web-based editors ◦ API Documentation with integrated sandbox ◦ Syntax based on YAML or Markdown ◦ Interesting language-level reuse ◦ REST patterns built in If we have to build an API modeling language, it has to be good. … any ideas?
- 9. Who’s going to model these APIs? COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. 9 …andDevelopers! Developers! Developers! Business Analysts Subject Matter Experts Data Professionals Enterprise Architects Make it accessible, readable, versatile. What she said.
- 10. Demo – RAPID Usability Features • Indent Syntax: ◦ Python-like block syntax, no curly braces ◦ Advanced highlighting to visualize structure • Order Assist: ◦ Flexible sequence for declarations ◦ Reset to standard sequence using QuickFix or auto-format • Optional Fluency ◦ Support terse and fluent syntax with optional keywords ◦ Semantic highlighting to distinguish optional vs. required keywords COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. 10
- 11. 11 Indent-Based Syntax Structural Highlighting - editbox.sourceforge.net New Syntax – indents define block scope Previous Syntax – braces for block scoping
- 12. Order Assist 12
- 14. Implementation Walkthrough COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. 14
- 16. 16 Work with Xtext Xtext Grammar (*.xtext file) Generated code (Parser and UI)
- 17. 17 Work with Xtext The rest (customized from generated code) Lexer Formatter Content assist Code Templates Prefs Cross- reference Scope Code highlighter Validation/ Quickfix Outline … Xtext Grammar (*.xtext file) Generated code (Parser and UI)
- 18. Description Grammar Code Customization for (each feature): 18 Three features
- 19. Feature 1: Automate canonical ordering 19
- 21. 21 Implementation: Grammar The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code
- 22. 22 Allow arbitrary order in grammar
- 23. 23 Allow arbitrary order in grammar
- 24. 24 Allow arbitrary order in grammar Disable code-assist and add validator for duplicate single- valued properties to prevent overriding
- 25. 25 Sort List< INode> Compare INodes: • Using sorting rules • Preserving: • List delimiters • Whitespaces • Attached symbols (*, +) • Order of elements of the same kind Grammar Tree Sorting rule: Sequence<GrammarElement>
- 26. 26 Implementation: Grammar The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code Validation
- 29. 29 Implementation: Grammar The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code Formatter
- 32. 32 FormatterDefaultNodeModelFormatter#format(): INode: • offset • text • length • grammarElement • semanticElement
- 33. 33 FormatterDefaultNodeModelFormatter#format(): Writes tokens to character stream and then flushes it The write methods create formatted LineEntries with use of formatter configuration (ElementLocators)
- 34. 34 FormatterDefaultNodeModelFormatter#format(): TextRegion – offset and length in the char stream populated in the write methods
- 36. 36 Implementation: Grammar The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code Preferences page
- 38. 38 Testing 1 2 3 4 5 How many permutations for five elements ? 5!=120
- 39. 39 Testing Xtend Active Annotations 120 green tests!
- 40. 40 Changes: Automate Canonical Ordering The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code Preferences pageFormatterValidation Enable arbitrary order
- 41. 41 The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code Preferences pageFormatterValidation • Errors for single-value property override • Warnings for "incorrect" order Changes: Automate Canonical Ordering
- 42. 42 The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code Preferences pageFormatterValidation Restore canonical order Changes: Automate Canonical Ordering
- 43. 43 The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code Preferences pageFormatterValidation enable/disable validation, reordering in formatter Changes: Automate Canonical Ordering
- 45. What is Optional Fluency? 45 Fluent Terse
- 46. 46 Implementation: Grammar The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code
- 49. 49 Changes: Optional Fluency The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code Optional fluency
- 50. Feature 3: Indent-based block scoping in Xtext 50
- 51. 51 Indent-based block scoping in Xtext Python-like notation With curly braces
- 52. 52 Indent-based block scoping in Xtext Highlighting of boxes- editbox.sourceforge.net
- 53. 53 Implementation: Grammar The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code
- 54. 54 Grammar NL - newline WS – whitespace BEGIN – indent END - outdent
- 55. 55 Grammar NL: not hidden NL - newline WS – whitespace BEGIN – indent END - outdent
- 56. 56 Grammar NL: not hidden NL - newline WS – whitespace BEGIN – indent END - outdent
- 57. 57 Implementation: Grammar The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code Lexer
- 58. 58 Custom Lexer Custom code Generated / Xtext code Indent-based Syntax in Xtext - tinyurl.com/pgcowct Insert BEGIN and END token
- 60. 60 Implementation: Grammar The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code Content Assist
- 61. 61 Custom Lexers Plural now Custom code Generated / Xtext code Content-assist needs its own lexer
- 63. Content Assist 63 Also see “Enhancing Content Assist” from XtextCON 2015 - tinyurl.com/onou2hu
- 64. Content Assist 64 Also see “Enhancing Content Assist” from XtextCON 2015 - tinyurl.com/onou2hu
- 65. Content Assist 65 Also see “Enhancing Content Assist” from XtextCON 2015 - tinyurl.com/onou2hu
- 67. Content-Assist Context Factory 67 Initialize data Analyze data to create content assist contexts Content Assist Contexts
- 68. Content-Assist Context Factory 68 Initialize data Analyze data to create content assist contexts Content Assist Contexts Syntax tree nodes – lastCompleteNode, lastVisibleNode, currentNode, datatypeNode Semantic model - current model
- 69. Initialize data Analyze data to create content assist contexts Content Assist Contexts Content-Assist Context Factory 69 Start a new token, complete current one, collect available alternatives, etc…
- 70. Initialize data Analyze data to create content assist contexts Content Assist Contexts Content-Assist Context Factory 70 Customize to respect indents
- 71. 71 Implementation: Grammar The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code Formatter
- 72. Formatter 72 Assign correct value to FormattingConfigBasedStream #indentationLevel No spaces around “invisible” rules – synthetic (BEGIN and END) and NL (newline) No line wraps should be introduced by formatter – line wraps change structure of Python-like models
- 73. 73 Changes: Indent-based Syntax The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code FormatterContent AssistLexer Add BEGIN and END terminal rules
- 74. 74 The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code FormatterContent AssistLexer Synthesize BEGIN and END tokens Changes: Indent-based Syntax
- 75. 75 The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code FormatterContent AssistLexer Show proposals according to indent level Changes: Indent-based Syntax
- 76. 76 The rest (customized from generated code) Xtext Grammar (*.xtext file) Generated code FormatterContent AssistLexer Preserve NLs and indents Changes: Indent-based Syntax
- 77. Summary • Still an ongoing experiment ◦ Early feedback from users has been very positive. ◦ We welcome your feedback! • API Studio is a commercial product, still pre-launch • Offering a free license with 1-year maintenance to Conference attendees (1 per organization) • Register at get.reprezen.com with invitation code eclipsecon2015 • Contacts: ◦ [email protected] ◦ [email protected] COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. 77
- 78. Questions THANK YOU! COPYRIGHT © 2015, MODELSOLV, INC. | ALL RIGHTS RESERVED. 78