Type Profiler: Ambitious Type Inference for Ruby 3

Type Profiler:
Ambitious Type Inference
for Ruby 3
Yusuke Endoh (@mametter)
RubyKaigi Takeout 2020
1

Goals of Ruby 3’s Static Analysis
•Make Ruby programming easier
• Bug detection before execution
• Completion and document in IDE
… with no type annotation in code!
(Ruby 3 Type Challenge)
2
42 + "str"
Is this a bug?
42.ti|
Do you mean:
42.times {|i|
|
}

Ruby 3 will provide three “type” items
3
# app.rbs
def inc:
(Integer) -> Integer
① Type Signature
Format (RBS)
# app.rb
def inc(n)
n+1
end
Ruby code
② Type Inference
(ruby-type-profiler)
③ Type Check
(Steep, Sorbet, RDL, …)

Ruby 3 Development Experience
4
gem
lib.rb
lib.rbs
app
app.rb
app.rbs
42.ti|
42 + "str"
Is this a bug?
Do you mean:
42.times {|i|
|
}
② Type
Inference
• Generate an RBS prototype
• Simply check the code
③ Type
Checker
• Verify .rb and .rbs
• Serve as LSP server
You can also write RBS
manually if you want
Type-guided
linter
Dynamic
type checker
More dedicated
dev. environments
Monitor and Harness
Ruby code in run-time
Code formatter by
leveraging types
Rubymine, Tabnine, and
other development tools
may use type informationhttps://github.com/pocke/rubocop-typedhttps://github.com/ruby/rbs/blob/master/
lib/rbs/test/type_check.rb
RBS may inspire other dreams
Today
we talk about
①RBS

Agenda
➔Type Profiler: Type Inference for Ruby 3
•Demo
•How to use Type Profiler
•Future plan
5

What is Type Profiler
Abstract ("type-level") interpreter of Ruby
Generates an RBS prototype by gathering
what types a method accepts and returns
6
def foo(n)
n.to_s
end
foo(42)
Integer
(not 42)
String
(not "42")
def foo:
(Integer) -> String

Why is the name "Type Profiler"?
•Just for a historical reason
• The initial version was runtime analysis (profiling)
• Now it is a bit confusing with a normal profiler
•Invite suggestions for the name
• Should it start with "S"? (Steep, Sorbet, …)
7

Difference from traditional type system
•Traditional type systems use
intra-procedural (per-method) analysis
• Can't handle unannotated method parameter well
• … especially when there are
many classes that respond to foo
8
def f(x)
x.foo
end
What type is "x"?
class Foo
def foo; ...; end
end
class Bar
def foo; ...; end
end
class Baz
def foo; ...; end
end
...

Difference from traditional type system
•Solutions
1. Write type annotation → Avoid this
2. Infer type based on its usage
→ Too strict or too conservative
• foo must be unique
• or, structural type inference?
3. Use "inter-procedural" analysis
• Pros: More powerful analysis
• Cons: Slow and hard to control
(Challenging)
9
def f(x: Foo)
x.foo
end
def f(x)
x.foo
end
f(Foo.new)
"x" is
a Foo!
def f(x)
x.foo
end
"x" is an object that
responds to foo that
accepts no argument and
returns the same type of
a return value of this method

There are many, many topics (but omit)
Theoretical issues
• Recursion and closures[RubyKaigi 2019]
• Type-changing variable assignment
• Container types and destructive
operations[Osaka Ruby Kaigi 2019]
• Flow-sensitive analysis[EuRuKo 2019]
• Context-insensitive analysis[PPL 2019]
• Aid of escape analysis
• Cumbersome "untyped" type
• Meta-programming features
• etc, etc.
Practical issues
• Trade-off between precision and
performance [Nagoya Ruby Kaigi 2019]
• Tuple-like and sequential array
• Method-local container type
[Osaka Ruby Kaigi 2019]
• Diagnosis features[Ruby 3 Summit]
• Unreachable method analysis
• Limitation of byte code
• Super-rich Ruby features
• Too complex Ruby features
• etc, etc.
10

Agenda
•Type Profiler: Type Inference for Ruby 3
➔Demo
• Simple case
• Real-world program case
• Library case
•How to use
•Future Plan
11

Demo 1: ao.rb
•Simple case
• A 3D renderer (~300 LoC)
• Written by Hideki Miura
• Original version was
written by Syoyo Fujita
https://code.google.com/archive/p/aobench/
•Analysis time < 1 sec.
12

Demo 1: ao.rb
13
class Vec
attr_accessor x : Float
attr_accessor y : Float
attr_accessor z : Float
def initialize : (Float, Float, Float) -> Float
def vadd : (Vec) -> Vec
def vsub : (Vec) -> Vec
def vcross : (Vec) -> Vec
def vdot : (Vec) -> Float
def vlength : -> Float
def vnormalize : -> Vec
end
NEW! attr_accessor
Formerly, "def x=" and "def x"

Demo 1: ao.rb
14
class Sphere
attr_reader center : Vec
attr_reader radius : Float
def initialize : (Vec, Float) -> Float
def intersect : (Ray, Isect) -> Vec?
end
class Isect
attr_accessor t : Float
attr_accessor hit : bool
attr_accessor pl : Vec
attr_accessor n : Vec
def initialize : -> Vec
end
NEW! optional type
Formerly, "Vec | NilClass"
NEW! bool type
Formerly,
"TrueClass | FalseClass"

Demo 2: Goodcheck
• Real-world program case
• A customizable linter for Ruby (~2000 LoC)
• It has "hand-written" RBS
• Analysis time < 30 sec.
• Note: It requires many libraries which have no RBS
• activesupport, concurrent-ruby, cgi, optparse, etc.
• Type Profiler analyzed not only Goodcheck but also them
• In future, we expect they have own RBS
• Type Profiler can use RBS instead of the code itself
15

Manually reformatted to make comparison easy
Demo 2: Goodcheck
16
class Goodcheck::Trigger
attr_reader patterns :
Array[(Goodcheck::Pattern::Literal |
Goodcheck::Pattern::Regexp |
Goodcheck::Pattern::Token)?]
attr_reader globs : Array[Goodcheck::Glob?]
attr_reader passes : Array[Array[untyped]]
attr_reader fails : Array[Array[untyped]]
attr_reader negated : bool
...
end
class Goodcheck::Trigger
attr_reader patterns :
Array[pattern]
attr_reader globs: Array[Glob]
attr_reader passes: Array[String]
attr_reader fails: Array[String]
attr_reader negated: bool
...
end
type Goodcheck::pattern
= Pattern::Literal|Pattern::Regexp|Pattern::Token
RBS inferred by Type Profiler Hand-written by Soutaro
Wrong guess
Type alias
Not so bad?
Extra optionalRedundant namescope

Demo 2: Goodcheck
17
class Goodcheck::Pattern::Token
attr_reader source : untyped
attr_reader case_sensitive : true
attr_reader variables : {}
def initialize : (source: untyped,
variables: {},
case_sensitive: true) -> true
@regexp : Regexp
def regexp : -> Regexp
def test_variables : (untyped) -> bool
def self.expand :
(untyped, untyped, ?depth: Integer) -> Array[Regexp]
def self.regexp_for_type :
(name: untyped, type: :__, scanner: untyped) -> Regexp?
def self.compile_tokens :
(untyped,
{},
case_sensitive: true) -> Regexp
@@TYPES : {}
end
class Goodcheck::Pattern::Token
attr_reader source: String
attr_reader case_sensitive: bool
attr_reader variables: Hash[Symbol, VarPattern]
def initialize: (source: String,
variables: Hash[Symbol, VarPattern],
case_sensitive: bool) -> void
def regexp: -> ::Regexp
def self.expand:
(String, String, ?depth: Integer) -> Array[::Regexp]
def self.regexp_for_type:
(name: Symbol, type: Symbol, scanner: StringScanner) -> ::Regexp
def self.compile_tokens:
(String source,
Hash[Symbol, VarPattern] variables,
case_sensitive: bool) -> void
@@TYPES: Hash[Symbol, ^(String) -> ::Regexp]
end
RBS inferred by Type Profiler Hand-written by Soutaro
Too specfic
Failed to track
the elements
C lib constant
(Lack of RBS)
void is intended
I think
not so bad

Demo 3: diff-lcs
•Real-world library case
Famous algorithmic library
•Hand-written entry point
•Analysis time < 1 sec.
18
https://bestgems.org/
require_relative "lib/diff/lcs"
class T; end
Diff::LCS.diff([T.new]+[T.new], [T.new]+[T.new]) {}

Demo: diff-lcs
19
class Diff::LCS::Change
include Comparable
attr_reader action : String
attr_reader position : Integer
attr_reader element : (Array[T] | T)?
def self.valid_action? : (String) -> untyped
def initialize : (String, Integer, (Array[T] | T)?) -> nil
def inspect : -> String
...
def == : (untyped) -> bool
def <=> : (untyped) -> Integer?
def adding? : -> bool
def deleting? : -> bool
def unchanged? : -> bool
def changed? : -> bool
def finished_a? : -> bool
def finished_b? : -> bool
end
predicate methods

Agenda
•Demo
➔ How to use Type Profiler
• Planned experience
• Specific usage
•Future plan
20

Typical usage and experience (Plan)
1. Write an entry point program
(if needed)
2. Apply Type Profiler
3. Partially write RBS for
wrong-guessed methods
4. Re-apply Type Profiler
21
lib.rb
app.rb
Type
Profiler
lib.rbs
(may include
wrong guesses)
partial RBS
for difficult methods
lib.rbs
(final)
①
②
③
④
"Partial RBS specification" has been implemented

How to use TP specifically
Will be written until the RubyKaigi Takeout!
(hopefully)
https://github.com/mame/ruby-type-profiler
22
I have written the document

Agenda
•Demo
•How to use Type Profiler
➔Future plan
• Recent updates
• Future plan
• Conclusion
23

Recent updates
• Improve cosmetics (attr_*, optional, bool, …)
• Import Array and Hash methods from RBS
• Type variable
• Support Enumerable, Enumerator, and Struct
• Support global variables
• Improve flow-sensitive analysis
• Improve analysis performance
• Fix many, many bugs
24

Future plan
• Until Ruby 3: Make it possible for plain Ruby code
• Support partial RBS specification
• Write document and release a gem
• Continue to experiment, improve, etc, etc.
• After the release: Support Sinatra app, Rails app…
• Maybe need dedicated hard-coding for the frameworks
• Concern is almost a language extension
• ActiveRecord is super-meta feature
• Please go easy on 🙏
25

Acknowledgment
•Hideki Miura
•Matz, Akr, Ko1
•Soutaro Matsumoto
•Katsuhiro Ueno
•Eijiro Sumii
•Sorbet developers (Stripe, Shopify, etc.)
•Jeff Foster
26

Conclusion
•Type Profiler is crawling to you
• Inviting suggestions for the tool name!
• Ask me anything: @mametter (Twitter)
27

Type Profiler: Ambitious Type Inference for Ruby 3

Recommended

More Related Content

What's hot (20)

Similar to Type Profiler: Ambitious Type Inference for Ruby 3 (20)

More from mametter (20)

Recently uploaded (20)

Type Profiler: Ambitious Type Inference for Ruby 3