![Introduction Sources of Incompatibility Conclusion Example Scala vs. Java
Example
Assume Analyzer is part of a library we produce. We decide that
its API has to evolve as follows:
class Analyzer { // old version class Analyzer { // new version
def analyze(issues: HashMap[ , ]) {...} def analyze(issues: Map[ , ]) {...}
} }
Further, assume the next expression was compiled against the old
library
new Analyzer().analyze(new HashMap[Any,Any])
Would the compiled code work if run against the new library?
The answer lies in the bytecode...
Mirco Dotta Managing Binary Compatibility in Scala](https://image.slidesharecdn.com/main-111014093020-phpapp02/85/Managing-Binary-Compatibility-in-Scala-Scala-Lift-Off-2011-3-320.jpg)
![Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Type Inferencer: Member Signature
Does the following evolution break binary compatibility?
class TextAnalyzer { // old version class TextAnalyzer { // new version
def analyze(text: String) = { def analyze(text: String) = {
val issues = Map[String,Any]() val issues = new HashMap[String,Any]
// ... // ...
issues issues
}} }}
Question
What is the inferred return type of analyze?
Let’s compare the two methods’ signature.
class TextAnalyzer { // old version class TextAnalyzer { // new version
public scala.collection.immutable.Map public scala.collection.immutable.HashMap
analyze(java.lang.String); analyze(java.lang.String);
} }
Mirco Dotta Managing Binary Compatibility in Scala](https://image.slidesharecdn.com/main-111014093020-phpapp02/85/Managing-Binary-Compatibility-in-Scala-Scala-Lift-Off-2011-6-320.jpg)
![Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Type Inferencer: Member Signature (2)
Question
Can we prevent the method’s signature change?
That’s easy! The method’s return type has to be explicitly
declared:
class TextAnalyzer { // bytecode compatible new version
def analyze(text: String): Map[String,Any] = {
val issues = new HashMap[String,Any]
// ...
issues
}}
Take Home Message
Always declare the member’s type. If you don’t, you may
inadvertently change the members’ signature.
Mirco Dotta Managing Binary Compatibility in Scala](https://image.slidesharecdn.com/main-111014093020-phpapp02/85/Managing-Binary-Compatibility-in-Scala-Scala-Lift-Off-2011-7-320.jpg)
![Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Type Parameters & Type Erasure
Assume NodeImpl <: Node. Does the following evolution break
binary compatibility?
class Tree[T <: NodeImpl] { // old version class Tree[T <: Node] { // new version
def contains(e: T): Boolean = {...} def contains(e: T): Boolean = {...}
} }
Type parameters are erased during compilation, but the erasure of
Tree.contains is different for the two declarations.
boolean contains(NodeImpl) {...} boolean contains(Node) {...}
Remember that the JVM looks up methods based on the textual
representation of the signature, along with the method name.
Take Home Message
Type parameters may change the members’ signature and hence
break binary compatibility.
Mirco Dotta Managing Binary Compatibility in Scala](https://image.slidesharecdn.com/main-111014093020-phpapp02/85/Managing-Binary-Compatibility-in-Scala-Scala-Lift-Off-2011-8-320.jpg)
![Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Lazy to Eager
Let’s take a simple example and look at the generated bytecode:
class ClassAnalyzer { // old version class ClassAnalyzer { // new version
lazy val superclasses: List[Class] = { val superclasses: List[Class] = {
// ... // ...
} }
} }
And here is the bytecode:
class ClassAnalyzer { // old version
class ClassAnalyzer { // new version
private List[Class] superclasses;
private final List[Class] superclasses;
volatile int bitmap$0
public int superclasses();
public int superclasses();
// ...
// ...
}
}
Take Home Message
Transforming a lazy value into an eager one does preserve
compatibility with pre-existing binaries.
Mirco Dotta Managing Binary Compatibility in Scala](https://image.slidesharecdn.com/main-111014093020-phpapp02/85/Managing-Binary-Compatibility-in-Scala-Scala-Lift-Off-2011-15-320.jpg)
Managing Binary Compatibility in Scala (Scala Lift Off 2011)
- 1. Managing Binary Compatibility in Scala Mirco Dotta Typesafe October 13, 2011 Mirco Dotta Managing Binary Compatibility in Scala
- 2. Introduction Sources of Incompatibility Conclusion Outline Introduction Example Scala vs. Java Sources of Incompatibility Type Inferencer Type Parameters Trait Lazy Fields Conclusion Mirco Dotta Managing Binary Compatibility in Scala
- 3. Introduction Sources of Incompatibility Conclusion Example Scala vs. Java Example Assume Analyzer is part of a library we produce. We decide that its API has to evolve as follows: class Analyzer { // old version class Analyzer { // new version def analyze(issues: HashMap[ , ]) {...} def analyze(issues: Map[ , ]) {...} } } Further, assume the next expression was compiled against the old library new Analyzer().analyze(new HashMap[Any,Any]) Would the compiled code work if run against the new library? The answer lies in the bytecode... Mirco Dotta Managing Binary Compatibility in Scala
- 4. Introduction Sources of Incompatibility Conclusion Example Scala vs. Java Example: Bytecode Let’s have a look at the method signature for the two versions: class Analyzer { // old version class Analyzer { // new version analyze(Lscala/collection/immutable/HashMap);V analyze(Lscala/collection/immutable/Map);V } } The expression compiled against the old library would look like: ... invokevirtual #9;// #9 == Analyzer.analyze:(Lscala/collection/immutable/HashMap;)V ⇒ The method’s name has been statically resolved at compile-time. Running it against the new library would result in the JVM throwing a NoSuchMethodException. ⇒ The evolution of class Analyzer breaks compatibility with pre-existing binaries. Mirco Dotta Managing Binary Compatibility in Scala
- 5. Introduction Sources of Incompatibility Conclusion Example Scala vs. Java Is Binary Compatibility a Scala issue? The short answer is No. The discussed example can be easily ported in Java or other languages targeting the JVM. Scala shares with Java many sources of binary incompatibility. But Scala offers many language features not available in Java: First-class functions. Type Inferencer. Multiple inheritance via mixin composition (i.e., traits). Lazy values. . . . Just to cite a few. ⇒ Scala code has new “unique” sources of binary incompatibility. Mirco Dotta Managing Binary Compatibility in Scala
- 6. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Type Inferencer: Member Signature Does the following evolution break binary compatibility? class TextAnalyzer { // old version class TextAnalyzer { // new version def analyze(text: String) = { def analyze(text: String) = { val issues = Map[String,Any]() val issues = new HashMap[String,Any] // ... // ... issues issues }} }} Question What is the inferred return type of analyze? Let’s compare the two methods’ signature. class TextAnalyzer { // old version class TextAnalyzer { // new version public scala.collection.immutable.Map public scala.collection.immutable.HashMap analyze(java.lang.String); analyze(java.lang.String); } } Mirco Dotta Managing Binary Compatibility in Scala
- 7. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Type Inferencer: Member Signature (2) Question Can we prevent the method’s signature change? That’s easy! The method’s return type has to be explicitly declared: class TextAnalyzer { // bytecode compatible new version def analyze(text: String): Map[String,Any] = { val issues = new HashMap[String,Any] // ... issues }} Take Home Message Always declare the member’s type. If you don’t, you may inadvertently change the members’ signature. Mirco Dotta Managing Binary Compatibility in Scala
- 8. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Type Parameters & Type Erasure Assume NodeImpl <: Node. Does the following evolution break binary compatibility? class Tree[T <: NodeImpl] { // old version class Tree[T <: Node] { // new version def contains(e: T): Boolean = {...} def contains(e: T): Boolean = {...} } } Type parameters are erased during compilation, but the erasure of Tree.contains is different for the two declarations. boolean contains(NodeImpl) {...} boolean contains(Node) {...} Remember that the JVM looks up methods based on the textual representation of the signature, along with the method name. Take Home Message Type parameters may change the members’ signature and hence break binary compatibility. Mirco Dotta Managing Binary Compatibility in Scala
- 9. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Trait Compilation Traits are a powerful language construct that enables multiple-inheritance on top of a runtime – the JVM – that does not natively support it. Understanding how traits are compiled is crucial if you need to ensure release-to-release binary compatibility. So, how does the Scala compiler generate the bytecode of a trait? There are two key elements: A trait is compiled into an interface plus an abstract class containing only static methods. “Forwarder” methods are injected in classes inheriting traits. Mirco Dotta Managing Binary Compatibility in Scala
- 10. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Trait Compilation Explained An example will help visualize how traits get compiled: // declared in a library trait TypeAnalyzer { def analyze(prog: Program) { // client code // the trait’s method impl code class TypingPhase extends TypeAnalyzer } } The following is the (pseudo-)bytecode generated by scalac: interface TypeAnalyzer { class TypingPhase implements TraitAnalyzer { void analyze(Program prog); // forwarder method injected by scalac } void analyze(Program prog) { abstract class TypeAnalyzer$class { // delegates to implementation static void analyze(TypeAnalyzer $this, TypeAnalyzer$class.analyze(this,prog) Program prog) { } // the trait’s method impl code } } } Mirco Dotta Managing Binary Compatibility in Scala
- 11. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Trait: Adding a concrete method Question Can we add a member in a trait without breaking compatibility with pre-existing binaries? Mirco Dotta Managing Binary Compatibility in Scala
- 12. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Trait: Adding a concrete method trait TypeAnalyzer { // new version // compiled against the old version def analyze(prog: Program) {...} class TypingPhase implements TraitAnalyzer { def analyze(clazz: ClassInfo) {...} // forwarder method injected by scalac } void analyze(Program prog) { // delegates to implementation TypeAnalyzer$class.analyze(this,prog) //TypeAnalyzer trait compiled } interface TypeAnalyzer { // missing concrete implementation! void analyze(Program prog); ??analyze(ClassInfo clazz)?? void analyze(ClassInfo clazz); } } abstract class TypeAnalyzer$class { static void analyze(TypeAnalyzer $this, Take Home Message Program prog) {...} static void analyze(TypeAnalyzer $this, Adding a concrete method in a trait ClassInfo clazz) {...} breaks binary compatibility. } Mirco Dotta Managing Binary Compatibility in Scala
- 13. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Lazy values decompiled Lazy values are a Scala language feature that has no correspondent in the JVM. ⇒ Their semantic has to be encoded in terms of the available JVM’s primitives. A lazy val is encoded into a private field and a getter plus a bitmap. The bitmap is used by the getter for ensuring that the private field gets initialized only once. The bitmap is of type Int, meaning that a maximum of 32 lazy value can be correctly handled by it. The bitmap is shared with subclasses, so that a new bitmap field is created only if strictly necessary. Mirco Dotta Managing Binary Compatibility in Scala
- 14. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Lazy to Eager Question Can we transform a lazy field into an eager one? Mirco Dotta Managing Binary Compatibility in Scala
- 15. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Lazy to Eager Let’s take a simple example and look at the generated bytecode: class ClassAnalyzer { // old version class ClassAnalyzer { // new version lazy val superclasses: List[Class] = { val superclasses: List[Class] = { // ... // ... } } } } And here is the bytecode: class ClassAnalyzer { // old version class ClassAnalyzer { // new version private List[Class] superclasses; private final List[Class] superclasses; volatile int bitmap$0 public int superclasses(); public int superclasses(); // ... // ... } } Take Home Message Transforming a lazy value into an eager one does preserve compatibility with pre-existing binaries. Mirco Dotta Managing Binary Compatibility in Scala
- 16. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Eager to Lazy Question Can we transform an eager field into a lazy one? Mirco Dotta Managing Binary Compatibility in Scala
- 17. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields Eager to Lazy The previous example demonstrates that lazy and eager fields are syntactically equivalent, from a client perspective. Does that mean that the transformation is safe? It depends. If you are in a closed world, then the transformation is safe (e.g., your class is final). Otherwise, you are taking a high risk on breaking the semantic of lazy, and no good can come out of that. Remember that the bitmap is shared between subclasses. Take Home Message Transforming an eager value into a lazy one may not preserve compatibility with pre-existing binaries. Mirco Dotta Managing Binary Compatibility in Scala
- 18. Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager Conclusion Ensuring release-to-release binary compatibility of Scala libraries is possible. Though, sometimes it can be difficult to tell if a change in the API of a class/trait will break pre-existing binaries. In the discussed examples we have seen that: Type inferencer may be at the root of changes in the member’s signature. Type parameters may also modify members’ signature. Traits are a sensible source of binary incompatibilities. Transforming an eager field into a lazy one can break semantic. It really looks like library’s maintainers’ life ain’t that easy... Mirco Dotta Managing Binary Compatibility in Scala
- 19. Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager Scala Migration Manager (MiMa) A few months ago we released the Scala Migration Manager! It’s free!! It can tell you, library maintainers, if your next release is binary compatible with the current one. It can tell you, libraries users, if two releases of a library are binary compatible. MiMa can collect and report all sources of “syntactic” binary incompatibilities between two releases of a same library. “Syntactic” means NO LinkageError (e.g., NoSuchMethodException) will ever be thrown at runtime. Now, it’s time for a short demo! Mirco Dotta Managing Binary Compatibility in Scala
- 20. Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager Future Work Reporting binary incompatibilities is only half of the story. We are working on a “companion” tool that will help you migrate binary incompatibilities. For the reporting there are many ideas spinning around. Your feedback will help us decide what brings you immediate value One that I believe is useful: Maven/Sbt integration. Mirco Dotta Managing Binary Compatibility in Scala
- 21. Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager Scala Migration Manager Visit http://typesafe.com/technology/migration-manager More information about the Migration Manager Download it and try it out, it’s free! We want to hear back from you. Success stories Request new features Report bugs Want to know more, make sure to get in touch! email: [email protected], twitter: @mircodotta Mirco Dotta Managing Binary Compatibility in Scala
- 22. Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager One last thing I didn’t mention... We will make the sources public! Mirco Dotta Managing Binary Compatibility in Scala