This year's Scala eXchange conference in London was opened by Martin Odersky's keynote who talked about his recent efforts to overcome binary compatibility problems that has been around in Scala ever since it became popular.
The problem in a nutshell is that having to deal with binary compatibility makes harder to upgrade and thus slows down innovation and improvement. You either break your client's builds or freeze and stop improving, but there has to be a better solution.
Through a handful of examples he showed how binary compatibility is different from source compatibility and how they're independent from each other. Source can remain compatible while binary compatibility breaks when upgrading a library, and vice versa. Compiler optimizations and bug fixes can affect binary compatibility too, and all items of the compiler pipeline may have effects on it. It's not only a problem for the Scala library but also for Shapeless, Akka or even your own old legacy lib that you can't rebuild because the people who wrote it are no longer there.
So far apart from MiMa, a tool that helps detecting binary incompatibilities, and conservative release policies there weren't many tools to cope with it. What's even worse that 3rd parties rarely enforce such policies, or if they do innovation is stifled.
Solutions what others do – like innovation happening on the JVM level and freezing the libraries as soon as they appear (in Java), allowing multiple versions of a library in an application (in OSGI) or reproducibly building Scala from source – have their own problems.
Instead of bytecode or source, Odersky is proposing to use typed trees as an interchange format which is more robust than source and more stable than the JVM bytecode. In practice this means splitting the compiler pipeline into two stages where the first one would produce the typed tree for each class we have, and that would be stored along with the JVM bytecode in the class files and a hash of the typed tree.
Other benefits of storing typed trees are:
- Optimization: interprocedural analysis allows simple inlining without binary compatibility issues
- Program analysis: ideal for context-dependent program analysis (FindBugs) and instrumentation
- Portability: allows retargeting to different backends (JVM, JS, LLVM)
It's important to note that for now this is only a proposal and Odersky's keynote was a call-out for the Scala community to comment on the subject, there is no plan to put it on the Scala compiler roadmap just yet.