This can be somewhat mitigated by AOT compilation. Here's a link to some experiments from using SubstrateVM to create a self-contained executable from sjsonnet: https://github.com/databricks/sjsonnet/issues/2
Netflix ICE has a few quirks but overall it's a pretty useful free tool.
Unfortunately, the project has been largely abandoned and the current version in the Netflix GitHub repository won't work out of the box for many companies due to lack of support for new AWS regions, instances, reservation types, and services, as well as showstopping bugs (such as https://github.com/Netflix/ice/issues/210). In order to effectively use ICE today most users will need to maintain their own patched fork.
The blog is referring to source- and binary-API compatibility guarantees that Spark provides.
IIRC, in order for DataFrames and Datasets to share a common parent class certain common methods' return types would need to change in a binary-incompatible way that would break DataFrame code which was compiled against older Spark versions.
Due to Spark's API compatibility and versioning policies, this type of compatibility break can only occur in major releases, so it can only be done in Spark 2.0.0 and not Spark 1.6.