Upgrade to Kryo 4.0.0

[chermenin]

No description provided.

[johnynek]

why remove the old one? This will break people, no?

[chermenin]

Well, keep ClassManifestSerializer back :)

[jodersky]

+1 to Kryo 4.0.0: it seems that #253 is blocked mainly by a dependency on an unreleased version of Kryo, namely 3.1. Upgrading to 4.0.0, will unblock further work on supporting Scala 2.12 (which has just had its first RC published) and help Spark move to 2.12 as well!

[Tolsi]

+1

[alexeykiselev]

+1

[johnynek]

I'll look at merging this Monday and doing a release.

Sorry for the delay.

[magro]

@johnynek You're most probably aware of this: in kryo 4 the compatibility issue with the biggest impact IMO is probably caused by FieldSerializers changed generics handling. You could restore the former default with FieldSerializerConfig().setOptimizedGenerics(true); or make it very easy for users to set this (not sure what that means in the different contexts).

[johnynek]

@magro I didn't actually know that. If I make that change as the default setting (which I can do by changing the default scala kryo instantiator here: https://github.com/twitter/chill/blob/develop/chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala#L56 ), do you expect that serialization of serialized objects will be compatible with 2.21, or just 3.x (were there any changes)?

To be honest, after some folks made the mistake of persisting kryo objects, the anxiety about upgrades adds cost to the point that these folks don't even want to try the upgrade.

[magro]

@johnynek I'd expect that serialization then is compatible with 2.21 if standard Input/Output is used. In kryo 3 the Unsafe-based IO was changed (as said in Changes 2.24.0 - 3.0.0 / Compatibility).
That's what I can say to the best of my knowledge, of course any upgrade of a serialization library deserves careful testing.

[isnotinvain]

Just wanted to chime in here -- I think we need to discuss if we want to upgrade to kryo 4 just yet. I think we will need kryo3 support at least for quite some time. That or maybe consider shading kryo and removing it form chill's APIs or something like that.

But chill is used heavily in scalding / summingbird, and summingbird uses Storm's API, which is only just now using kryo3 as I understand it.