Try to cross-compile Scio-core for Scala 3 #4638
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Adjust the build.sbt, add an additional option for checking S3 macros * Remove macro methods from classes and move them to separate modules adding them as mix-ins (e.g. the fallback method from Coder and safe from To) * adjust the Magnolia usage to accomodate the new Scala 3 API * some macro were reimplemented, other are still unimplemented (like To.safe) * references to static annotation macro were removed for now, until the experimental support for them will be added * redesign the usage of ClassTag in ScioContext for Scala 3 (now we generate ClassTag[Nothing] ourselves, if necessary) * some Scala 3 macros were moved to scio-core, for typechecking reasons * other minor changes, like added type annotations
This was done to avoid hacks in build.sbt, and to keep the scio-avro dependency for Scala 2 scio-test, while removing it for Scala 3. scio-avro was not yet migrated to SCala 3, as it has some macro dependencies, that are still waiting for the macro static annotation support.
* Since the module depends on scio-avro (not yet being compiled for Scala 3), some test suites that depend on it were moved to a scala-2 only module. * Some individual tests that were failing to compile were commented out for now - most of them test magnolia-based Coder derivation, which is only implemented for case classes for now
|
Thanks for the draft PR @jchyb ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is meant to explore the possibility of making scio cross compile for Scala-3. For now the
scio-coreand a large part ofscio-testdo compile, with some of that compiled scio-test failing (more on that later).Missing dependencies
There are some missing dependencies, most notably from the twitter stack:
com.twitter%%algebird-corecom.twitter%%chillcom.twitter%%chill-algebirdNone of them are too problematic, requiring only the use of
CrossVersion.for3Use2_13. This does mean that there are some additional dependencies of which the 2.13 version has to be resolved, as otherwise we would get classpath conflicts (same thing will have to be done for any downstream projects).Missing shapeless 2 for Scala 3 is not a major issue, as it only leads to 2 changes:
Various
Magnolifymodules are used inscio-testandscio-avro.Magnolifyappears to already have a similar draft PR, where they are waiting for a solution of a few pending issues, including some we independently need solved for here as well. For now magnolify modules and code that depends on it was moved to a scala-2 only module.Macros
Some macros were rewritten, with the missing ones having a suitable comment near them. Most of the problematic (for now) macros stem from the use of Magnolia.
While
Magnoliadoes have a Scala 3 version, it has a fundamentally different API and implementation from the Scala 2Magnolia, for now using Mirrors instead of macros. This means that not all previously supported types will work, including AnyVal, or any other non case class types. The good news is that an alternative implementation of Magnolia was being investigated (softwaremill/magnolia#321), where it could be possible to support all those types. Absolute worst case scenario - we can fork that.Another macro that remains unported for now is the
To.safemethod. It looks doable, but we would have to construct schemas manually, based on the passed types instead of using eval like before. It should be possible, but I did not have the time to take care of this myself yet, especially considering it is relatively non critical to compiling and running the core module and tests.Any macro annotations are being commented-out / ignored for now, until they will be added to the language.
Testing
Because of the previously mentioned missing magnolify dependencies, some test suites that relied on them were moved to a separate Scala-2 only module. Additionally, there were some individual tests that did not compile either, mostly because of the aforementioned lack of support of Scala 3 Magnolia for non case class types - those tests were simply commented out for now.
I haven't had the time to properly setup the environment and analyze the test outputs, but for now many more tests fail on Scala 3 than on Scala 2, with the common error message being something like: