Reducing compilation times & reducing dependency on parity repo

[lght]

ethcore is at the heart of both Parity and Solaris. It is also the largest single dependency required by Solaris.

This issue is to track discussion and strategy for minimizing ethcore into a manageable, modular crate. The goal of minimizing / splitting apart ethcore is to decrease compilation times, and enable other tools to select which ethcore components they would like to use.

After discussion with @tomusdrw and @kirushik, there are a few different strategies to pursue:

• mark default features in ethcore
  • use --no-default when building Solaris
  • cherry-pick needed components with feature flags
• split ethcore modules into sub-crates
• create a new, minimal ethcore-light with only necessary features
  • starting from "scratch"
  • use a sub-crate architecture

The above ideas are starting points, and subject to change. Personally, would prefer the last option of creating a new ethcore-light, having the long-term vision of replacing the current ethcore with a sub-crate architecture.

Refactoring such a critical piece of Parity must obviously be done with care, and starting a new ethcore-light greatly reduces the impact to / from other Parity components while it's under development.

[kirushik]

@0x7CFE is doing an amazing work on the ethcore refactoring in openethereum/parity-ethereum#7038
We need to at least coordinate feature splitting with him (and in the best case he will also help us with determining the scope of those features, or even reduce the amount of external dependencies ethcore has at all).

[lght]

That work is great, will be happy to coordinate with @0x7CFE on strategy. ~~~Think it may still be safe to get started on~~~ Will hold for now on creating a new / splitting off unnecessary components from ethcore.

[rphmeier]

FYI, ethcore-light already exists and stores the light client database schema, import logic, and network code.

[tomusdrw]

I think the real question is what we want to achieve, we have couple of goals, but with a completely different scopes and timelines. Main pain points currently: long compilation, large repo to fetch, no way to release to crates.io.

Long term solution (perfect world)

Solaris is using small crates referenced from crates.io - fast compilation speed, no dependency on Parity, Solaris can be released to crates.io.

Mid term solution (soonish)

Solaris is using medium crates referenced from Parity repository or smaller repositories. - fast compilation speed, no need to download the whole Parity repo while building.

Short term solution (something that we should aim for)

Improving compilation time, but getting rid of unnecessary stuff in ethcore crate:

• rocksdb
• custom build command
• any code generation
• any long type inference (will require debugging)

[lght]

ethcore-light already exists and stores the light client database schema, import logic, and network code

@rphmeier, I did not realize this was a thing. Perhaps a new name entirely for the components we need would be better than a confusing modification of ethcore-something?

Improving compilation time, but getting rid of unnecessary stuff in ethcore crate

@tomusdrw, compilation time was my main pain point. Will start working on stripping out the unnecessary parts of ethcore. The mid-term solution should be possible, and much easier, once @0x7CFE finishes the Client refactor.

Since ethcore-light is already close to what we want in the short term, will start there for the minimization effort. Working title of ethcore-mini.

[snd]

@tomusdrw is there something that comes to mind that would require little/medium work but should give us a big reduction in ethabi compilation time?

[tomusdrw]

@snd we are considering extracting some crate to a separate repo: paritytech/common, might be worth to include evm and test_client there as well, cc @dvdplm
Definitely a huge gain would be to get rid of rocksdb dependency (if not done already), the easiest way would be to have it feature-gated.