feat: initial multi-objective support #180
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds multi-objective support to the NePS backend, meaning that users can now return multiple objectives and the runtime can handle it properly. NOTE No optimizers other than random or grid-search can actually handle multi-objective at the moment, this is only such that the infrastructure of NePS can handle it.
To return multiple objectives:
Since we never concretely define what a user can return, there is now a very dedicated parsing function which is pretty exhaustive about it and comes with a lot of tests.
UserResult.parse(whatever_the_user_returns_from_evaluate_pipeline)
I also had to go through quite a bit of the summary related things to account for this. I also discovered CLI and plotting stuff was broken from before, and consequently, is also still broken during this change. This is a further thing that needs to be done.
We should consider if we really want to provide plotting code. It's usually quite custom per use-case, what's better is to provide examples on how to do it, but also just a way to export dataframes easily which people can interpret and plot as they like. This is now done by
full_frame, short_summary = neps.status(root_directory)
, wherefull_frame
contains all the information relating to trials, while short summary is more for printing purposes and is just a pd.Series.