-
Notifications
You must be signed in to change notification settings - Fork 310
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make BoTorch test problems into ParamBasedTestProblems #2944
Conversation
This pull request was exported from Phabricator. Differential Revision: D63639190 |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2944 +/- ##
=======================================
Coverage 95.68% 95.68%
=======================================
Files 504 504
Lines 49533 49547 +14
=======================================
+ Hits 47395 47411 +16
+ Misses 2138 2136 -2 ☔ View full report in Codecov by Sentry. |
Summary: Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Differential Revision: D63639190
Summary: Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Differential Revision: D63639190
afe1ad4
to
fd4ae44
Compare
This pull request was exported from Phabricator. Differential Revision: D63639190 |
Summary: Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Differential Revision: D63639190
fd4ae44
to
62cde69
Compare
This pull request was exported from Phabricator. Differential Revision: D63639190 |
Summary: Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Reviewed By: Balandat Differential Revision: D63639190
62cde69
to
bf7b0b0
Compare
This pull request was exported from Phabricator. Differential Revision: D63639190 |
bf7b0b0
to
57ed1a9
Compare
Summary: Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Reviewed By: Balandat Differential Revision: D63639190
This pull request was exported from Phabricator. Differential Revision: D63639190 |
Summary: Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Reviewed By: Balandat Differential Revision: D63639190
Summary: Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Reviewed By: Balandat Differential Revision: D63639190
Summary: Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Reviewed By: Balandat Differential Revision: D63639190
57ed1a9
to
b501d0c
Compare
This pull request was exported from Phabricator. Differential Revision: D63639190 |
Summary: Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Reviewed By: Balandat Differential Revision: D63639190
Summary: Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Reviewed By: Balandat Differential Revision: D63639190
Summary: Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Reviewed By: Balandat Differential Revision: D63639190
Summary: Pull Request resolved: facebook#2944 Context: In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem. * BoTorchTestProblemRunner has a BoTorch problem * ParamBasedTestProblemRunner has a ParamBasedTestProblem * SurrogateRunner has a SurrogateProblem * `BoTorchTestProblemRunner` and `ParamBasedTestProblemRunner` subclass `BenchmarkRunner` * `SurrogateRunner` subclasses `BenchmarkRunner` This diff will enable going from 5 runners to 3, wrapping BoTorch problems with `ParamBasedTestProblem`. This will allow for combining `BoTorchTestProblemRunner`, `ParamBasedTestProblemRunner`, and `SyntheticProblemRunner`. This PR: * Introduces `BoTorchTestProblem`, which is a `ParamBasedTestProblem` and thus can be used with `ParamBasedTestProblemRunner`. It localizes tensor-related logic, including `modified_bounds`, into the BoTorch problem so it doesn't need to be handled by the runner. * Gets rid of `SyntheticProblemRunner`, merging it with `ParamBasedTestProblemRunner`. We may want to rename this class after more consolidation. * Makes `BoTorchTestProblemRunner` a do-nothing subclass of `ParamBasedTestProblemRunner` Differential Revision: D63639190 Reviewed By: Balandat
This pull request has been merged in 8dcb808. |
Summary:
Context:
In the benchmarks, we currently have a zoo of five runners designed to handle different ways of generating data. This will make it challenging to support other changes to runners, such as running trials asynchronously. This diff is a step towards consolidating into one Runner, abstracting away the data-generating process into a test problem.
BoTorchTestProblemRunner
andParamBasedTestProblemRunner
subclassBenchmarkRunner
SurrogateRunner
subclassesBenchmarkRunner
This diff will enable going from 5 runners to 3, wrapping BoTorch problems with
ParamBasedTestProblem
. This will allow for combiningBoTorchTestProblemRunner
,ParamBasedTestProblemRunner
, andSyntheticProblemRunner
.This PR:
BoTorchTestProblem
, which is aParamBasedTestProblem
and thus can be used withParamBasedTestProblemRunner
. It localizes tensor-related logic, includingmodified_bounds
, into the BoTorch problem so it doesn't need to be handled by the runner.SyntheticProblemRunner
, merging it withParamBasedTestProblemRunner
. We may want to rename this class after more consolidation.BoTorchTestProblemRunner
a do-nothing subclass ofParamBasedTestProblemRunner
Differential Revision: D63639190