This repo contains the code that can reproduce experimental results in Cost-Aware Gradient Entropy Search for Efficient Local Multi-Fidelity Bayesian Optimization. The paper has been accepted to the 63rd IEEE Conference on Decision and Control (CDC)
cartpole : Please replace the original cartpole.py file inside the gymnasium package (~/gymnasium/envs/classic_control/cartpole.py) with this file to enable the changes of step time (tau).
src\environment_api: Interface for interactions with reinforcement learning environments of OpenAI Gym.
RL_function_new: Function that takes policy parameters and qualatative variable as input and return the reward for the RL problem.
test_function: Test problem studied in this paper, including multi-information sources Rosenbroc.
We provide the code scripts for executing different BO algorithm including Vanillo BO, GIBO [2], and CAGES. Augmented random search algorithm (ARS) [3] is also provided.
Run the following command to fit a MixedSingleTask model with specified test function and training/testing data. Example contains the multi-information sources (1 qualatative variable with 2 level) 12D Rosenbrock function trained with 160 data (80 for each information source). A parity plot will be generated.
python Parity_MixedSingleTask_Rosenbrock.pyUser should get the following figure by executing the above command.
Run the following command to execute vanilla BO with log expected imrpovement. User can specify which problem (Rosenbrock, or Cartpole) to be optimized in line 32-33.
Two numpy files (EI_cost.np/EI_reward.np) will be saved in user's local directory including the accumulated cost and best found value per iteration.
python Vanilla_BO.pyRun this following command to execute GIBO or ARS algorithm. User can specify which problem (Rosenbrock, or Cartpole) to be optimized in line 38-39 and specify GIBO or ARS in line 42.
Two numpy files (GIBO_cost.np/GIBO_reward.np) will be saved in user's local directory including the accumulated cost and best found value per iteration.
cd GIBO
python local_GIBO_exe.pyRun the CAGES notebook to execute CAGES for the multi-information Rosenbrock test problem. Two numpy files (Rosenbrock_cost_CAGES.np/Rosenbrock_reward_CAGES.np) will be saved in user's local directory including the accumulated cost and best found value per iteration.
Run the following command to execute CAGES for the multi-information Cartpole RL control problem.
Two numpy files (cartpole_cost_CAGES.np/cartpole_reward_CAGES.np) will be saved in user's local directory including the accumulated cost and best found value per iteration.
python CAGES_RL.pyRun the CAGES-LVGP to execute CAGES with Latent variable Gaussian process (LVGP) [3] on the multi-information Rosenbrock test problem.
Please run the following command to load saved numpy files and generate plots. User may need to modify the path to load the file.
cd Plotting
python Plotting_new.pyHere are example plots for the Rosenbrock and Cartpole problem that user can reproduce based on the np files under ~\Results:
[1] Müller, S., von Rohr, A., & Trimpe, S. (2021). Local policy search with Bayesian optimization. Advances in Neural Information Processing Systems, 34, 20708-20720.
[2] Mania, H., Guy, A., & Recht, B. (2018). Simple random search of static linear policies is competitive for reinforcement learning. Advances in neural information processing systems, 31.
[3] Zhang, Y., Tao, S., Chen, W., & Apley, D. W. (2020). A latent variable approach to Gaussian process modeling with qualitative and quantitative factors. Technometrics, 62(3), 291-302.