BoTorch optimisation using GPUs: tuning the optimisation based on hardware

[sambitmishra98]

I am currently studying how 5 parameters can be optimised within 100-200 optimisation iterations using Knowledge Gradient (qKG), Expected Improvement (qNEI) methods in BoTorch package using an NVIDIA GPU. I can see multiple tunable options in BoTorch, like num_samples and num_restarts (and fantasies for qKG). I also have an option of increasing the number of optimisable parameters from 5 to 10 parameters. What I am unsure of is how to arrive at a suitable values for all of the above. The following are some issues I am going into:

• Although there is a general increasing trend in wall-time taken for candidates to be suggested by the optimiser, I noticed the deviation in this wall-time varies quite a lot from 20 seconds to 2 minutes for qKG method. I am guessing all other methods will have the same issue. So I am not sure of a more rigorous way to quantify the increase in wall-time.
• Given that I know the memory in a GPU, I am not sure how to look into the memory usage by BoTorch for the calculations. This is particularly important to me because I am parallely running a CFD simulation which uses the same GPU. I want to know if both the CFD simulation and the BoTorch optimisation are given sufficient memory within the GPU.
• I am currently increasing num_samples, num_restarts, fantasies and number of optimisable parameters in a conservative trial-and-error way untill optimisation wall-time feels too long (more than 3 minutes). I want to know if memory usage and computational work increases with each increasing data-point.

So in gist, I want to be able to run a series of rigorous tests to quantify the increasing memory usage and computational work in an NVIDIA GPU for qKG and qEI methods as I increase the number of iterations from 1 to 200.

[saitcakmak]

Hi @sambitmishra98. Sorry for the late response.

Although there is a general increasing trend in wall-time taken for candidates to be suggested by the optimiser, I noticed the deviation in this wall-time varies quite a lot from 20 seconds to 2 minutes for qKG method. I am guessing all other methods will have the same issue.

The deviations beyond the expected increasing pattern are mostly due to variations in when the optimizer terminates. By default, we use scipy.minimize to optimize the acquisition functions. It determines when to terminate optimization based on 2 main criteria: maxiter/maxfun (# of iterations of the optimizer / function evaluations) & ftol/gtol (convergence tolerance for function / gradient values). You can pass these options as part of the options input to optimize_acqf. To reduce the variation in the wall-time, you can reduce the tolerances (so that you guarantee terminating based on # of iterations / evaluations) and set maxfun to some value that works for you. We typically use maxiter (default value is 1500) rather than maxfun to limit optimization effort, but it is a bit less precise since LBFGS can do up to 20 (based on when the line search terminates) function evaluations within each iteration.

Given that I know the memory in a GPU, I am not sure how to look into the memory usage by BoTorch for the calculations. This is particularly important to me because I am parallely running a CFD simulation which uses the same GPU. I want to know if both the CFD simulation and the BoTorch optimisation are given sufficient memory within the GPU.

BoTorch uses the GPU through PyTorch, so you can use PyTorch utilities to check the memory usage. You can expect roughly quadratic (based on # of training inputs) increase in memory usage between iterations. To reduce the memory usage, you can limit the batch evaluation within the optimizer by passing in batch_limit & init_batch_limit as part of options to optimize_acqf. You can also change the sampler used by the acquisition function and reduce the # of samples produced (also # of fantasies for qKG), which would also reduce the memory usage.

I am currently increasing num_samples, num_restarts, fantasies and number of optimisable parameters in a conservative trial-and-error way untill optimisation wall-time feels too long (more than 3 minutes). I want to know if memory usage and computational work increases with each increasing data-point.

This is partly answered in the previous point. Increasing each of these will lead to additional computation, and possibly additional memory requirements. For num_samples and num_restarts the memory usage can be controlled using the batch_limit mentioned above. For num_fantasies, you should expect a linear increase in the memory usage.

[sambitmishra98]

Thank you @saitcakmak and @Balandat for helping me out every time with such questions.

Currently my motive is to constrain the time taken for BoTorch to suggest a candidate to lesser than 0.1% of the cost function evaluation in real-time. It did occur to me that I could get a better estimate by running the simulation multiple times. I want to make use of the wall-time of optimisation and memory usage at a given instant to determine the way to limit them both.

Based on the above it seems like I could control this better by passing in a lower value for maxiter to scipy.minimize when I see that the candidate generation time crosses this 0.1% threshold.

As for ensuring that we do not have out-of-memory issues that I am facing, if the memory usage is exceeding a certain threshold (say 5GB), I am thinking of decreasing num-fantasies in real-time to keep memory usage under check.

Ultimately, since I am running both my cost function evaluation and optimisation in the same GPU, I would want to be able to limit memory usage and wall-time for candidate generation within a GPU in real-time based on time taken and memory used by the cost function evaluation performed in the same GPU.

Are the above two strategies reasonable to better control the compute time and memory usage? To handle the two issues above I have been replacing old low-quality data which has high errors with new high-quality data with lower errors. But based on our previous discussions #1387, deleting data is probably not a good idea.

[Balandat]

if the memory usage is exceeding a certain threshold (say 5GB), I am thinking of decreasing num-fantasies in real-time to keep memory usage under check.

That seems like a reasonable approach to me.

Are you actually doing the generation and cost function evaluation in parallel here? That is are you running this either fully sequentially or in a synchronized batched fashion where you finish evaluating all configs before generating the next? If that's the case and the function evaluation is the most costly setup then could you maybe just deallocate all the botorch objects that take up GPU memory during function eval so you don't have to worry about GPU memory?

[saitcakmak]

I would want to be able to limit memory usage and wall-time for candidate generation

One additional tool for limiting the wall-time is to pass in timeout_sec to optimize_acqf. This is an alternative to maxiter/maxfun that was recently added. It stops the optimization when the wall-time exceeds timeout_sec.

[Balandat]

One additional tool for limiting the wall-time is to pass in timeout_sec to optimize_acqf

Right - I should have suggested this since I was the one to add this...

[sambitmishra98]

I am currently generating a single candidate based on past tested candidates and am testing the candidate. Then I obtain the cost for the candidate and add the now tested candidate to the dataset and generate the next candidate. So cost evaluation and optimisation are done sequentially. They are both run within a single python code, so I would think their memory allocation is all done together.

Cost function evaluation ranges quite a lot depending on what I am trying to optimize. For a certain problem optimisation cost function evaluation takes about 2 minutes. The BoTorch optimisation that I do catches up to time taken for cost function evaluation in 100 iterations/candidates.

To keep such problems aside, I have set the acquisition function and other variables such that I don't go beyond 1 minute of optimisation wall time. Same goes for the memory issues I was facing.

I dont think deallocating the memory would help me much.

I think timeout_sec is what I wanted. I can limit this as a factor of cost function evaluation time.

As for the memory issue, I think I found a way to find how much memory torch is using in a GPU and accordingly control the acquisition function via keyword arguments. I will start with tweaking num_fantasies for this.

Let me know if there is anything else I could try out.

@saitcakmak @Balandat thanks a lot for guiding me through this!

[sambitmishra98]

I updated my BoTorch version and set timeout_sec as follows:

        X_cand, _ = optimize_acqf(acq_function = _acquisition_function, q = 1,
            bounds = self.normalise.transform(self._bnds),
            num_restarts = num_restarts,raw_samples = raw_samples,
            timeout_sec = 10000,
            )

It didn't seem to work as well for whatever number I set there. I get the following error with similar time-out of order of 10^-3.

[OptimizationWarning('Optimization timed out after 0.004253573017194867 seconds.')]

[saitcakmak]

10_000 seconds is pretty large, so it wouldn't surprise me to see that it didn't have any effect.

[OptimizationWarning('Optimization timed out after 0.004253573017194867 seconds.')]

This warning is just a way to let the user know that the optimization was terminated due to time out and not by to any other convergence criteria. The time out just tells the optimizer to stop optimizing when the time limit is reached. It does not provide any convergence guarantees, so this warning is just a way to let the user know how the optimization was terminated.

[sambitmishra98]

I tried values 10,60,120,200 as well. None of them made any difference.

I expected the warning to show after around the set number of seconds has elapsed.

[saitcakmak]

According to the warning above, it only took 0.004253573017194867 seconds. to optimize the acqf. AFAIK, this does not include the time spent on generating the initial candidates. It only counts the time spent within the scipy.minimize call (cc @Balandat). It is likely that the longer runtime is due to initial candidate generation, which can take a while with KG and is not controlled by these settings.

The initial candidate generation for KG also solves an optimization problem under the hood, so it's more costly than others.

[Balandat]

Yes that is correct this currently just considers the time spent in scipy.minimize (cc @esantorella with who we had discussed this in the past).

Do you have a repro for this that we can run to understand the relative runtimes of the initial candidate generation and the optimization?

[sambitmishra98]

I am still working on the code for my research. I will share the details as soon as I get the paper out. For now I will just set the parameters so it optimisation takes more than a minute with as much as 1000 candidates.