Potential performance degradation comparing version 9.10.4067 with 9.11.4210.

[forceoffire]

Attachments

01_small.zip
01_small_ortools_v_10.zip
01_small_ortools_v_11.zip
02_medium.zip
02_medium_ortools_v_10.zip
02_medium_ortools_v_11.zip
03_large.zip
03_large_ortools_v_10.zip
03_large_ortools_v_11.zip
performance.zip

What version of OR-Tools and what language are you using?

Comparing version 9.10.4067 with 9.11.4210

Language: MiniZinc, but comparing after converting to FlatZinc.

Which solver are you using (e.g. CP-SAT, Routing Solver, GLOP, BOP, Gurobi)

CP-SAT

What operating system (Linux, Windows, ...) and version?

Windows 10, on 16 CPU machine with 64GB of RAM.

What did you do?

I constructed three instances of the same model (over three different time horisons) resulting in a small, medium and large model (all examples available in 01_small.zip, 02_medium.zip & 03_large.zip files attached). Apologies for the large fzn files, but the observed behaviour below is best illustrated using larger models. These three models were run against version 9.10.4067 and 9.11.4210, using the following typical instruction (see tests.bat):

c:\MiniZinc\or-tools_x64_VisualStudio2022_cpp_v9.10.4067\bin\fzn-cp-sat.exe 01_small.fzn -fz_logging=true --threads=8 --cp_model_stats=true --display_all_solutions=true --statistics=true > 01_small_ortools_v_10.out

In all cases 8 threads were used. Only the version was changed, and the three models run separately, i.e. six separate runs. The results are available in the ortools files. Performance Monitor available in Windows was used to track performance of the fzn-cp-sat.exe instance. It allows one to export performance measures of the last two minutes. Those results can be seen in the files inside performance.zip.

What did you expect to see

I expected that the different versions would have similar memory requirements and similar performances, with v11 perhaps using somewhat more memory and potentially be faster at finding solutions.

What did you see instead?

Both memory usage and performance of v10 were significantly better compared to v11, something which became especially prominent with the largest model. Because of the 64GB restriction on my machine, the largest model appears to have run out of memory (but without crashing) causing the calculations to slow down. For the small and medium models, the performance of both v10 and v11 was similar until solutions were found, at which point v11 slowed down. A consistent result was that CPU usage decreased once memory usage started to increase in v11. The CPU performance decrease became more pronounced as memory usage increased.

From the Performance Monitor, one can see that for the large model, v10 used ~15 GB of RAM (from the 03_large_ortools_v_10.perf_out file, searching for Working Set Peak), whereas v11 used ~54 GB of RAM (03_large_ortools_v_11.perf_out). While v10 levelled out at 15 GB, it is conceivable that v11 intended to use more than 54 GB but reached a hard limit due to the hardware's physical restrictions.

While it could be argued that this performance issue would be resolved by merely adding additional RAM to the machine, it was evident from the small and medium runs that some performance degradation also occurred as memory usage increased, even though memory requirements here were still far below hard limits. There appears to be a fairly consistent trade-off where increased memory usage is coupled with decreased CPU usage.

Given the hardware limitations available to me, I would thus use v10 rather than v11 as I have found that, for my models, v10 typically finds at least 1 solution within the hour, whereas v11 often finds no solution even after two hours. Also, for scenario testing, I often run two 8-CPU runs simultaneously using v10 (at 15 GB each, which is still far below the 64 GB limit). With v11, even a single 8-CPU run is currently too slow to be viable in production. Difference in behaviour was, in my opinion, sufficient to warrant this report.