Running sopflow with a larger number of scenarios and contingencies

[nkoukpaizan]

Issue type

• New feature
• Bug
• Discussion
• Other

Relates to

• OPFLOW
• SOPFLOW
• SCOPFLOW
• TCOPFLOW
• CMake build system
• Spack configuration
• Manual
• Web docs
• Other

Summary

While performing weak and strong scaling analyses on Summit, we were unable to run with more that ~64,000 recourse problems (number of scenarios times number of contingencies). The error message is the following:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Overflow in integer operation: https://petsc.org/release/faq/#64-bit-indices
[0]PETSC ERROR: Global size overflow 3371958272. You may consider ./configure PETSc with --with-64-bit-indices for the case you are running
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.21.5, Aug 30, 2024
[0]PETSC ERROR: /gpfs/alpine2/proj-shared/eng142/nicholson/10-18-2024/builds/exago-install/bin/sopflow on a  named a01n01 by Unknown Mon Oct 21 14:41:54 2024
[0]PETSC ERROR: Configure options --prefix=/gpfs/alpine2/stf006/world-shared/nkouk/exago-spack-install/linux-rhel8-power9le/gcc-10.2.0/petsc-3.21.5-cqs7gehe3g2ue7qzrwmmnlo3p7t6jnxw --with-s
[0]PETSC ERROR: #1 PetscSplitOwnership() at /gpfs/alpine2/stf006/world-shared/nkouk/exago-spack-cache/build-stage/spack-stage-petsc-3.21.5-cqs7gehe3g2ue7qzrwmmnlo3p7t6jnxw/spack-src/src/sys
[0]PETSC ERROR: #2 PetscLayoutSetUp() at /gpfs/alpine2/stf006/world-shared/nkouk/exago-spack-cache/build-stage/spack-stage-petsc-3.21.5-cqs7gehe3g2ue7qzrwmmnlo3p7t6jnxw/spack-src/src/vec/is
[0]PETSC ERROR: #3 VecCreate_MPI_Private() at /gpfs/alpine2/stf006/world-shared/nkouk/exago-spack-cache/build-stage/spack-stage-petsc-3.21.5-cqs7gehe3g2ue7qzrwmmnlo3p7t6jnxw/spack-src/src/v
[0]PETSC ERROR: #4 VecCreate_MPI() at /gpfs/alpine2/stf006/world-shared/nkouk/exago-spack-cache/build-stage/spack-stage-petsc-3.21.5-cqs7gehe3g2ue7qzrwmmnlo3p7t6jnxw/spack-src/src/vec/vec/i
[0]PETSC ERROR: #5 VecSetType() at /gpfs/alpine2/stf006/world-shared/nkouk/exago-spack-cache/build-stage/spack-stage-petsc-3.21.5-cqs7gehe3g2ue7qzrwmmnlo3p7t6jnxw/spack-src/src/vec/vec/inte
[0]PETSC ERROR: #6 VecSetTypeFromOptions_Private() at /gpfs/alpine2/stf006/world-shared/nkouk/exago-spack-cache/build-stage/spack-stage-petsc-3.21.5-cqs7gehe3g2ue7qzrwmmnlo3p7t6jnxw/spack-s
[0]PETSC ERROR: #7 VecSetFromOptions() at /gpfs/alpine2/stf006/world-shared/nkouk/exago-spack-cache/build-stage/spack-stage-petsc-3.21.5-cqs7gehe3g2ue7qzrwmmnlo3p7t6jnxw/spack-src/src/vec/v
[0]PETSC ERROR: #8 SOPFLOWSetUp() at /gpfs/alpine2/proj-shared/eng142/nicholson/10-18-2024/builds/ExaGO/src/sopflow/interface/sopflow.cpp:1162
[0]PETSC ERROR: #9 main() at /gpfs/alpine2/proj-shared/eng142/nicholson/10-18-2024/builds/ExaGO/applications/sopflow_main.cpp:94
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -ctgcfile /gpfs/alpine2/proj-shared/eng142/exascale/datafiles/contingencies_ACTIVSg10k_kpp2.cont (source: command line)
[0]PETSC ERROR: -log_view (source: command line)
[0]PETSC ERROR: -netfile /gpfs/alpine2/proj-shared/eng142/exascale/datafiles/case_ACTIVSg10k.m (source: command line)
[0]PETSC ERROR: -opflow_ignore_lineflow_constraints (source: command line)
[0]PETSC ERROR: -opflow_include_loadloss_variables 0 (source: command line)
[0]PETSC ERROR: -opflow_include_powerimbalance_variables 0 (source: command line)
[0]PETSC ERROR: -opflow_initialization FROMFILE (source: command line)
[0]PETSC ERROR: -opflow_output_format MATPOWER (source: command line)
[0]PETSC ERROR: -opflow_tolerance 1e-6 (source: command line)
[0]PETSC ERROR: -save_output 0 (source: command line)
[0]PETSC ERROR: -scenfile /gpfs/alpine2/proj-shared/eng142/exascale/datafiles/scenarios_ACTIVSg10k_kpp2.csv (source: command line)
[0]PETSC ERROR: -sopflow_enable_multicontingency (source: command line)
[0]PETSC ERROR: -sopflow_flatten_contingencies (source: command line)
[0]PETSC ERROR: -sopflow_Nc 8191 (source: command line)
[0]PETSC ERROR: -sopflow_Ns 16 (source: command line)
[0]PETSC ERROR: -sopflow_output_directory /gpfs/alpine2/proj-shared/eng142/nicholson/10-18-2024/weak-scaling-resolve-glu-1/outputs/sopflow_out_case10k_16.8191.43009.3800786 (source: command
[0]PETSC ERROR: -sopflow_solver HIOP (source: command line)
[0]PETSC ERROR: -sopflow_subproblem_model POWER_BALANCE_POLAR (source: command line)
[0]PETSC ERROR: -sopflow_subproblem_solver IPOPT (source: command line)
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to [email protected]

I had tried building PETSc with +int64 in Spack as the message suggests, but that seems to require corresponding changes within ExaGO.
This may require improvements to the code that partitions scenarios and contingencies.

cc @pelesh @abhyshr

[nkoukpaizan]

cc @maksud