"ERROR: MPI exchange of back wall info timed out [...]"

[FireResearch-BK]

A case of mine crashed with the error message below:

I've never encountered this one before and the relevant lines in the source code don't contain any further comments or information: https://github.com/firemodels/fds/blob/master/Source/main.f90

Looks like a general communication error in the MPI protocol, but why would that happen on back wall info specifically or was this just coincidence?

FDS did not terminate from the error. I had to kill it from Task Manager.

Has anybody encountered this or similar errors before and what was the cause?

 

This is with Windows 11 Pro on a Core i7 13700H with 16 GB RAM, in case that's relevant.

[rmcdermo]

It might be helpful to have both the input file and the exact command you used to launch the case. Also, is this the latest code, 6.9.1?

[mcgratta]

Also, this should be submitted as an Issue, not a Discussion. For a Discussion, we cannot assign a specific person to work on it, and what inevitably happens is that it falls far down the list of discussion posts.

But as to your inquiry, this error can be caused by a bug in FDS or a "hiccup" in your inter-node network. We have to run the case on our computer to be sure.

[FireResearch-BK]

Posting the entire case in public here or as part of an issue is not possible as it contains client-sensitive data.
This bare-bones file without any obstruction information (and anonymized) is all I can provide:

Case.txt

In this form, the case will run, but naturally not produce sensible results for lack of obstructions. It's just to show that there's nothing too out of the ordinary going on.
Note that the complex stochiometry uses a fuel formula that was engineered to obtain specific CO and CO2 yields.

The case was launched with FDS 6.9.1 with [path\to\FDS_Local.bat] -p 14 -n 1 [path\to\case_file.fds].

This is the .out file of the original case (again, anonymized and adapted to the rough inputs of the test case above; pressure zone information unaltered). Note the stop and restart after time step 8430.

Original_out.txt

[mcgratta]

If this case does not fail, there is nothing that we can do. Does it fail?

[FireResearch-BK]

I'd obviously have to run it the same way that I did the original case, which involves a few days.

After a bit of digging, I found that the error messages have been "beautified" at some point to show the specific information that MPI failed to exchange instead of its error code. So what was "REQ6 timed out for MPI process [...]" is now "MPI exchange of back wall info timed out for MPI process [...]".
Involving the old REQ[n] codes in a web search indeed turned up some old issues and discussions related to MPI timeouts, all without a specific solution.

What is interesting about them, from what I could gather, is that most, if not all, have been encountered on machines where there is some higher than normal degree of communication latency between CPUs, so basically by clustering some in a network, having two CPUs on a single mainboard or, as in this case, on a heterogenous CPU design with distinct performance and efficiency cores like this 13th gen Intel mobile CPU.
On homogenous CPUs (AMD Ryzen or Threadripper, old Intel i7s) at least, I've never encountered any timeout errors so far, despite using every single available physical and virtual core on them.
Maybe something to keep in mind for future encounters with timeout errors.

Hydra has a "I_MPI_JOB_TIMEOUT" parameter, which I guess would cause jobs to time out under certain conditions. Its default value is zero, which means there's no timeouts at all.
This parameter is not touched in FDS_Local.bat and I assume not anywhere else in FDS' handling of MPI, correct?

I also found I_MPI_WAIT_MODE and some discussions about it. Maybe I'll give the "-O" flag for FDS_local.bat (which sets I_MPI_WAIT_MODE=1) a try.

P.S:
One thing I've noticed is that FDS_local.bat still references "I_MPI_WAIT" in line 210 when it should be "I_MPI_WAIT_MODE"

[mcgratta]

If you want to experiment with these kinds of options, I suggest that you not use fds_local. Just write out the full command. We have not tested these options other than to ensure that the command line is written properly.

You are right about unbalanced loads on the compute cores. That does seem to lead to more hangs than when all the meshes are the same. The problem is that it is very difficult to fix these problems of hanging jobs. Also, if you are running under Windows, you may see more problems than linux because Windows is typically not used for high performance computing.