missing registered fields in diag_field_log.out

[rem1776]

Describe the bug
The diag_manager has a nml variable that will create a output log of all register_*_field calls, but during SPEAR runs the output log is missing some register calls in the FV3 code.

To Reproduce
Run the spear regression xml with do_diag_field_log set to true in the diag_manager namelist.

Expected behavior
the log should have every registered field.

System Environment
gaea c5 with intel23-classic

Additional context
It seems like the mom code is likely the culprit. i had the code print the pe for every write to the log, mom starts calling with a different root pe and that seems to erase whatever was in the file up to that point.

Thanks @wfcooke for spotting this!

[thomas-robinson]

@marshallward is there a MOM wrapper for this?

[wfcooke]

Ryan Nice find. Have you tried this with CM4? It shouldn't just be a SPEAR thing. I'll guess that it is pe0 writing to the file and then pe2304 (or whatever the MOM root-pe is) opening and clobbering the file. Do we need two diag_field output files? MOM actually writes out its own diag field output regardless (to available_diags.002304 in one of my runs) Will

…

On Thu, Apr 25, 2024 at 7:04 AM Tom Robinson ***@***.***> wrote: @marshallward <https://github.com/marshallward> is there a MOM wrapper for this? — Reply to this email directly, view it on GitHub <#1496 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHCVICYEZVE76TXLJBPCWDY7DPLBAVCNFSM6AAAAABGXTHIR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZWHEZDIMJSHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[marshallward]

I'm not familiar with this feature but I am looking at it now.

It seems like this is all handled within register_static_field and enabled by the do_diag_field_log in diag_manager_nml. I would guess that MOM does not control any of this?

[marshallward]

@wfcooke I was under the impression that available_diags.* was a MOM6 product separate from the diag manager. (i.e. it is created by the "diag mediatior" which uses the FMS diag manager).

Your explanation seems plausible, although I always had the impression that MOM6 was using the same diag manager as the other components. (That is, all submodels were using the same module-level variables.)

[wfcooke]

Marshall That's my impression too. MOM is producing the available_diags file of its own accord. With do_diag_field_log = true, there will be two copies of the MOM6 (and SIS2?) in the output. Does the code at https://github.com/NOAA-GFDL/FMS/blob/7f585284f1487c0629f8075be350385e6e75dd2f/diag_manager/diag_manager.F90#L3928 try to open the same file with write permissions for ATM_ROOT_PE and OCN_ROOT_PE? Should that be append? Will

…

On Thu, Apr 25, 2024 at 11:08 AM Marshall Ward ***@***.***> wrote: @wfcooke <https://github.com/wfcooke> I was under the impression that available_diags.* was a MOM6 product separate from the diag manager. (i.e. it is created by the "diag mediatior" which uses the FMS diag manager). Your explanation seems plausible, although I always had the impression that MOM6 was using the same diag manager as the other components. (That is, all submodels were using the same module-level variables.) — Reply to this email directly, view it on GitHub <#1496 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHCVIGGUFBFSA7ZCVEHP5DY7EL47AVCNFSM6AAAAABGXTHIR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZXGQ4TQMZUGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rem1776]

@wfcooke Yeah I believe thats the problem. The ATM root pe opens the file, and then the OCN root pe opens it again as a new file when it calls diag_manager_init.

I have a PR in (#1497) to include the root pe number at the end of the filename, similar to what we do with logfile.out. My first thought was to just open it in append mode as well, but as Rusty pointed out this way we won't run into any race conditions from two root pe's writing to the same file.

[marshallward]

You could just check if the file has already been opened before reopening creating it. There is still a slight possibility of a TOCTOU race condition, but it would be much safer than the current approach.