Way to selectively turn off large output files from IES?

[sjepsen395]

Dear development team,

I'd like to turn off output of some of the larger files from IES. Some of these files are not currently necessary for our work and cause us to lose all of our data on Azure Batch. This is because Azure Batch has a set timeout for zipping and transferring all output files to the client application (I was told 15 minutes). If all of the IES output files are too large to zip/transfer within that time window, all data is lost and not recoverable (how crazy is that). This just happened to me on a $175 run (~ 5,000 model simulations).

It would be great if the user of IES had an option to turn off output of certain files. Is this currently an option, or could this option possibly be added in the future? Perhaps an on-off flag for each file type? The files I'd like to turn off output for at the present time are:

<case_name>.<iter_num>.obs.csv
<case_name>.adjusted.weights.csv
<case_name>.obs+noise.csv
<case_name>.weights.csv

It would also be nice to just output the last *.obs.csv file for use in a possible restart, but I think I could get around having that if I had the *.par.csv files. Similar to the way that PEST always outputs the final *.par and *.jco file (if enabled).

Thanks,
Steve

[jtwhite79]

This is doable - I'll work on it. Probably will have a more general output-verbose level, rather than a more granular per-file thing.

The reason to save every iterations results is in case you want to see how the ensemble is changing across iterations (this can be useful...) or in case you want to use an early iteration as the "final" results to mitigate overfitting/preserve more forecast variance

[sjepsen395]

Jtwhite79,

Great, thank you so much. Yes, that makes sense. I wish we could keep all of the output files. For my cloud application, the only big file that I'd try to keep is the final *.obs file, assuming that was an IES option. That would allow me to do a restart if I wanted (I've never actually done this so could be off there). I'd also keep the *.par files, objective function files, and other relatively small files.

Steve