access logs for live pipeline

[pbellec]

There is currently no clean way to access the logs of a job while it is running.

[poquirion]

Do you mean that you need to know which worker is running the job to access its log, but you do not have direct knowledge of that information?

[pbellec]

logs appear in worker.log (pooled with many other logs) for an unknown worker. The logs also appear in a job_name.log file, but they get flushed in there all at once when the job finishes, not continuously. The logs are collected when the job is finished, at which point they appear in PIPE_logs.mat. But even then, if the user were to use psom_pipeline_visu, this could conflict with the garbage collector and create a conflict. We need the log file to update continuously, and psom_pipeline_visu to look for this file instead of PIPE_logs.mat.

[poquirion]

Would redirecting stdout and stderr to a file in the qsub cmd do the trick or would it need to be an "inside octave code" modification of stderr and stdout value when code is executed by worker?

[pbellec]

This problem is new in psom2. When the deamon (formerly the manager) executes a worker (formerly a job), it does so by running a script and redirecting stdout and stderr. This results in the correct behaviour, currently seen in worker.log. But in psom2, individual jobs are all ran into a single session (of the worker). Individual job logs file are generated with the diary command of octave, and that thing flushes everything at once when issuing the command diary off at the end of the job. We need to work around that behaviour. There may be an octave command. As a last resort, we may have to start new (local) octave sessions to run individual jobs. This would add unnecessary overhead, but all the machinery is already in place. This would be a minor change (calling psom_run_script instead of psom_run_job).

[poquirion]

Ok, got it. We can redefine stdout and sdterr in octave, we could set that as the first step after a clear call.
then

fclose (stdout);
fclose (stderr);
stdout = fopen ("file_for_log.out", "w");
stderr = fopen ("file_for_log.err", "w");

[pbellec]

not sure what the logic is. Does not seem to work as is.

[pbellec]

I googled the problem quite a bit, and at this stage I don't see a solution from within octave.

[poquirion]

Oups we can only do that at the c level it seems... not directly in octave.

[poquirion]

Can we recompile diary as psom_diary to do what we want it to do?