there is possible race condition with job start callback

[volodymyrss]

since this callback is sent very quickly after query is sent to the backend

reported by @dsavchenko

[dsavchenko]

This callback is added in nb2workflow by oda-hub/nb2workflow#135

[dsavchenko]

It was also mentined here oda-hub/dispatcher-app#665 (comment)

[burnout87]

I think I see why the race condition happens:

• the first time the run_analysis is called (so when the request is sent, for instance, for the first time, from the frontend), the call_back is also called, and before the run_analysis completes
• so, the call_back completes by writing the progress status within the scratch_dir, but then, it might be overwritten to submitted during the completion of the run_analysis, thus not returning the progress status

I also tested this behavior locally, and I could see that the call-back is not started before the completion of the first run_analysis, but this is expected

This is my guess

[dsavchenko]

What can we do with it? I think of the lock file (won't help by itself, still good to have) and also the condition in the job manager that it can't "lower" the status

[burnout87]

How would you implement the lock? I guess it'd be on the file.

So to make sure that, in relation to what described here, first the run_analysis completes, and then the run_call_back will execute? So we'd have a consistent sequence of states?

also the condition in the job manager that it can't "lower" the status

And about this, when do you see this needed? In the case the first run_call_back called from the nb2service happens to finish before the first call to run_analysis? I will look into it anyway

[dsavchenko]

I just tried again to track possible causes in the code (because I'm not 100% sure about it), but I get lost because I don't really understand a purpose and the logic of "job aliasing". Could someone explain?
May this aliasing also lead to similar problem, e.g. callback and run_query operate in different dirs?

And about this, when do you see this needed? In the case the first run_call_back called from the nb2service happens to finish before the first call to run_analysis?

Yes, this particular case. Probably, just one restriction that "progress" can't become "submitted" would be enough.

How would you implement the lock? I guess it'd be on the file.

I just thought of using a library, like https://py-filelock.readthedocs.io/en/latest/index.html

[dsavchenko]

Another possible race condition: progress report callback may overwrite the "done" status. It's not fully confirmed, but it's possible and I suspect it may be the cause of frontend stuck in "progress" intil re-request with status "new". We observe this occasionally.

[volodymyrss]

Yeah, that's what I see. It looks very weird, there is a product modal, but the only product is "progress" which can be viewed. Eventually actual product also appears.

[burnout87]

ok ,good to know, I will also observe it and test locally

[burnout87]

Another possible race condition: progress report callback may overwrite the "done" status. It's not fully confirmed, but it's possible and I suspect it may be the cause of frontend stuck in "progress" intil re-request with status "new". We observe this occasionally.

oda-hub/dispatcher-app#670 is intended to fix this

[volodymyrss]

Let's close this until confirmed that it happens again.

[dsavchenko]

I will reopen, at least I see this on staging. Example - PhotoZ instrument, Run_phosphoros_basic, it doesn't internally report progress from the notebook, and the job is "submitted" but not "progressing" up until the notebook is completed (or failed).
A consequent inconvenience is that a notebook preview isn't available up until the result is there.