You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This issue is just for keeping track of an investigation into observed "slowdowns" when multiple processes call get_meta() / get_obs() on different wafers (i.e. different framefiles), both within a single book and from separate books. This is from within the LoadContext operator, so each process creates a context, does the operation (either get_meta or get_obs) and then closes the context.
Mostly this is just anecdotal so far. For example running a single process that loads 7 wafers in sequence from one observation takes about 60 seconds per wafer (perlmutter compute node, reading data from CFS) to call get_meta + get_obs. Running with 8 processes, each reading 7 wafers in sequence from different observations, seems to take considerably longer.
A more systematic test is needed. The changes in #845 should also be tested to see if they help.
The text was updated successfully, but these errors were encountered:
This issue is just for keeping track of an investigation into observed "slowdowns" when multiple processes call
get_meta()
/get_obs()
on different wafers (i.e. different framefiles), both within a single book and from separate books. This is from within theLoadContext
operator, so each process creates a context, does the operation (either get_meta or get_obs) and then closes the context.Mostly this is just anecdotal so far. For example running a single process that loads 7 wafers in sequence from one observation takes about 60 seconds per wafer (perlmutter compute node, reading data from CFS) to call get_meta + get_obs. Running with 8 processes, each reading 7 wafers in sequence from different observations, seems to take considerably longer.
A more systematic test is needed. The changes in #845 should also be tested to see if they help.
The text was updated successfully, but these errors were encountered: