Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filesystem / DB contention with multiple readers #1005

Open
tskisner opened this issue Oct 17, 2024 · 0 comments
Open

Filesystem / DB contention with multiple readers #1005

tskisner opened this issue Oct 17, 2024 · 0 comments
Assignees

Comments

@tskisner
Copy link
Member

This issue is just for keeping track of an investigation into observed "slowdowns" when multiple processes call get_meta() / get_obs() on different wafers (i.e. different framefiles), both within a single book and from separate books. This is from within the LoadContext operator, so each process creates a context, does the operation (either get_meta or get_obs) and then closes the context.

Mostly this is just anecdotal so far. For example running a single process that loads 7 wafers in sequence from one observation takes about 60 seconds per wafer (perlmutter compute node, reading data from CFS) to call get_meta + get_obs. Running with 8 processes, each reading 7 wafers in sequence from different observations, seems to take considerably longer.

A more systematic test is needed. The changes in #845 should also be tested to see if they help.

@tskisner tskisner self-assigned this Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant