Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support large dataset preprocessing #452

Open
wants to merge 31 commits into
base: develop
Choose a base branch
from
Open

Support large dataset preprocessing #452

wants to merge 31 commits into from

Conversation

chiang-yuan
Copy link
Contributor

This PR tries to resolve OOM error and improve performance when loading very large dataset like MPTrj (1.58M) or even bigger ones. To use this file, mpi4py is needed.

Additional file preprocessing_data_mpi.py is added to ensure back compatibility and the refactoring is reduced to minimum, but ideally preprocessing_data.py could be replaced with the new file as long as we consider the import dependency on mpi4py

@ilyes319 ilyes319 changed the base branch from main to develop June 21, 2024 14:12
@ilyes319
Copy link
Contributor

Hey @chiang-yuan thank you. Is that ready to be merged?

@chiang-yuan
Copy link
Contributor Author

it still needs some refactoring. It seems like only modifying the ase read part is not enough. I will refactor all the hdf5 file writing part as well but it might take sometime...

chiang-yuan and others added 22 commits July 1, 2024 22:47
@chiang-yuan
bugfix: printing only for rank 0
6196ed5
@JPDarby @chiang-yuan
remove n_atoms factor
6ebfd11
@chiang-yuan
updated tests
a5b1891
@ilyes319 @chiang-yuan
fix preprocess data local error
9d4b5de
@ilyes319 @chiang-yuan
fix mace_off float32 finetuning
d3ff5c0
@ilyes319 @chiang-yuan
change numpy<2.0 in setup.cfg
47999ee
@ilyes319 @chiang-yuan
fix import sort and linting
e004bb9
@ilyes319 @chiang-yuan
fix universal loss stress by default
76ea313
@chiang-yuan
add wandb_dir in argparser, changed wandb directory related parts
c261ed8
@RokasEl @chiang-yuan
initial
695a8ae 
argparse support schedulefree

support specifying betas

fix single layer logging

Add optional dependency and a loading test
@RokasEl @chiang-yuan
change tempdir to tempfile to be consistent with other tests
3f29950
@ElliottKasoar @chiang-yuan
Bump Actions versions
e0c427a
@ElliottKasoar @chiang-yuan
Add initial release workflow
975f3cf
@ElliottKasoar @chiang-yuan
Add template for DOI badge
628ca17
@ilyes319 @chiang-yuan
upade the readme installation instructions
34fbefb
@ilyes319 @chiang-yuan
formatting in the readme installation
14e3c22
@ilyes319 @chiang-yuan
add more details on the GPU cuda installation
a55e913
@ilyes319 @chiang-yuan
remove DOI from badge
41980c3
@ilyes319 @chiang-yuan
add finetuning to readme TOC
72a7a3c
@ilyes319 @Nilsgoe @chiang-yuan
add support for hessian in calculator
ad900db 
Co-Authored-By: Nils Goennheimer <[email protected]>
@ilyes319 @Nilsgoe @chiang-yuan
fix torchscript compilation for hessian
4c8245f 
Co-Authored-By: Nils Goennheimer <[email protected]>
@ilyes319 @chiang-yuan
prepare the model for hessian
986f540
@chiang-yuan chiang-yuan mentioned this pull request Sep 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@chiang-yuan @ilyes319 @JPDarby @RokasEl @ElliottKasoar @jameskermode