dp train on multiple systems giving list index error

[kimiaGF]

I am attempting to train a model on a large number of systems, each with different frames and number of atoms (same atom types). After parsing the data using dpdata, I was left with the following folder structure ~/deepmd/<systems>/set.[...]. I then separated the data into two folders such that all sets labeled set.000 are pulled out for validation:
training_data/<systems>/set.*{1..9} and validation_data/<systems>/set.000
In the input script, i defined the location of the training and validation sets as follows:

"training" : {
          "training_data":{
          "systems":      "/home/kimia.gh/blue2/processed_data/training_data/",
          "batch_size":   1
          },
          "validation_data":{
          "systems":      "/home/kimia.gh/blue2/processed_data/validation_data/",
          "batch_size":   1
          },

however when running dp train input.json I get the following error:

2022-03-07 15:12:26.738020: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:From /home/kimia.gh/.conda/envs/deepmd/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
Traceback (most recent call last):
  File "/home/kimia.gh/.conda/envs/deepmd/bin/dp", line 10, in <module>
    sys.exit(main())
  File "/home/kimia.gh/.conda/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/main.py", line 437, in main
    train_dp(**dict_args)
  File "/home/kimia.gh/.conda/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 91, in train
    jdata = update_sel(jdata)
  File "/home/kimia.gh/.conda/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 341, in update_sel
    descrpt_data = update_one_sel(jdata, descrpt_data)
  File "/home/kimia.gh/.conda/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 317, in update_one_sel
    tmp_sel = get_sel(jdata, rcut)
  File "/home/kimia.gh/.conda/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 280, in get_sel
    _, max_nbor_size = get_nbor_stat(jdata, rcut)
  File "/home/kimia.gh/.conda/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 255, in get_nbor_stat
    train_data = get_data(jdata["training"]["training_data"], max_rcut, type_map, None)
  File "/home/kimia.gh/.conda/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 199, in get_data
    data = DeepmdDataSystem(
  File "/home/kimia.gh/.conda/envs/deepmd/lib/python3.9/site-packages/deepmd/utils/data_system.py", line 79, in __init__
    DeepmdData(
  File "/home/kimia.gh/.conda/envs/deepmd/lib/python3.9/site-packages/deepmd/utils/data.py", line 68, in __init__
    self.test_dir = self.dirs[-1]

after debugging I can confirm that it's the location of the file that is causing the error, however I'm not sure what exactly is wrong with the formatting of the training and validation data.

[njzjz]

It should be something like

"systems":      [
    "/home/kimia.gh/blue2/processed_data/training_data/system1",
    "/home/kimia.gh/blue2/processed_data/training_data/system2",
    "/home/kimia.gh/blue2/processed_data/training_data/system3"
]

instead.

[njzjz]

os.listdir("/home/kimia.gh/blue2/processed_data/training_data/")

[wanghan-iapcm]

or you may write

"systems":  "/home/kimia.gh/blue2/processed_data/training_data/",

if all the systems are in folder /home/kimia.gh/blue2/processed_data/training_data

[kimiaGF]

I tried both of the methods above and I get the same "list index out of range error".

I even individually listed every directory, but I still get the same error.

[njzjz]

Can you confirm every directory contains at least one set folder?

[kimiaGF]

That was the problem! Thank you so much. For future reference, what is the best way to separate validation and training sets for multiple systems? the set_size option in the .to_npy or .to_raw only takes in a fixed number, so when systems are of different sizes it separates them into a different number of total sets.