多进程预处理--preprocessing_num_workers 不生效

[trotsky1997]

Reminder

• I have read the README and searched the existing issues.

System Info

            padded_dataset = dataset.map(pad_sequence, batched=True)
            sp_dataset = padded_dataset.map(sp_split, batched=True)

好像是这两句.map方法未接受--preprocessing_num_workers的输入

Reproduction

...

Expected behavior

No response

Others

No response

[trotsky1997]

            padded_dataset = dataset.map(pad_sequence, batched=True,num_proc=data_args.preprocessing_num_workers)
            sp_dataset = padded_dataset.map(sp_split, batched=True,num_proc=data_args.preprocessing_num_workers,)

改成这个就行了

[trotsky1997]

我发个pr吧

[HaoshengZou]

感谢！逗号后面加个空格就好
batched=True, num_proc=

[trotsky1997]

感谢！逗号后面加个空格就好 batched=True, num_proc=

哈哈，是的

[trotsky1997]

感谢！逗号后面加个空格就好 batched=True, num_proc=

#8