Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test data padding does not match world_size? #149

Closed
chujiezheng opened this issue Jan 28, 2025 · 1 comment
Closed

Test data padding does not match world_size? #149

chujiezheng opened this issue Jan 28, 2025 · 1 comment

Comments

@chujiezheng
Copy link

I encountered a problem that the padded test data here

test_gen_batch_padded, pad_size = pad_dataproto_to_divisor(test_gen_batch, self.actor_rollout_wg.world_size)
test_output_gen_batch_padded = self.actor_rollout_wg.generate_sequences(test_gen_batch_padded)

cannot be dispatched properly (AssertionError: only support equal chunk.)

splitted_args, splitted_kwargs = _split_args_kwargs_data_proto(worker_group.world_size, *args, **kwargs)
return splitted_args, splitted_kwargs

So I wonder whether the divisor self.actor_rollout_wg.world_size is the same as the chunk number worker_group.world_size?

vermouth1992 added a commit that referenced this issue Jan 28, 2025
…150)

- As titled
- Solved: #149 

Waiting for testing from @chujiezheng

---------

Co-authored-by: Chi Zhang <[email protected]>
@vermouth1992
Copy link
Collaborator

Fix in #150

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants