Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue: Error with _eos_token_tensor when using Generator with GenerationMixin #35767

Open
1 of 4 tasks
surenoobster opened this issue Jan 18, 2025 · 1 comment
Open
1 of 4 tasks
Labels

Comments

@surenoobster
Copy link

System Info

e-readability-summarization/src/inference$ python run_2.py
Traceback (most recent call last):
File "/home/surenoobster/Documents/project/src/inference/run_2.py", line 87, in
output = generator.generate(
^^^^^^^^^^^^^^^^^^^
File "/home/surenoobster/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/surenoobster/Documents/project/src/inference/generation_2.py", line 572, in generate
stopping_criteria = self.model._get_stopping_criteria(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/surenoobster/anaconda3/lib/python3.12/site-packages/transformers/generation/utils.py", line 1126, in _get_stopping_criteria
if generation_config._eos_token_tensor is not None:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'GenerationConfig' object has no attribute '_eos_token_tensor'

Who can help?

No response

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

I'm not able to get it , getting problem with device

Expected behavior

The Generator should initialize the required token tensors correctly to ensure compatibility with GenerationMixin and avoid errors.

@zucchini-nlp
Copy link
Member

Hey!

The attributes like _eos_token_tensor were added for torch.compile compatibility and are created on-the-fly when calling generate(). Since in your code you are using private generation methods for custom code, I'd recommend you to call this mthod just before preparing stopping criteria

def _prepare_special_tokens(
self,
generation_config: GenerationConfig,
kwargs_has_attention_mask: Optional[bool] = None,
device: Optional[Union[torch.device, str]] = None,

cc @gante

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants