[Question]: Does Flair support ModernBert 8k tokens context size?

[heukirne]

Question

Recently ModernBert was launched with 8k context lenght: https://huggingface.co/blog/modernbert

Does Flair support fine tuning and inference of models with context lenght higher than 512?

Thanks!

[heukirne]

I was able to run it, just need to set the mex_length parameter:

from flair.embeddings import TransformerWordEmbeddings

embeddings = TransformerWordEmbeddings(model='answerdotai/ModernBERT-base',
                                       layers="all",
                                       subtoken_pooling="first",
                                       fine_tune=True,
                                       use_context=True, 
                                       transformers_tokenizer_kwargs={'model_max_length': 8192}
                                       )

Remember to install the main transformer, flash-attn and triton

pip install git+https://github.com/huggingface/transformers.git
pip install flash-attn --no-build-isolation
pip install triton

ModernBERT will be included in v4.48.0 of transformers. Until then, it requires installing transformers from main.

[stefan-it]

Hey @heukirne ,

if you plan to use ModernBERT for Token Classification tasks, please have a look at my NER repo:

https://github.com/stefan-it/modern-bert-ner

ModernBERT has currently a tokenizer issue, which is (probably) not seen for text classification tasks...