Skip to content

Charabia v0.6.0

Compare
Choose a tag to compare
@meili-bot meili-bot released this 22 Aug 12:11
· 494 commits to refs/heads/main since this release
faafb12

Changes

Breaking changes ⚠️

Add option to disable char map creation (#109) @matthias-wright

Token::original_lengths(..) method, used to find the original index of a character in a normalized string, needs the TokenizerBuilder::create_char_map(..) settings set to true to work properly.

Thanks again to @ManyTheFish, @aFluffyHotdog, @matthias-wright and @mosuka! 🎉