[Ready] Reduce reload word tokenizer engine in word_tokenize

[new5558]

What does this changes

Fixes #973

What was wrong

How this fixes it

Fixes #...

Your checklist for this pull request

• Passed code styles and structures
• Passed code linting checks and unit test
• Test Attacut and icu manually
• Benchmark speed improvement

[pep8speaks]

Hello @new5558! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2025-01-11 15:09:30 UTC

[coveralls]

coverage: 52.795% (-0.04%) from 52.836%
when pulling d8d22f6 on new5558:dev
into 3aa57c6 on PyThaiNLP:dev.

[new5558]

Manual test of Attacut & ICU tokenizers results and benchmark results can be found in this Colab Notebook

Performance Improvements

(Time per tokenizer function call)

Tokenizer	Original (ms)	This PR (ms)	Original 50 threads (ms)	This PR 50 threads (ms)
Attacut	9.83 ms	1.67 ms	17.18 ms	2.83 ms
ICU	0.03 ms	0.03 ms	0.19 ms	0.27 ms
Longest	0.02 ms	0.01 ms	0.21 ms	0.20 ms

Attacut engine is 6X Faster when tokenizing words "สวัสดี PythaiNLP".
Not much improvement in ICU and Longest engines,

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[wannaphong]

Thank you! 💯