Fix Hardcoded Tokenizer (#71) #305

Job	Run time
info	1s
build-macos (x86_64)	3m 12s
build-linux (x86_64)	3m 17s
build-macos (aarch64)	2m 21s
build-windows (x64)	10m 47s
build-linux (x86)	8m 37s
build-windows (x86)	7m 26s
build-linux (aarch64)	4m 14s
build-linux (armv7)	4m 18s
sdist	18s
tests (3.8, Check Python style, isort --check tests/python/ && isort --check python/ black --chec...	4m 17s
tests (3.8, Check Rust style, rustfmt --edition 2021 src/*.rs --check )	5m 12s
tests (3.8, Lint, flake8 tests/python/ && flake8 python/ )	4m 28s
tests (3.8, Type check, mypy tests/python/ && mypy python/ )	7m 39s
tests (3.8, Test, pytest -vs --color=yes tests/python/ )	3m 44s
Release	0s
	1h 9m 51s

Provide feedback