Add a Sentence struct, replace Vec<Token> with Sentence where possible

[bminixhofer]

This PR shifts the focus from Vec<Token> to a new Sentence struct and moves the fields on the Token which semantically belonged to the sentence (sentence and tagger) to the sentence. Includes some general improvements like:

• Returning an iterator over Sentence / IncompleteSentence instead of a Vec<Vec<Token>> / Vec<Vec<IncompleteToken>>.
• Fixing indices on the token (byte_span / char_span) by making them relative to the input text (as opposed to relative to each sentence, as currently the case) and making them a Range instead of (usize, usize).

Future work in this direction would include rethinking the distinction between IncompleteSentence / Sentence and possibly unifying them or removing the need for one of them, but that's out of scope here.

[bminixhofer]

This is ready for a review @drahnr if you have the time. It turned out more substantial than I initially thought. In addition to the things mentioned above:

• there is now Span and Position structs which track the byte and char indices at the same time. (so instead of token.char_span() you now write token.span().char()).
• fields on the Suggestion, IncompleteToken and Token are now private and have getters.
• there is a private new MatchSentence struct which represents a sentence prepared for matching on the rules. So the special SENT_START token is not visible in the public API anymore and each token in the Sentence and IncompleteSentence corresponds to at least one character in the input text.

There's a big diff because of many trivial changes, most of the interesting ones are in the tokenizer.rs and types.rs. Again, I'd especially appreciate feedback regarding the changes and additions to the public API. Thanks!

[drahnr]

That looks like it was quite a bit of work, so much appreciated!

A few nitpicks which are intertwined:
.span().char().start() does add quite a bit of visual clutter, I would recommend to go for fn char_range() -> Range<usize> { .. } which allows direct modification of the inner types, so .char_range().start could be used instead and avoid 2x () all over the place.

I think a few redundant calls to .chars().count() could (and eventually should) be avoided.

But again, just nits 👍

[bminixhofer]

Thanks a lot for the review!

I'm a bit conflicted regarding .char_range and .byte_range. Adding them would allow us to make .span pub(crate) which avoids the naming problem but it means:

• trying to enforce that .char_range and byte_range are never used internally (internally, .span() should be used since it allows e.g. getting the starting byte and char index with .span().start()).
• Code duplication by adding a char_range and byte_range method to every struct that has a Span (Suggestion, Sentence, IncompleteSentence, IncompleteToken, Token).
• It makes some things which are easy to do with the Span API harder e.g. getting the start byte and char index as Position is .span().start() but would need .char_range().start and .byte_range().start otherwise. This is then easy to pass to e.g. rshift on the Suggestion which needs a Position.

so I think I'll keep it as it is. I'll look over this PR once more tomorrow and merge it then. Also if you haven't seen it already you might be interested in #50 (comment), an update regarding the spellchecking discussion.