public API changes

[OmarTawfik]

as discussed in #617:

• Remove ProductionKind completely, since it is no longer needed. We only need to expose fn parse(NonTerminalKind): Clean up precedence parser codegen since DSL v2 #699
• Rename TokenKind to TerminalKind, since it will also refer to trivia. (Consider Rule/Token => Nonterminal/Terminal rename #834)
• Rename RuleKind to NonTerminalKind, since "rule" is ambiguous. (Consider Rule/Token => Nonterminal/Terminal rename #834)
• Remove TerminalKind::SKIPPED from TerminalKind, and add new Node::Unrecognized(string) and Node::Missing(kind) variants instead. (Distinguish between unrecognized and missing input in the CST when recovering parse #835)
• Hide LexicalContext and fn scan(TokenKind) from the public API: Hide Lexer-related facilities from the public API #728
• Since EndOFFileTrivia no longer exists, ParseResult should collect any remaining trivia at the end of the input, and include it in the ParseResult returned, for any kind of non-terminal, not just SourceUnit.
• Emit trivia as sibling tokens and don't generate a RuleKind for it #737

[Xanewok]

I've opened #834 and #835 to track the remaining work items that we should probably still discuss, so closing this umbrella issue; the Area in the project board serves to show that these are related to the public API, so there's no use tracking them in a group here.