[ES|QL] add a robust system for prioritizing suggestions

[drewdaemon]

The ask

We want to be able to do fine-tuned ordering in the suggestions menu. For example, we may wish to prioritize popular commands above others.

We have implemented custom ordering before such as when we prioritized ECS fields in #187922. However, these efforts have always been ad-hoc and do not constitute a scalable system for assigning suggestion priority.

Another problem we have is that we do not have any test coverage for suggestion priority. This has led to ordering regressions in the past such as #196576.

Technical notes

Monaco provides ordering customization through the CompletionItem.sortText field. It would be up to our system to properly assign sortText values such that the suggestions appear in the correct order. We should consider abstracting the contents of sortText as an implementation detail. I can imagine a rules-based system where we define laws for suggestion ordering and the engine assigns appropriate sortText values based on these rules.

A rules-based system might benefit from another abstraction: more granular suggestion kinds that are then translated into Monaco's CompletionItemKind before being sent to Monaco. This would facilitate prioritization within a single class of suggestions such as ECS fields over other fields. (Typescript does this as well—they translate the fine-grained ScriptElementKind to CompletionItemKind here)

One question that should be answered is if the system needs to support context-dependent ordering. Can we assume that all ordering rules are universal? If not, we need to design a system that can respond to context (e.g. in this particular command, prioritize this, but in that one...).

At one point we did have tests that ostensibly validated the ordering of suggestions, but it didn't perform an accurate simulation of Monaco's sorting behavior and was therefore removed (see #182273 (comment)).

[elasticmachine]

Pinging @elastic/kibana-esql (Team:ESQL)