fix: filter long and invalid prompts in future messages VSCODE-614

[gagik]

Description

When someone sends a long message (or some message which could cause it to be i.e. filtered by the responsible AI service), we keep resending this in future prompts, causing the chat to be broken for the whole session. This PR excludes them from the results.

Checklist

• New tests and/or benchmarks are included
• Documentation is changed or added
• I have signed the MongoDB Contributor License Agreement (https://www.mongodb.com/legal/contributor-agreement)

Motivation and Context

• Bugfix
• New feature
• Dependency update
• Misc

Open Questions

Dependents

Types of changes

• Backport Needed
• Patch (non-breaking change which fixes an issue)
• Minor (non-breaking change which adds functionality)
• Major (fix or feature that would cause existing functionality to change)

[gagik]

I figured it'd be better to get the error types out of the scope of telemetry since it might be useful to handle specific errors.

[alenakhineika]

I am thinking, what if instead of removing the long message from the history we should introduce a max prompt size, similar like we do in compass, and inform users after submission, that their prompt is too long. We have even mentioned doing something like this in tech design in the "Building Chat Participant" section. @Anemy @GaurabAryal do you maybe have some thoughts?

[gagik]

I am thinking, what if instead of removing the long message from the history we should introduce a max prompt size, similar like we do in compass, and inform users after submission, that their prompt is too long. We have even mentioned doing something like this in tech design in the "Building Chat Participant" section. @Anemy @GaurabAryal do you maybe have some thoughts?

That'd be good for that specific case, which should be common enough to be worth handling.

Beyond that I assume we may still end up in scenario where i.e. maybe the user has some bad words which are filtered by the Responsible AI Service (which I think does do filtering beyond just prompt size), in which case we'd probably want to dismiss its contents in the following messages; otherwise again we'll leave the user stuck for the whole chat just because they made a mistake. I read people complain about the filter before, so this might be quite annoying.

[nirinchev]

I agree - we probably want to do both. Note that the max prompt size concerns both the user prompt, but also the history, so if a user has a very long conversation, they may end up exceeding the limit, even if their last message was a short one. That means we'll need to build out a system for dropping the oldest history messages.