🧪 Expectations

[hejny]

Large language models often do not give you what you need, this is called prompt drift, which can ruin the entire pipeline execution if the problem occurs at the beginning of the middle of the pipeline.

BUT it is often very easy to find out and solve:

Count expectations

- EXPECT MIN 5 Words
- EXPECT MAX 25 Words
- EXPECT MIN 15 Characters
- EXPECT MAX 1000 Characters
- EXPECT MIN 1 Sentence
- EXPECT MAX 5 Paragraphs

Format expectations

- EXPECT JSON

- EXPECT XML

- EXPECT VCARD

Format with schema

- EXPECT JSON https://promptbook.studio/sample/schemas/contacts.schema.json

This is not ready yet, check 📑 Formats

---

See also ✂️ Postprocessing and for more advanced expectations check 👮 Agent adversary expectations and 🔙 Expectation-aware generation

And in future we would maybe do 🔴 Anomaly detection

[hejny]

🔎 Other existing solutions

(Do some research / discussion)

• Has anyone else had the same problem?
• Has a project come up with the solution
• Is there a research paper about it?
• Is there an article, video, podcast about it?
• Is this even a good idea?

[hejny]

Should we allow async expectations check - for example checking if the domain is free?