⚡ Test suite for LLM prompts before pushing them to PROD ⚡
Our work is inspired by this paper Demystifying Prompts in Language Models via Perplexity Estimation which showed that
Over a wide range of tasks, we show that the lower the perplexity of the prompt (i.e. closer to zero) is, the better the prompt is able to perform the task.
To get started with betterprompt, you will need to first install the library by running the following command:
pip install betterprompt
Once installed, you can import it into your python project by running the following:
import betterprompt
betterprompt currently provides 1 main functions - calculate_perplexity.
The calculate_perplexity function takes in a prompt as a string and calculates the perplexity of the text using the OpenAI GPT-3 API. It returns the perplexity as a float.
Here is an example of how to use Betterprompt to calculate the perplexity of a given prompt:
import betterprompt
prompt = "This is a sample prompt"
perplexity = betterprompt.calculate_perplexity(prompt)
print(perplexity) # Outputs the perplexity of the text
The current version of Betterprompt is 0.4.0.
We welcome contributions to Betterprompt! Please read our contributing guidelines for more information.
Betterprompt is released under the MIT License.