Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Basic Token Usage Tracking #8

Open
Blaizzy opened this issue Jul 11, 2024 · 1 comment
Open

Implement Basic Token Usage Tracking #8

Blaizzy opened this issue Jul 11, 2024 · 1 comment
Labels
good first issue Good for newcomers help wanted Extra attention is needed

Comments

@Blaizzy
Copy link
Collaborator

Blaizzy commented Jul 11, 2024

Description:

We'd like to add a simple token usage tracking feature to our FastMLX application. This will help users understand how many tokens their requests are consuming.

Objective:

Implement a function that counts the number of tokens in the input and output of our AI models.

Tasks:

  1. Create a new function count_tokens(text: str) -> int in the utils.py file.
  2. Use the appropriate tokenizer from our AI model to count tokens.
  3. Integrate this function into the main request processing flow in main.py.
  4. Update the response structure to include token counts.

Example Implementation:

from transformers import AutoTokenizer

def count_tokens(text: str) -> int:
    tokenizer = AutoTokenizer.from_pretrained("gpt2")  # or use our model's tokenizer
    return len(tokenizer.encode(text))

# In main request processing:
input_tokens = count_tokens(user_input)
output_tokens = count_tokens(model_output)
total_tokens = input_tokens + output_tokens

response = {
    "output": model_output,
    "usage": {
        "prompt_tokens": input_tokens,
        "completion_tokens": output_tokens,
        "total_tokens": total_tokens
    }
}

Guidelines:

  • Focus on basic functionality first. We can optimize later.
  • Make sure to handle potential errors, like invalid inputs.
  • Add comments to explain your code.
  • If you're unsure about anything, feel free to ask questions in the comments!

Resources:

Definition of Done:

  • Function implemented and integrated into main flow.
  • Response includes token usage information.
  • Basic error handling is in place.
  • Code is commented and follows our style guide.

We're excited to see your contribution! This feature will help our users better understand and manage their token usage. Good luck!

@Blaizzy Blaizzy added help wanted Extra attention is needed good first issue Good for newcomers labels Jul 11, 2024
@Blaizzy Blaizzy pinned this issue Jul 15, 2024
@antunsz
Copy link

antunsz commented Jul 17, 2024

Hi @Blaizzy . Perhaps this issue could be an opportunity to implement LLM tracking using AgentOps [https://github.com/AgentOps-AI/agentops] (for example). Or do you see this as a step for the future, or is that not the right approach? What are your thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants