Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Token usage mismatch between Kernel token usage and custom HttpClient response #10263

Open
CodePlacer opened this issue Jan 22, 2025 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@CodePlacer
Copy link

we noticed a discrepancy in token usage between the values retrieved using custom http client and the token usage reported in the agent's chat response.

Preproducing Steps:

  1. Create a Kernel Object
  2. Import Lights plugin to kernel and set ToolCallBehavior to AutoInvoke
  3. Prompt to turn on the lamp.
  4. Note the token usage from function result.
  5. After 4th step, pass custom http client handler to kernel object and parse the response object and track the token usage from response body.

On Analysis it look like , the Token Usage available in kernel object is only showing the usage of last LLM calls. Token usage of tool calls are not captured properly.

Sample code is added to https://github.com/CodePlacer/Semantic-Kernel/tree/main/TokenDiff

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants