Skip to content

Commit

Permalink
Add temperature sanity check as 0 is invalid (#1211)
Browse files Browse the repository at this point in the history
A user was passing 0 and this results in a divide by zero in the SoftMax
calculation, which results in +INF/NaN floating point results that are
not obviously caused by a zero temperature.

This lets users know at token selection time that the current
temperature is invalid.
  • Loading branch information
RyanUnderhill authored Feb 1, 2025
1 parent dba39b4 commit 9483f18
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions src/generators.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -446,6 +446,8 @@ void Generator::GenerateNextToken() {
throw std::runtime_error("top_p must be between 0.0 and 1.0");
if (search.top_k < 0)
throw std::runtime_error("top_k must be 0 or greater");
if (search.temperature <= 0.0f)
throw std::runtime_error("temperature must be greater than 0");

if (search.top_p > 0.0f && search.top_p < 1.0f && search.top_k > 1) {
search_->SampleTopKTopP(search.top_k, search.top_p, search.temperature);
Expand Down

0 comments on commit 9483f18

Please sign in to comment.