Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: runtime configuration of vision-capable models #5919

Merged
merged 7 commits into from
Dec 22, 2024
Merged

feat: runtime configuration of vision-capable models #5919

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a433d16
feat: use regex patterns for vision models and allow adding capabilit…
Yiming3 Dec 10, 2024
74c4711
Merge remote-tracking branch 'upstream/main' into feature/flexible-vi…
Yiming3 Dec 20, 2024
ed8c358
test: add unit tests for isVisionModel utility function
Yiming3 Dec 21, 2024
210b29b
refactor: remove NEXT_PUBLIC_ prefix from VISION_MODELS env var
Yiming3 Dec 21, 2024
149d732
Merge remote-tracking branch 'upstream/main' into feature/flexible-vi…
Yiming3 Dec 21, 2024
ea1329f
fix: add optional chaining to prevent errors when accessing visionModels
Yiming3 Dec 21, 2024
a127ae1
docs: add VISION_MODELS section to README files
Yiming3 Dec 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions app/constant.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,21 @@ export const DEFAULT_TTS_VOICES = [
"shimmer",
];

export const VISION_MODEL_REGEXES = [
/vision/,
/gpt-4o/,
/claude-3/,
/gemini-1\.5/,
/gemini-exp/,
/learnlm/,
/qwen-vl/,
/qwen2-vl/,
/gpt-4-turbo(?!.*preview)/, // Matches "gpt-4-turbo" but not "gpt-4-turbo-preview"
/^dall-e-3$/, // Matches exactly "dall-e-3"
];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

πŸ’‘ Codebase verification

Let me generate the final response based on the test results.

Regex Patterns in VISION_MODEL_REGEXES Need Adjustments

The test results reveal some issues with the regex patterns:

  • The /gpt-4-turbo(?!.*preview)/ pattern is not working as intended in bash regex - it's not matching "gpt-4-turbo" as expected
  • The /gemini-1\.5/ pattern works correctly, matching "gemini-1.5-pro"
  • The /^dall-e-3$/ pattern works as expected, matching exactly "dall-e-3" but not "dall-e-3-beta"

Consider:

  • Using a simpler negative pattern for gpt-4-turbo, like ^gpt-4-turbo$ to match exactly "gpt-4-turbo"
  • Adding test cases for other patterns (vision, gpt-4o, claude-3, etc.) to ensure they match their intended targets
πŸ”— Analysis chain

Verify Correctness of VISION_MODEL_REGEXES Patterns

Ensure that each regex in VISION_MODEL_REGEXES accurately matches the intended vision models without unintended matches. Pay special attention to:

  • /gemini-1\.5/: Confirm that the backslash is correctly escaping the dot to match gemini-1.5.
  • /gpt-4-turbo(?!.*preview)/: Ensure that this pattern matches "gpt-4-turbo" but not "gpt-4-turbo-preview".

Run the following script to test the regex patterns against a list of model names:

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Test VISION_MODEL_REGEXES against existing model names

# Define model names for testing
model_names=(
  "gpt-4-turbo"
  "gpt-4-turbo-preview"
  "gemini-1.5-pro"
  "gemini-1x5-pro"
  "dall-e-3"
  "dall-e-3-beta"
)

# Define regex patterns
regex_patterns=(
  "vision"
  "gpt-4o"
  "claude-3"
  "gemini-1\\.5"
  "gemini-exp"
  "learnlm"
  "qwen-vl"
  "qwen2-vl"
  "gpt-4-turbo(?!.*preview)"
  "^dall-e-3$"
)

# Test each model name against each regex
for model in "${model_names[@]}"; do
  for pattern in "${regex_patterns[@]}"; do
    if [[ $model =~ $pattern ]]; then
      echo "Model '$model' matches regex '$pattern'"
    fi
  done
done

Length of output: 5253


export const EXCLUDE_VISION_MODEL_REGEXES = [/claude-3-5-haiku-20241022/];

const openaiModels = [
"gpt-3.5-turbo",
"gpt-3.5-turbo-1106",
Expand Down
29 changes: 9 additions & 20 deletions app/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { RequestMessage } from "./client/api";
import { ServiceProvider } from "./constant";
// import { fetch as tauriFetch, ResponseType } from "@tauri-apps/api/http";
import { fetch as tauriStreamFetch } from "./utils/stream";
import { VISION_MODEL_REGEXES, EXCLUDE_VISION_MODEL_REGEXES } from "./constant";

export function trimTopic(topic: string) {
// Fix an issue where double quotes still show in the Indonesian language
Expand Down Expand Up @@ -252,27 +253,15 @@ export function getMessageImages(message: RequestMessage): string[] {
}

export function isVisionModel(model: string) {
// Note: This is a better way using the TypeScript feature instead of `&&` or `||` (ts v5.5.0-dev.20240314 I've been using)

const excludeKeywords = ["claude-3-5-haiku-20241022"];
const visionKeywords = [
"vision",
"gpt-4o",
"claude-3",
"gemini-1.5",
"gemini-exp",
"learnlm",
"qwen-vl",
"qwen2-vl",
];
const isGpt4Turbo =
model.includes("gpt-4-turbo") && !model.includes("preview");

const envVisionModels = process.env.NEXT_PUBLIC_VISION_MODELS?.split(",").map(
Yiming3 marked this conversation as resolved.
Show resolved Hide resolved
(m) => m.trim(),
);
if (envVisionModels?.includes(model)) {
return true;
}
Dogtiti marked this conversation as resolved.
Show resolved Hide resolved
return (
!excludeKeywords.some((keyword) => model.includes(keyword)) &&
(visionKeywords.some((keyword) => model.includes(keyword)) ||
isGpt4Turbo ||
isDalle3(model))
!EXCLUDE_VISION_MODEL_REGEXES.some((regex) => regex.test(model)) &&
VISION_MODEL_REGEXES.some((regex) => regex.test(model))
);
}

Expand Down