-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Chat template error with MLX Community LLava models (moved from FastMLX) #51
Comments
Hey @stewartugelow I still haven't managed to replicate your issue. Here is what I did:
And still works as expected on two difference machines. mlx-community/llava-phi-3-mini-4bitmlx-community/llava-llama-3-8b-v1_1-8bitmlx-community/llava-1.5-7b-4bit |
Pip list | grep mlx fastmlx 0.1.0
mlx 0.15.2
mlx-lm 0.16.0 /Users/prince_canuma/Documents/Projects/LLMs/mlx-lm/llms
mlx-vlm 0.0.11 |
@stewartugelow could you share the output of: from mlx_vlm.utils import load
model_path = "mlx-community/llava-phi-3-mini-4bit"
model, processor = load(model_path)
print(processor.__dict__) and prompt = processor.tokenizer.apply_chat_template(
[{"role": "user", "content": f"<image>What are these?"}],
tokenize=False,
add_generation_prompt=True,
)
print(prompt) |
I have a similar issue:
|
@BoltzmannEntropy could you share the version of mlx-vlm you aer running? |
Sure:
|
@BoltzmannEntropy the problem is fixed. It was a missing key in the config :) |
While this works:
I never had to authenticate before |
It's not a bug. You have to access the model config of a gated model :) |
|
This is a gradio problem. They had many breaking changes recently. I will fix it on the next release tomorrow and pin the version to avoid such cases. |
@stewartugelow I managed to replicate your issue as well. And will address it : |
@stewartugelow @BoltzmannEntropy |
Update gradio: pip install -U gradio |
|
You didn't install from source. To install from source, first clone the branch then run
|
Continued from: https://github.com/Blaizzy/fastmlx/issues/6
When I tried this at the command line: "python -m mlx_vlm.chat_ui --model mlx-community/llava-1.5-7b-4bit", I get the same chat template errors with all of the following:
models--mlx-community--llava-1.5-7b-4bit
models--mlx-community--llava-llama-3-8b-v1_1-8bit
models--mlx-community--llava-phi-3-mini-4bit
models--mlx-community--llava-v1.6-mistral-7b-8bit
Logs:
mlx-community/llava-1.5-7b-4bit
mlx-community/llava-v1.6-mistral-7b-8bit
mlx-community/llava-llama-3-8b-v1_1-8bit
mlx-community/llava-phi-3-mini-4bit
The text was updated successfully, but these errors were encountered: