-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
评测mmbench_video会报错 #530
Comments
您好 @helloworld01001 ,我看到我们并不支持llavaonevision-qwen这个模型,想问一下您具体使用的是下列的哪个模型呢~ |
您好我用的是qwen2vl跑的videomme,能不能单独开帖子问您 |
想问一下您报错的时候,build_prompt_nopack这个函数传入的line是什么参数类型呢 |
是字符串类型的,build_prompt_nopack中 line 的类型是: <class 'str'>,可是InternVL2-2B和MiniCPM-V-2可以正常评测 |
还有就是评测InternVL2-2B的时候是不是没有调用build_prompt_nopack函数,因为没有打印我调试的代码 |
是的,InternVL2-2B是走的模型自身的build_custom_prompt这个函数 |
有line的具体信息吗(是怎么样一个字符串),因为之前确实没有遇到过这种情况 |
你好,我在评测mmbench_video会报以下错误,请问该怎么解决?CUDA_VISIBLE_DEVICES=5 python /share/project/open-compass_1011/VLMEvalKit/run.py --data MMBench-Video --model llavaonevision-qwen --work-dir /share/project/open-compass_1011/VLMEvalKit/test-results/llava_onevision_qwen --nframe 16 --pack
0it [00:00, ?it/s]
2024-10-19 14:27:49,039 - RUN - ERROR - Model llavaonevision-qwen x Dataset MMBench-Video combination failed: string indices must be integers, skipping this combination.
Traceback (most recent call last):
File "/share/project/open-compass_1011/VLMEvalKit/run.py", line 121, in main
model = infer_data_job_video(
File "/share/project/open-compass_1011/VLMEvalKit/vlmeval/inference_video.py", line 143, in infer_data_job_video
model = infer_data(
File "/share/project/open-compass_1011/VLMEvalKit/vlmeval/inference_video.py", line 94, in infer_data
struct = dataset.build_prompt(
File "/share/project/open-compass_1011/VLMEvalKit/vlmeval/dataset/mmbench_video.py", line 139, in build_prompt
return self.build_prompt_nopack(line, num_frames, video_llm)
File "/share/project/open-compass_1011/VLMEvalKit/vlmeval/dataset/mmbench_video.py", line 120, in build_prompt_nopack
question = line['question']
TypeError: string indices must be integers
The text was updated successfully, but these errors were encountered: