[Bug] 图片格式识别错误导致 Claude 模型视觉对话可能出现的潜在问题

[breakstring]

📦 部署环境

Docker

📌 软件版本

1.46.3

💻 系统环境

macOS

🌐 浏览器

Other

🐛 问题描述

从别的地方复制过来的图片/上传的图片，在使用 Claude 3.5 进行视觉功能来进行图片识别时，如果图片的源格式不是 png 则会报错。

📷 复现步骤

1. 从一个微信聊天中复制一个.jpg 格式的文件，粘贴到 claude 3.5 的对话中；或者直接上传一个.jpg 后缀的图片到对话中。然后输入视觉识别方面的提示词。
2. 返回错误：

{
  "error": {
    "error": {
      "type": "error",
      "error": {
        "type": "invalid_request_error",
        "message": "messages.0.content.1.image.source.base64.data: Image does not match the provided media type image/png"
      }
    },
   ......
  },
  "endpoint": "https://api.anthropic.com",
  "provider": "anthropic"
}

🚦 期望结果

尽管上传的是 jpg 格式的文件，但是似乎都被以 png 格式的 media type 发给了 Claude。
希望能够正确发送媒体格式到 claude(或者如果 claude 那边不接受除了 png 之外的图片的话能够给转换一下发过去？)。

📝 补充信息

顺便，Azure OpenAI 的对话中同一个图片似乎正常，是两边发送图片的方式不一样么？

[lobehubbot]

👀 @breakstring

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.
非常感谢您提交 issue。我们会尽快调查此事，并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

[dosubot]

I found a similar closed issue that might be relevant to your problem: [Bug] Claude 3.5 在图片的内容分析上有问题. This issue was resolved by a pull request that fixed the handling of image URLs with S3 ^[1].

Regarding your issue, it seems that the lobe-chat framework determines the media type of an image by using the parseDataUri function. If the image is not in the expected format, it might be incorrectly sent as a PNG media type, which could be causing the error you are encountering ^[2].

To address this, you might want to ensure that the image format is correctly identified and converted if necessary before sending it to Claude. If Claude only accepts PNG images, implementing a conversion step in the process flow might be necessary.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}