[smolagents] llm input messages are empty

[andstor]

Describe the bug
huggingface/smolagents#220 modifies llm message contents from a simple string to a list of subcontent.

from:

messages = [
  {"role": "user", "content": "I'm doing great. How can I help you today?"},
]

to:

messages = [
  {"role": "user", "content": [{"type": "text", "text": "I'm doing great. How can I help you today?"},
{"type":"image"}
]

To Reproduce
Follow standard tutorial on huggingface.
https://github.com/huggingface/smolagents/blob/022947a2a587483ac897f403f52a1e0a50f53667/docs/source/en/tutorials/inspect_runs.md

This results in the following error:

Invalid type dict in attribute 'llm.input_messages.0.message.content' value sequence. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or None

Screenshots

[dosubot]

Hey @andstor! I'm here to help you with any bugs, questions, or contributions you have for huggingface/smolagents. Let's tackle this issue together!

The error you're encountering is due to the change in the message content structure from a simple string to a list of subcontent dictionaries. The Arize Phoenix framework expects the llm.input_messages to be a list of objects where each message's content is a string, not a list of dictionaries ^[1].

To resolve this, you need to adjust the message content to match the expected structure. You can either:

1. Convert the list of subcontent dictionaries into a single string before passing it to Arize Phoenix.
2. Modify the processing logic in your application to handle the new structure if you have control over the Arize Phoenix integration.

If you need to maintain the new structure for other parts of your application, consider creating a separate processing step specifically for compatibility with Arize Phoenix.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[nate-mar]

Thanks for reporting! Working on a fix for this. Hang tight! :D