sqlmodel doesn't merge the fields in Annotated[SomeType, Field(...), Field(...)] (but pydantic does)

[ryangalamb]

First Check

• I added a very descriptive title here.
• I used the GitHub search to find a similar question and didn't find it.
• I searched the SQLModel documentation, with the integrated search.
• I already searched in Google "How to X in SQLModel" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to SQLModel but to Pydantic.
• I already checked if it is not related to SQLModel but to SQLAlchemy.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from typing import Annotated

import pydantic
import sqlmodel


class PydanticExample(pydantic.BaseModel):
    value: Annotated[
        int,
        pydantic.Field(description="some-description"),
        pydantic.Field(title="some-title"),
    ]


print(f"PydanticExample  {PydanticExample.model_fields['value']!r}")
#> PydanticExample  FieldInfo(annotation=int, required=True, title='some-title', description='some-description')


class SQLModelExample(sqlmodel.SQLModel):
    value: Annotated[
        int,
        sqlmodel.Field(description="some-description"),
        sqlmodel.Field(title="some-title"),
    ]


print(f"SQLModelExample  {SQLModelExample.model_fields['value']!r}")
#> SQLModelExample  FieldInfo(annotation=int, required=True, title='some-title')

Description

If multiple Fields are on the same attribute (e.g., via Annotated), then sqlmodel should merge the values, just like how pydantic does. But sqlmodel does not merge the fields together.

Operating System

Linux

Operating System Details

No response

SQLModel Version

0.0.22

Python Version

3.12

Additional Context

Why is this important?

Because it lets me abstract away db implementation details while still being able to the pydantic fields. If merging fields worked, I could do cool ✨ stuff ✨ like ✨ this ✨

# Create a base annotation that handles the fiddly sqlalchemy bits
MyDateTime = Annotated[
    datetime.datetime,
    Field(sa_column=sa.Column(sa.DATETIME(timezone=True).with_variant(mysql.DATETIME(timezone=True, fsp=6)))),
]

class SomeObject(SQLModel):
    create_time: Annotated[
        MyDateTime,  # Annotate with extra fields to customize further
        Field(default_factory=lambda: datetime.datetime.now(datetime.timezone.utc), nullable=False),
    ]

    other_time: Annotated[
        MyDateTime,
        Field(index=True),
    ]

How do we fix this?

I've already worked out why this isn't working in sqlmodel (even though it works in pydantic)

• In sqlmodel.Field all the default values are None or other valid values.
• In pydantic.Field all the default values are PydanticUndefined

When pydantic creates a FieldInfo object, it creates a self._attributes_set containing all the keys/values where the values aren't PydanticUndefined. That's what's being merged when there's multiple Fields. Since sqlmodel.Field always creates a fully populated FieldInfo, pydantic acts as if we had explicitly None'd out all the unset properties in each Field.

After replacing all the defaults in sqlmodel.Field with PydanticUndefined (except for schema_extra because it gets **'d), my example produces the same merged FieldInfo for both pydantic and sqlmodel 🎉

I will add tests and put this into a PR when I get a chance, hopefully early next week.

[ryangalamb]

Quick update: My initial approach of replacing all the defaults in Field with PydanticUndefined is not enough.

Whenever pydantic merges multiple Field annotations, it puts the merged values into a new pydantic.FieldInfo (losing the sqlmodel-specific attributes.)
pydantic.Field.merge_field_infos is a staticmethod that always creates/returns a pydantic.FieldInfo when it merges files.

I can hack around it in sqlmodel's code by using mock.patch to replace pydantic.fields.FieldInfo with sqlmodel.FieldInfo right before pydantic merges the annotations. Pretty unhinged, but hey it works.

We could also submit a PR to pydantic to use the class of the first FieldInfo object that it's merging (instead of hardcoding it to be pydantic.FieldInfo.) It's like a 2 line change, and it feels in keeping with the rest of the method. (When it "merges" one FieldInfo, it creates a copy and updates that, preserving the original class.)

tl;dr I see two options:

1. Hack around it with mock.patch. It's hacky, but effective.
2. Try to change it in pydantic. Assuming they're cool with the change (they may not be), it further restricts the versions of pydantic that sqlmodel is compatible with.

@tiangolo Do you have a preference? I'm happy to work on whichever approach you prefer.

I'll submit a PR with the mock.patch hack, so you can get a better idea of what I'm talking about.