I tested this in a fork of the deltalake repo but could not reproduce the error:

DeltaLake Version: 0.23.0
Polars Version: 1.20.0

@kosiew you are testing it against an older version of deltalake from what I can see in the commit: https://github.com/delta-io/delta-rs/blob/d8080b13f5724aa09fb268b17f507dfd8559255f/python/pyproject.toml

In that commit you can see it was Datafusion v43, we are now at datafusion v44.

Good catch @ion-elgreco ☝!
I managed to reproduce the error after fetching the latest main from the upstream.

I investigated the delta-rs repo, because I earlier tested the coercion in datafusion v44, but could not trigger the error:

https://github.com/kosiew/datafusion/blob/f845a233cb90758b42c0ab2d3ea1c1e02da36aa3/datafusion/optimizer/src/analyzer/type_coercion.rs#L2140-L2199

Given the description on this PR it appears it is a regression (something that used to work, but now does not). Is this the case?

I'll put it on the list of things to investigate before the 45 relese

@alamb I'll have to double check, but I think it worked in DF43

@alamb yes this indeed worked fine on DF43. I tested it against deltalake v0.23.3

Possibly related:

• Support List type coercion for CASE-WHEN-THEN expression #12490 (released in DataFusion 44.0.0)
• Coerce Array inner types #13452 (also released in DataFusion 44.0.0)

Ao the next step for this PR is to find a DataFusion only reproducer that works in DF 43 but not in DF 44. I will try to do so tomorrow

I tried to make a datafusion only reproducer but it turns out I can't create LargeLists of structs via SQL 🤔 I'll have to think about how to do so a bit more tomorrow...

Well, I found another bug when working on this one

• Case coercion of Structs loses field names  #14383

I still haven't found a reproducer for this one. But I have another idea

Ok, I have a datafusion only reproducer:

create or replace table t as values
(
 100,                                         -- column1 int (so the case isn't constant folded)
 [{ 'foo': arrow_cast('baz', 'Utf8View') }],  -- column2 has List of Struct w/ Utf8View
 [{ 'foo': 'bar' }],                          -- column3 has List of Struct w/ Utf8
 [{ 'foo': 'blarg' }]                         -- column4 has List of Struct w/ Utf8
);

SELECT column2, column3, column4  FROM t;

SELECT
  case
    when column1 > 0 then column2
    when column1 < 0 then column3
    else column4
  end
FROM t;

Fails with

Error during planning: Failed to coerce then ([List(Field { name: "item", data_type: Struct([Field { name: "foo", data_type: Utf8View, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Struct([Field { name: "foo", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })]) and else (Some(List(Field { name: "item", data_type: Struct([Field { name: "foo", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }))) to common types in CASE WHEN expression

It works with datafusion-cli on 43:

> SELECT
  case
    when column1 > 0 then column2
    when column1 < 0 then column3
    else column4
  end
FROM t;
+-----------------------------------------------------------------------------------------------------------+
| CASE WHEN t.column1 > Int64(0) THEN t.column2 WHEN t.column1 < Int64(0) THEN t.column3 ELSE t.column4 END |
+-----------------------------------------------------------------------------------------------------------+
| [{foo: baz}]                                                                                              |
+-----------------------------------------------------------------------------------------------------------+
1 row(s) fetched.

This works fine with only structs, just not with structs in lists 🤔

I think this PR fixes the issue (it is related, but not the same as #14383)

• Fix regression list Type Coercion List with inner type struct which has large/view types #14385