schema checks and unit tests.

[WaVEV]

I added the schema checks in get_transform, I didn't find any other place to put it.

The code got a bit dirty if we use embedded field with jsonfield. This use case is strange but possible.
I couldn't find any other way to do this task without breaking JSON.

[timgraham]

Sorry for the delay in looking at this. Let's focus on the field validation (first commit) and move the querying bits to another PR.

[WaVEV]

Sorry for the delay in looking at this. Let's focus on the field validation (first commit) and move the querying bits to another PR.

I reorder the commits to have the first 5 for validation and the other 2 for testing.

[timgraham]

I rebased embedded-model-field branch on main, so if you rebase your branch on that, it should fix the issue.

[timgraham]

Refresh my memory... we can't support foreign key because of the double underscore lookup syntax collision, correct? In that case, I'm wondering if we'll need to revisit how we do querying in order to allow foreign key (even if it's a future task) in embedded documents. (I'm not sure whether or not we need to support it... have to check with Jib + team.)

[WaVEV]

Refresh my memory... we can't support foreign key because of the double underscore lookup syntax collision, correct? In that case, I'm wondering if we'll need to revisit how we do querying in order to allow foreign key (even if it's a future task) in embedded documents. (I'm not sure whether or not we need to support it... have to check with Jib + team.)

🤔 Yes, the problem is when Django interprets the query. The setup_joins function handles it, but it always treats the __ as joins (if it’s not a lookup). So, if a join comes before a path query into the embedded field, it works, but not the reverse (a join using an embedded field's field) because a different flow handles the query interpretation. If we want to support this kind of behavior, we should treat the embedded fields as fake-joins.

I don’t know if we can define a foreign key as a field in a JSONField. I don’t think so, but if it’s possible, we should take a look.

[timgraham]

No foreign key in JSON. It's always just "data."

See Djongo's EmbeddedField API for an alternate query syntax we could consider.

[WaVEV]

No foreign key in JSON. It's always just "data."

See Djongo's EmbeddedField API for an alternate query syntax we could consider.

🤔 This changes things a bit, I didn’t know this kind of syntax existed 😬. I have two questions:

In Django, filtering usually works with an argument structured as column_field + lookup, and the value is just the value to filter by. But here, we have a mixed approach where the value part includes both field + value, while the argument name is embedded_field + lookup. Is that correct?
The tests I wrote are based on the other format, so they’re not valid for this one.
Also, even though I don’t know how to implement a foreign field inside an embedded field, how would this syntax support such queries? I can think of two possible options:

entries = Entry.objects.filter(embedded_field__foreign_field__name__startswith={'name': 'Beatles'})
entries = Entry.objects.filter(embedded_field__startswith={'foreign_field__name': 'Beatles'})
Which one (if either) would we use?

[Jibola]

Refresh my memory... we can't support foreign key because of the double underscore lookup syntax collision, correct? In that case, I'm wondering if we'll need to revisit how we do querying in order to allow foreign key (even if it's a future task) in embedded documents. (I'm not sure whether or not we need to support it... have to check with Jib + team.)

I don't think it needs to be supported as I'd rather we remove any opportunities to promote the ForeignKey field usage; however, having a task to investigate its work sounds just fine to me.

[Jibola]

I added the schema checks in get_transform, I didn't find any other place to put it.

The code got a bit dirty if we use embedded field with jsonfield.

What would be a use-case you'd imagine? If someone is using an EmbeddedModel in a JSONField that feels counterintuitive.

I'm all for not supporting this usecase if it's too much work; work has already been done to support it, but if it's creating complications, you can take out the requirement. If someone is using EmbeddedField with JSONField, I'm hard-pressed to understand if that's worthwhile.

[Jibola]

Added some comments. I haven't reviewed all the test cases yet, but will come back to it today. :)

[Jibola]

shouldn't this be city has no field name president? or address has no field named city.president?

[WaVEV]

🤔 I think you are right. I will check this out.

[WaVEV]

I used "Address.city has no field named 'president'" if you don't like it, let me know is a easy change

[timgraham]

It could be a mistyped field OR transform, so the message should reflect that. It would be neat to replicate the logic here which could offer suggestions for similar lookups AND fields.
https://github.com/django/django/blob/40d5516385448a73426aad396778f369a363eda9/django/db/models/sql/query.py#L1440-L1452

[WaVEV]

I added the schema checks in get_transform, I didn't find any other place to put it.
The code got a bit dirty if we use embedded field with jsonfield.

What would be a use-case you'd imagine? If someone is using an EmbeddedModel in a JSONField that feels counterintuitive.

I'm all for not supporting this usecase if it's too much work; work has already been done to support it, but if it's creating complications, you can take out the requirement. If someone is using EmbeddedField with JSONField, I'm hard-pressed to understand if that's worthwhile.

I agree, the use case is unusual, and maybe we should suggest avoiding it. But as long as the embedded fields are fully typed, the user might have a reason for wanting semi-structured data inside embedded field.

[timgraham]

This work has been integrated in PRs to the main repo.