Are there any libs that implement schema validation in a streaming fashion, e.g., not loading the whole object in memory?

[mezhaka]

I have put a 400 experience point bounty on that question (https://stackoverflow.com/q/79389053/18775) with some valuable comments from @jeremyfiel (btw. I could not join slack channel for some reason). I just wanted to cross-post it here in hope to get more details on that topic. I appologize if this is not the place to discuss such matters.

I have a large JSON that I do not want to load into memory. I would like to validate it against a JSON schema in a streaming fashion. All libraries I could find so far, only validate completely loaded JSON objects (like Pydantic or https://github.com/python-jsonschema/jsonschema). What I rather need is some way to validate it feeding the original JSON chunk by chunk, i.e, control the size of the buffer.

This could look like this:

import pydantic  # I use V2
import ijson
import pathlib

class A(pydantic.BaseModel):
    i: int
    a: list[int]
    s: str

jsonpath = pathlib.Path("some.json")
validator = MyValidator(schema=A.model_json_schema())
with jsonpath.open("rb") as file:
    for prefix, event, value in ijson.parse(file, use_float=True):
        validator.event((prefix, event, value))

print(validator.errors)

Imagine some.json file is ~50 MB large A instance with a very long array. I do not want to load the whole object into memory (this is what Pydantic would do), but I want to make sure that some.json complies with the schema of A. The validator.errors could give me a list of errors which is going to be empty in cases none where discovered.

[EDIT 2025-01-31] The term "streaming fashion" means that event is seen exactly once and there is no way to see it again. However, I am willing to accept an answer if there is a way to do the validation with multiple scans, i.e., in my example above multiple file scans are fine with me.

[jeremyfiel]

Try here. https://join.slack.com/t/json-schema/shared_invite/zt-1rptj13ch-ZygpGVq7jUxQYpct0vwsGQ

[jdesrosiers]

@awwright wrote a streaming validator a while back. It's in JavaScript, so you probably won't be able to use it. But, it shows that it's possible. I don't remember if there were any limitations.