Diff snapshot requires two test runs for new tests and two to 'stabilise' after changes to base snapshot

[huonw]

Describe the bug

Currently, using diff seems to take two test runs with --snapshot-update to have the diff computed correctly in two cases:

• if a test is newly added with both a base snapshot and a diff, the diff snapshot will fail with a SnapshotDoesNotExist error on the first run, but work on the second
• if the "base"/original snapshot has a change, the first test run will have those changes appear in the diff, and a second run is required to clear them out

To reproduce

For a new test:

1. Create a new test like the following:

   base = {"A": 1}
   def test_foo(snapshot):
       assert {**base} == snapshot(name="a")
       assert {**base, "B": 2} == snapshot(name="b", diff="a")

2. run pytest --snapshot-update
3. BUG: observe it fails with an error like:

   >       assert {**base, "B": 2} == snapshot(name="b", diff="a")
   E       assert [+ received] == [- snapshot]
   E         syrupy.exceptions.SnapshotDoesNotExist
   E         Traceback (most recent call last):
   E           File "/private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/tmp.Kn7Y7pGLWR/.venv/lib/python3.10/site-packages/syrupy/assertion.py", line 322, in _assert
   E             raise SnapshotDoesNotExist()
   
   test_foo.py:4: AssertionError
   

4. run pytest --snapshot-update again (no other changes) and observe it works

For an existing test:

5. Take the test above (after running pytest --snapshot-update twice) and edit base to have an extra key, base = {"A": 1, "C": 3}
6. Run pytest --snapshot-update
7. BUG: observe the __snapshot__/test_foo.ambr file contains the following, and, in particular, the diff says that 'C': 3 was added... but it appears in the a snapshot too, i.e. appears in both snapshots and thus shouldn't be in the diff!

   # serializer version: 1
   # name: test_foo[a]
     dict({
       'A': 1,
       'C': 3,
     })
   # ---
   # name: test_foo[b]
         ...
     +   'B': 2,
     +   'C': 3,
       ...
   # ---
   

8. Run pytest (no --snapshot-update) and observe failures:

   >       assert {**base, "B": 2} == snapshot(name="b", diff="a")
   E       AssertionError: assert [+ received] == [- snapshot]
   E               ...
   E           +   'B': 2,
   E         - +   'C': 3,
   E             ...
   
   test_foo.py:4: AssertionError
   

An all-in-one shell script reproducer: save it and run with bash ./script.sh

cd $(mktemp -d)

python -m venv .venv
. .venv/bin/activate

pip install syrupy==4.8.0 colored==1.4.4 exceptiongroup==1.2.2 iniconfig==2.0.0 packaging==24.2 pluggy==1.5.0 pytest==7.4.4 tomli==2.2.1

cat > test_foo.py <<EOF
base = {"A": 1}
def test_foo(snapshot):
    assert {**base} == snapshot(name="a")
    assert {**base, "B": 2} == snapshot(name="b", diff="a")
EOF

echo '**** BUG: first run fails with `raise SnapshotDoesNotExist`'
pytest test_foo.py --snapshot-update
echo '**** OKAY: second run passes'
pytest test_foo.py --snapshot-update

# add a new field C to base
cat > test_foo.py <<EOF
base = {"A": 1, "C": 3}
def test_foo(snapshot):
    assert {**base} == snapshot(name="a")
    assert {**base, "B": 2} == snapshot(name="b", diff="a")
EOF

echo '**** BUG: running after a change has the diff snapshot think C is added (should only be B)'
pytest test_foo.py --snapshot-update
cat __snapshots__/test_foo.ambr
echo '**** ... which means a second non-snapshot test run fails'
pytest test_foo.py

Expected behavior

Running pytest --snapshot-update should make the complete set of updates required, i.e. assuming the snapshots are reproducible, the both:

• running pytest --snapshot-update a second time should make no changes
• running pytest alone should pass

Screenshots

Environment (please complete the following information):

• OS: macOS 15
• Syrupy Version: 4.8
• Python Version: 3.10

Additional context

Maybe very slightly related to #608

(Thanks for syrupy! 🥞 )

[noahnu]

🎉 This issue has been resolved in version 4.8.1 🎉

The release is available on:

• v4.8.1
• GitHub release

Your semantic-release bot 📦🚀