Don't ignore all exceptions during coercions for nullable fields #490

nikhaldi · 2019-05-02T17:36:12Z

Only ignore exceptions if a value for a nullable field is not null.
Previously all exceptions during custom coerction were ignored with nullable
fields. The developer and/or user most likely wants to know about
exceptions that happen during coercion - this leads to hard to debug
problems otherwise.

The purpose of the e is not TypeError condition was unclear. As far as I
know this condition can never be false (e is an instance of an exception,
not an exception class).

I think this is still in line with the original intent of this code, which
was to make common coercions like float work with nullable fields out
of the box: #269

nikhaldi · 2019-05-02T17:44:24Z

The Travis failure for test_nested_error_paths is only on Python 3.5. Unclear to me what the problem is. It doesn't reproduce for me locally on 3.5. I'm going go force push a noop change to get Travis to rerun.

nikhaldi · 2019-05-02T17:50:19Z

The same test passed on the second try, so just appears to be flaky. The failure was https://travis-ci.org/pyeve/cerberus/jobs/527412260

funkyfuture

thanks for bringing this up. could you please also update the CHANGES.rst and add yourself to the AUTHORS file?

cerberus/tests/test_normalization.py

funkyfuture · 2019-05-03T10:50:56Z

cerberus/tests/test_normalization.py

@@ -170,6 +170,15 @@ def test_nullables_dont_fail_coerce():
    assert_normalized(document, document, schema)


+def test_nullables_fail_coerce_on_non_null_values(validator):
+    schema = {'foo': {'coerce': lambda x: 1 / 0, 'nullable': True, 'type': 'integer'}}
+    document = {'foo': None}


this is already covered in the previous test.

True, but I feel the repetition is a good idea here because the new behavior is about the difference between these two cases (the value is None or not).

i'm not saying you're wrong, but we don't do that elsewhere (oh dear, formulating this makes me feel old). i'm gonna amend this when merging.

funkyfuture · 2019-05-03T10:52:17Z

cerberus/validator.py

@@ -723,7 +723,7 @@ def __normalize_coerce(self, processor, field, value, nullable, error):
        try:
            return processor(value)
        except Exception as e:
-            if not nullable and e is not TypeError:
+            if not nullable or value is not None:


i find this term more obvious: not (nullable and value is None)

second thought: should the coercion get tried at all when the part in brackets is true?

I changed the condition.

Not sure there's an obvious answer to whether the coercion should be tried or not. Somebody might want to coerce None to something else? Or would the validation in that case fail at a later stage anyway?

I just realized that NOT trying the coercion for nullable fields when the value is None would fix this issue: #427

To be honest, I don't feel I'm in a position to make the right call here. So happy to make this change or not.

thanks for digging this out. that issue concluded to postpone changes until the 2.0 release; and i think i shall document that this still needs to be layed out design-wise.

please don't mark this conversation as resolved for visibility.

funkyfuture · 2019-05-03T10:59:27Z

The same test passed on the second try, so just appears to be flaky.

yes, Python 2.7 and 3.5 sort errors kind of erraticly. no worries.

nikhaldi

Thanks for the quick review! I updated AUTHORS and CHANGES too

cerberus/tests/test_normalization.py

nikhaldi · 2019-05-06T16:38:00Z

cerberus/tests/test_normalization.py

@@ -170,6 +170,15 @@ def test_nullables_dont_fail_coerce():
    assert_normalized(document, document, schema)


+def test_nullables_fail_coerce_on_non_null_values(validator):
+    schema = {'foo': {'coerce': lambda x: 1 / 0, 'nullable': True, 'type': 'integer'}}
+    document = {'foo': None}


True, but I feel the repetition is a good idea here because the new behavior is about the difference between these two cases (the value is None or not).

nikhaldi · 2019-05-06T16:40:32Z

cerberus/validator.py

@@ -723,7 +723,7 @@ def __normalize_coerce(self, processor, field, value, nullable, error):
        try:
            return processor(value)
        except Exception as e:
-            if not nullable and e is not TypeError:
+            if not nullable or value is not None:


I changed the condition.

Not sure there's an obvious answer to whether the coercion should be tried or not. Somebody might want to coerce None to something else? Or would the validation in that case fail at a later stage anyway?

Only ignore exceptions if a value for a nullable field is not null. Previously all exceptions during custom coerction were ignored with nullable fields. The developer and/or user most likely wants to know about exceptions that happen during coercion - this leads to hard to debug problems otherwise. The purpose of the `e is not TypeError` condition was unclear. As far as I know this condition can never be false (`e` is an instance of an exception, not an exception class). I think this is still in line with the original intent of this code, which was to make common coercions like `float` work with nullable fields out of the box: pyeve#269

nikhaldi · 2019-05-06T16:55:50Z

Side note, test_nested_error_paths has failed 4 out of 5 times on Travis for me now. I'm not going to force a Travis rerun again until I know this is ready to merge.

funkyfuture · 2019-05-06T18:10:41Z

has failed 4 out of 5 times on Travis for me now

now, that's bad luck. but just ignore it.

nikhaldi force-pushed the fix/nullable-coercions branch from f5ece6b to 7921688 Compare May 2, 2019 17:45

funkyfuture requested changes May 3, 2019

View reviewed changes

nikhaldi force-pushed the fix/nullable-coercions branch from 7921688 to 2797119 Compare May 6, 2019 16:40

nikhaldi commented May 6, 2019

View reviewed changes

nikhaldi force-pushed the fix/nullable-coercions branch from 2797119 to 299bd92 Compare May 6, 2019 16:46

nikhaldi force-pushed the fix/nullable-coercions branch from 299bd92 to c51059d Compare May 6, 2019 16:52

funkyfuture mentioned this pull request May 6, 2019

Cerberus 2: Coercion of nullable fields needs to (re)defined #491

Closed

funkyfuture merged commit c51059d into pyeve:master May 6, 2019

funkyfuture approved these changes May 6, 2019

View reviewed changes

nicolaiarocci added a commit that referenced this pull request May 7, 2019

Fix tiny changelog issue introduced with #490

ba77291

dependabot bot mentioned this pull request Dec 23, 2020

build(deps): bump cerberus from 1.2 to 1.3.2 in /automan tier4/AutomanTools#114

Merged

dependabot bot mentioned this pull request Jan 25, 2021

build(deps): bump cerberus from 1.2 to 1.3.2 in /automan TakehiroMatsumoto/AutomanTools#9

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't ignore all exceptions during coercions for nullable fields #490

Don't ignore all exceptions during coercions for nullable fields #490

nikhaldi commented May 2, 2019

nikhaldi commented May 2, 2019

nikhaldi commented May 2, 2019

funkyfuture left a comment

funkyfuture May 3, 2019

nikhaldi May 6, 2019

funkyfuture May 6, 2019

funkyfuture May 3, 2019

funkyfuture May 3, 2019

nikhaldi May 6, 2019

nikhaldi May 6, 2019

funkyfuture May 6, 2019 •

edited

Loading

funkyfuture commented May 3, 2019

nikhaldi left a comment

nikhaldi May 6, 2019

nikhaldi May 6, 2019

nikhaldi commented May 6, 2019

funkyfuture commented May 6, 2019

Don't ignore all exceptions during coercions for nullable fields #490

Don't ignore all exceptions during coercions for nullable fields #490

Conversation

nikhaldi commented May 2, 2019

nikhaldi commented May 2, 2019

nikhaldi commented May 2, 2019

funkyfuture left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

funkyfuture May 6, 2019 • edited Loading

Choose a reason for hiding this comment

funkyfuture commented May 3, 2019

nikhaldi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikhaldi commented May 6, 2019

funkyfuture commented May 6, 2019

funkyfuture May 6, 2019 •

edited

Loading