Spintest task with wrong format does not seems to return to correct error

[TitouanTheophile]

I have encounter an error will doing some simple test with the library.

I'm using

python==3.9.7
spintest==0.4.2

I'm testing a dummy API having it running or not does not affect the behavior that I'm encountering.

Here is my python file

from spintest import spintest
from spintest.types import Int

urls = ["http://localhost:8080/"]
task = [{
    "method": "POST",
    "route": "rollback/",
    "output": "rollback_doc",
    "body": {"task": "string"},
    "expected": {
        "code": 201,
        "body": {
            "task": "string",
            "id": None
        },
        "expected_match": "partial"
    },
    "ignore": True
}, {
    "method": "GET",
    "route": "rollback/5",
    "expected": {
        "code": 200,
        "body": {
            "task": 'string',
            "id": Int("{{ rollback_doc['id']}}"),
            "status": "RUNNING"
        },
        "expected_match": "partial"
    },
    "fail_on": {
        "expected_match": "partial",
        "body": {"status": "ERROR"}
    },
    "rollback": ["delete_rollback"]
}, {
    "name": "delete_rollback",
    "method": "DELETE",
    "route": "rollback/5",
    "expected": {"code": 204}
}]
result = spintest(urls, task)
assert True is result

My tasks have 3 steps

• Creating a resource with a POST -> no issue expected (ignore: True so that you don't need to have a running API)
• Retrieving the resource with a wrong status -> Should trigger the rollback but will fail due to parsing issue
• Deleting the resource -> Should always delete

When i'm executing the task i'm getting the following output

 ERROR - {
    "name": null,
    "status": "FAILED",
    "timestamp": "Thu Apr 21 10:14:59 2022",
    "duration_sec": 0.03,
    "url": "http://localhost:8080/",
    "route": "rollback/",
    "message": "Request failed.",
    "code": null,
    "body": null,
    "task": {
        "method": "POST",
        "route": "rollback/",
        "output": "rollback_doc",
        "ignore": true,
        "body": {
            "task": "string"
        },
        "expected": {
            "code": 201,
            "expected_match": "partial",
            "body": {
                "task": "string",
                "id": null
            }
        },
        "retry": 0,
        "delay": 1,
        "headers": {
            "Accept": "application/json",
            "Content-Type": "application/json"
        },
        "duration_sec": 0.03
    },
    "ignore": true
}
Traceback (most recent call last):
  File "/lib/python3.9/site-packages/spintest/manager.py", line 193, in run
    results.append(await self._next())
  File "/lib/python3.9/site-packages/spintest/manager.py", line 178, in _next
    return await self.stack.__anext__()
  File "/lib/python3.9/site-packages/spintest/manager.py", line 116, in _executor
    result = await Task(
  File "/lib/python3.9/site-packages/spintest/task.py", line 174, in run
    return self._response(
  File "/lib/python3.9/site-packages/spintest/task.py", line 48, in _response
    log_level.get(status, logger.critical)(json.dumps(result, indent=4))
  File "python/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "python/lib/python3.9/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "python/lib/python3.9/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "python/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "python/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "python/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  [Previous line repeated 1 more time]
  File "python/lib/python3.9/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "python/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Int is not JSON serializable

On this python file i have noticed that some the 'fail_on' value under task[1]['fail_on'] has the incorrect type.
It should be a list of dict and not directly a dict.
The issue here is that the error raised does not match the issue at end.

By moving the faulty key ('fail_on') to the first task. I'm getting a more coherent error

2022-04-21 10:10:45,564 - ERROR - {
    "name": null,
    "status": "FAILED",
    "timestamp": "Thu Apr 21 10:10:45 2022",
    "duration_sec": null,
    "url": "http://localhost:8080/",
    "route": "rollback/",
    "message": "Task must follow this schema : Schema({'method': <class 'str'>, Optional('route'): <class 'str'>, Optional('name'): <class 'str'>, Optional('body'): <class 'dict'>, Optional('headers'): <class 'dict'>, Optional('output'): <class 'str'>, Optional('expected'): {Optional('code'): <class 'int'>, Optional('body'): Or(<class 'dict'>, <class 'str'>), Optional('expected_match'): Or('partial', 'strict')}, Optional('fail_on'): [{Optional('code'): <class 'int'>, Optional('body'): Or(<class 'dict'>, <class 'str'>), Optional('expected_match'): Or('partial', 'strict')}], Optional('retry'): <class 'int'>, Optional('delay'): <class 'int'>, Optional('ignore'): <class 'bool'>, Optional('rollback'): [Or(<class 'str'>, <class 'dict'>)]}).",
    "code": null,
    "body": null,
    "task": {
        "method": "POST",
        "route": "rollback/",
        "output": "rollback_doc",
        "body": {
            "task": "string"
        },
        "expected": {
            "code": 201,
            "body": {
                "task": "string",
                "id": null
            },
            "expected_match": "partial"
        },
        "fail_on": {
            "expected_match": "partial",
            "body": {
                "status": "ERROR"
            }
        }
    },
    "ignore": false
}
Traceback (most recent call last):
  File "pydevd.py", line 1415, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "test_spin.py", line 158, in <module>
    assert True is result
AssertionError

From what i can understand.

the parsing of the second task fails here
.smoke/lib/python3.9/site-packages/spintest/task.py:172
validated_task = input_validator(self.task, TASK_SCHEMA)

Which lead to not parsing the task and skipping the casting the pytest.Int.
The task then failed here
.smoke/lib/python3.9/site-packages/spintest/task.py:48
log_level.get(status, logger.critical)(json.dumps(result, indent=4))

because result still have the "id": Int("{{ rollback_doc['id']}}"), unparsed causing the json.dumps to fail

---

I don't know the full depth of the parsing order but have noticed that in
.smoke/lib/python3.9/site-packages/spintest/validator.py:41

modifiing the

    try:
        return input_schema.validate(input)
    except SchemaError:
        return None

to

    try:
        return input_schema.validate(input)
    except SchemaError:
        raise

Raises the correct error schema.SchemaError: Key 'fail_on' error:{'expected_match': 'partial', 'body': {'status': 'ERROR'}} should be instance of 'list'

I don't know the full side effect of doing this change but doing something like this
or
Parsing the task (to remove cast) might be better to avoid dumping incorrect json.