TESTed-DSL inconsistencies and improvements

[pdawyndt]

Inconsistencies and suggested improvements in the design of TESTed-DSL.

Descriptions for objects with shorthands

Some objects have a shorthand, which allows them to be described as a single YAML-value. When the shorthand is expanded into a full-blown YAML object, the attribute describing the shorthand value is not named consistently.

• stdout: data (named content in file)
• stderr: data (named content in file)
• description: description

File descriptors

TESTed-DSL allows to describe the content of some file descriptors (stdin, stdout, stderr, file) inline, whereas the content of other file descriptors (files) is described with a reference to an external file. Both options make sense (small vs large content) and should be available for all file descriptors. The way the content is described in the DSL should determine how that content is represented in the feedback as we also need to take into account the size of the content there.

Inline file descriptor content

Named data with stdout and stderr. Named content with file

Generated files

Only a single generated file can be tested (file).

The naming of files for files available on the file system before testing (and references in the feedback) and file for testing generated files is confusing for users and should be clarified.

Oracles

These objects can have an associated oracle:

• stdout
• stderr
• return
• file

Oracles are not defined in the same way. All these objects should have the same name for the expected value (if inline) and all oracles should have the same definition. The latter should better be a separate oracle attribute:

stdout:
  data: "expected value"
  oracle:
    name: "test_stdout"
    file: "oracles.py"

Programming language vs. natural language

A map with alternatives for different programming languages has an optional !programming_language-tag when used in a place that otherwise has no other map value. A map with alternatives for different natural languages has a mandatory !natural_language-tag even when used in places that otherwise have no other map value. The latter was chosen for consistency across the DSL, to make the type of the map explicit and to allow future changes that might introduce maps in the same place. We could opt to also make the !programming_language-tag mandatory in future versions of the DSL.

Optional top-level

A test suite with a single unit and no top-level properties could be reduced to the description of the unit only. So

"units":
- "unit": "IO"
  "scripts":
  - "stdin": "A sentence must contain forty letters to be true."
    "stdout": "\U0001F44D"
  - "stdin": "This sentence does have a total of forty letters."
    "stdout": "\U0001F44D"
  - "stdin": "Therefore the second sentence is a true sentence."
    "stdout": "\U0001F44E (41 letters)"

could be reduced to

"unit": "IO"
"scripts":
- "stdin": "A sentence must contain forty letters to be true."
  "stdout": "\U0001F44D"
- "stdin": "This sentence does have a total of forty letters."
  "stdout": "\U0001F44D"
- "stdin": "Therefore the second sentence is a true sentence."
  "stdout": "\U0001F44E (41 letters)"

This can be done because a unit has a mandatory unit attribute. Now it can only be reduced to

- "unit": "IO"
  "scripts":
  - "stdin": "A sentence must contain forty letters to be true."
    "stdout": "\U0001F44D"
  - "stdin": "This sentence does have a total of forty letters."
    "stdout": "\U0001F44D"
  - "stdin": "Therefore the second sentence is a true sentence."
    "stdout": "\U0001F44E (41 letters)"