From 1e6edcc1a718fc186ca5928a8e076cf03a68a73d Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Thu, 4 May 2023 11:56:00 -0700 Subject: [PATCH 1/8] doc: document how a test executor works This describes the JSON specification for others who might want to write their own test executors. --- doc/specification.md | 52 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 doc/specification.md diff --git a/doc/specification.md b/doc/specification.md new file mode 100644 index 00000000..47b28c79 --- /dev/null +++ b/doc/specification.md @@ -0,0 +1,52 @@ +# Test Specification + +This document describes how to use the JSON test specifications to write your own test executor for +[`wasi-testsuite`](..). The test executor included in this project provides a completely-usable +reference implementation, but projects with other requirements may want to run the tests in their +own way (e.g., no Python dependency). The JSON test specifications provide a simple way to verify +that the tests indeed passed. + +### Configuration + +Before executing anything, a test executor is expected to: +- find all `*.wasm` and `*.wat` files in a given subdirectory — these are the _test cases_ +- for each test case, look for a `.json` file in the same directory matching the base name (e.g., + `foo.json` for `foo.wasm`) — parse this _specification_ +- if no `.json` file is present, use a default specification; a (conceptual) default specification + would look like: + + ```json + { + "args": [], + "dirs": [], + "env": {}, + "exit_code": 0, + "stderr": "", + "stdout": "" + } + ``` + +- if the specification is missing fields, use default values + +### Execution + +To execute the tests, the test executor is expected to: +- `env`: construct an environment from each key-value pair in `env`; the environment should only + contain these keys and otherwise should be empty (note that this environment is the WASI + environment, whether the engine inherits the shell environment or explicitly configures it via + `--env` parameters) +- `dir`: add each path listed in `dir` as WASI preopen directories (some engines use a `--dir` + flag) +- `args`: pass each argument in order to the WASI program (most CLI engines allow appending these + after the module path) + +The test executor runs the WebAssembly test case with the above context and records its results. + +### Checking + +With the execution results in hand, the test executor is expected to: +- `exit_code`: check that the WASI exit code matches `exit_code`, or `0` if none was provided +- `stderr`: if a `stderr` field is provided, check that the bytes printed to `stderr` match it +- `stdout`: if a `stdout` field is provided, check that the bytes printed to `stdout` match it + +A test case _passes_ if all of the checks are true. From 5659d1fd3baf92c4851c0e9ddc5863ce2f20bcf3 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Thu, 4 May 2023 11:56:50 -0700 Subject: [PATCH 2/8] doc: tweak README to link to specification document The primary purpose of this change is to link to `doc/specification.md` but while I was here I made some minor text cleanups. --- README.md | 74 ++++++++++++++++++++++++++++++++----------------------- 1 file changed, 43 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index 750cb508..b32e9456 100644 --- a/README.md +++ b/README.md @@ -1,43 +1,53 @@ -# WASI tests +# `wasi-testsuite` -This repository contains tests for WebAssembly System Interface (WASI) and a test executor for running WASI tests on a selected WebAssembly runtime. +This repository contains tests for WebAssembly System Interface (WASI) and a test executor for running WASI tests against a selected WebAssembly runtime. -WASI is a modular collection of standardized APIs. Currently, WASI has not reached a v1 with a defined set of APIs. -However, a snapshot of experimental APIs exists ([`wasi_snapshot_preview1`](https://github.com/WebAssembly/WASI/blob/main/legacy/preview1/docs.md)). -The repository only holds tests of APIs included in this snapshot. It does not include tests for other in-progress proposals or other experimental APIs. +WASI is a modular collection of standardized APIs. Currently, WASI has not reached version 1.0 +stability but a snapshot of experimental APIs does exist ([`wasi_snapshot_preview1`]). This +repository only holds tests of APIs included in this snapshot. It does not include tests for other +in-progress proposals or other experimental APIs, though the test executor can run tests from other repositories (e.g., see the [wasi-threads] tests). -The test executor included in the repository can however be used to run tests defined for proposals along with tests defined in this repository. +[`wasi_snapshot_preview1`]: https://github.com/WebAssembly/WASI/blob/main/legacy/preview1/docs.md +[wasi-threads]: https://github.com/WebAssembly/wasi-threads/tree/main/test -## Getting started +The test executor matches execution output against a JSON specification. Writing your own test +executor is quite simple; see the [specification] document for the details. -1. Clone repository - Use `prod/testsuite-base` branch as it already includes precompiled test binaries (see [Branch structure](#branch-structure) paragraph). +[specification]: doc/specification.md -```bash -git clone --branch prod/testsuite-base git@github.com:WebAssembly/wasi-testsuite.git -``` +## Getting started + +1. Clone the repository; use the `prod/testsuite-base` branch as it already includes precompiled + test binaries (see [branch structure](#branch-structure)): -2. Make sure there's already an adapter for the runtime in the [`adapters`](adapters) directory; if not, create one (see [the doc](doc/adapters.md) for details). -3. Install python3 - 1. Ubuntu + ```bash + git clone --branch prod/testsuite-base git@github.com:WebAssembly/wasi-testsuite.git ``` - $ sudo apt install python3 python3-pip + +2. Make sure there is already an adapter for the runtime in the [`adapters`](adapters) directory; if + not, create one (see [the doc](doc/adapters.md) for details). + +3. Install `python3` (e.g., on Ubuntu): + + ```bash + sudo apt install python3 python3-pip ``` -4. Install test runner dependencies: -```bash -python3 -m pip install -r test-runner/requirements.txt -``` +4. Install the test runner dependencies: -5. Execute test suites from this repository + ```bash + python3 -m pip install -r test-runner/requirements.txt + ``` -```bash -python3 test-runner/wasi_test_runner.py \ - -t ./tests/assemblyscript/testsuite/ `# path to folders containing .wasm test files` \ - ./tests/c/testsuite/ \ - ./tests/rust/testsuite/ \ - -r adapters/wasmtime.py # path to a runtime adapter -``` +5. Execute the test suites from this repository: + + ```bash + python3 test-runner/wasi_test_runner.py \ + -t ./tests/assemblyscript/testsuite/ `# path to folders containing .wasm test files` \ + ./tests/c/testsuite/ \ + ./tests/rust/testsuite/ \ + -r adapters/wasmtime.py # path to a runtime adapter + ``` Optionally you can specify test cases to skip with the `--exclude-filter` option. @@ -83,7 +93,11 @@ Here is some additional information for developers who are willing to contribute ### Cleaning up temporary resources -Some of the tests (e.g. [pwrite-with-access](./tests/c/testsuite/pwrite-with-access.c)) generate output artifacts and their existence can affect consecutive test executions. Tests should clean up the artifacts they generate, but there might be cases where the test fails early. Test runner will automatically delete all the files and directories in the test suite directory with the `.cleanup` suffix. +Some of the tests (e.g. [pwrite-with-access](./tests/c/testsuite/pwrite-with-access.c)) generate +output artifacts and their existence can affect consecutive test executions. Tests should clean up +the artifacts they generate, but there might be cases where the test fails early. The test runner +will automatically delete all the files and directories in the test suite directory with the +`.cleanup` suffix. ### Programming languages for tests @@ -95,8 +109,6 @@ The repository currently consists of tests implemented in the following language The list of supported languages can be extended if needed. -### Test - ### Branch structure Apart from development branches for various features, we identify the following branches as critical (i.e. they won't be removed or force-updated): From 8dd770bf391114860e48008e109df850672b4a69 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Tue, 9 May 2023 06:45:05 -0700 Subject: [PATCH 3/8] Update README.md Co-authored-by: Marcin Kolny --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b32e9456..9dba1a26 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ in-progress proposals or other experimental APIs, though the test executor can r [wasi-threads]: https://github.com/WebAssembly/wasi-threads/tree/main/test The test executor matches execution output against a JSON specification. Writing your own test -executor is quite simple; see the [specification] document for the details. +executor is quite simple; see the [specification] document for the details and the [reference Python implementation](../test-runner). [specification]: doc/specification.md From 3486a8d18d5841597edac4cc035b5775b368cd7b Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Tue, 9 May 2023 07:43:05 -0700 Subject: [PATCH 4/8] fix: repair broken link --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 9dba1a26..5433367e 100644 --- a/README.md +++ b/README.md @@ -11,9 +11,11 @@ in-progress proposals or other experimental APIs, though the test executor can r [wasi-threads]: https://github.com/WebAssembly/wasi-threads/tree/main/test The test executor matches execution output against a JSON specification. Writing your own test -executor is quite simple; see the [specification] document for the details and the [reference Python implementation](../test-runner). +executor is quite simple; see the [specification] document for the details and the reference Python +[implementation]. [specification]: doc/specification.md +[implementation]: ./test-runner ## Getting started From 8c1213e585bad254adc908c9a5150db48bb2a0cf Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Tue, 9 May 2023 07:44:18 -0700 Subject: [PATCH 5/8] review: use HTTP link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5433367e..6649e8c4 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ executor is quite simple; see the [specification] document for the details and t test binaries (see [branch structure](#branch-structure)): ```bash - git clone --branch prod/testsuite-base git@github.com:WebAssembly/wasi-testsuite.git + git clone --branch prod/testsuite-base https://github.com/WebAssembly/wasi-testsuite ``` 2. Make sure there is already an adapter for the runtime in the [`adapters`](adapters) directory; if From 882f2d8ec423e34fc44ea0c80cd0c29798b7d5ef Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Tue, 9 May 2023 07:45:34 -0700 Subject: [PATCH 6/8] review: link to test-runner again --- doc/specification.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/specification.md b/doc/specification.md index 47b28c79..8dc2759d 100644 --- a/doc/specification.md +++ b/doc/specification.md @@ -2,9 +2,9 @@ This document describes how to use the JSON test specifications to write your own test executor for [`wasi-testsuite`](..). The test executor included in this project provides a completely-usable -reference implementation, but projects with other requirements may want to run the tests in their -own way (e.g., no Python dependency). The JSON test specifications provide a simple way to verify -that the tests indeed passed. +reference [implementation](../test-runner), but projects with other requirements may want to run the +tests in their own way (e.g., no Python dependency). The JSON test specifications provide a simple +way to verify that the tests indeed passed. ### Configuration From 3e7e54882ba2940093924cc032183406756e0651 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Tue, 9 May 2023 07:48:09 -0700 Subject: [PATCH 7/8] review: limit to *.wasm files --- doc/specification.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/specification.md b/doc/specification.md index 8dc2759d..17b9732e 100644 --- a/doc/specification.md +++ b/doc/specification.md @@ -9,7 +9,7 @@ way to verify that the tests indeed passed. ### Configuration Before executing anything, a test executor is expected to: -- find all `*.wasm` and `*.wat` files in a given subdirectory — these are the _test cases_ +- find all `*.wasm` files in a given subdirectory — these are the _test cases_ - for each test case, look for a `.json` file in the same directory matching the base name (e.g., `foo.json` for `foo.wasm`) — parse this _specification_ - if no `.json` file is present, use a default specification; a (conceptual) default specification From ea55a43a1e0e307f69205a86b011a30612f255d0 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Tue, 9 May 2023 07:51:04 -0700 Subject: [PATCH 8/8] review: add line about *.cleanup files --- doc/specification.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/specification.md b/doc/specification.md index 17b9732e..258f0b50 100644 --- a/doc/specification.md +++ b/doc/specification.md @@ -10,6 +10,8 @@ way to verify that the tests indeed passed. Before executing anything, a test executor is expected to: - find all `*.wasm` files in a given subdirectory — these are the _test cases_ +- find all `*.cleanup` files in a given subdirectory and remove them — these are test + artifacts that can be generated during testing - for each test case, look for a `.json` file in the same directory matching the base name (e.g., `foo.json` for `foo.wasm`) — parse this _specification_ - if no `.json` file is present, use a default specification; a (conceptual) default specification