Update documentation for adding new tests #5113
Conversation
Note: Changes may take a few seconds to appear on GitHub Pages. Please refresh the page if you do not see the updates immediately. |
|
hey @jafingerhut can you please review this pr |
docs/P4c-test-documentation.md
Outdated
|
|
||
| ## **4. Marking Tests as "Expected to Fail" (Xfail)** | ||
|
|
||
| An Xfail test is one that is expected to fail due to a known issue or limitation in the P4 compiler. The CI system handles these tests differently based on their type (positive or negative). |
There was a problem hiding this comment.
An XFail is a test that should pass (by the language spec and architecture), but currently fails due to known problems in the compiler.
docs/P4c-test-documentation.md
Outdated
|
|
||
| - If a change to the P4 compiler modifies the expected outputs, you can regenerate them by rerunning the tests with: | ||
| ```bash | ||
| ./build/p4test path/to/source_file.p4 --update-outputs |
There was a problem hiding this comment.
I can find no mention of update-outputs anywhere in the p4c repo. Where did you find such an option?
|
@jafingerhut @ChrisDodd I have made the changes as per your review. Please review them |
| ``` | ||
| - To generate or update the expected output file for a single test case, use: | ||
| ```bash | ||
| P4TEST_REPLACE=1 ./build/p4test path/to/source_file.p4 |
There was a problem hiding this comment.
I can try out this command this weekend on a personal system of mine, but have you tried it to verify that it actually generates or updates the expected output files? If so, I am surprised, because p4test has nothing built into it to tell which directory to put the expected output files in.
| - **`p4_14_samples`**: Contains tests specific to the older P4\_14 language version. | ||
| - **`p4_14_samples_outputs`**: Stores the expected outputs for tests in `p4_14_samples`. | ||
| - **`bmv2`**: Holds tests targeting the BMv2 backend, including packet-processing logic and examples. | ||
| - **`psa`**: Includes tests for the PSA (Portable Switch Architecture) target. |
There was a problem hiding this comment.
PSA is not a target. It is a P4 architecture.
In general, we could make a table with a row each target, and a column for each P4 architecture, and fill in the entries of the table with "yes" if that target has an implementation for that architecture, and "no" if it does not. e.g. the P4 DPDK target implements the PSA and PNA architectures (at least partially), but it does not implement the v1model architecture at all. The BMv2 back end implements the v1model architecture, and it has the beginnings of implementations of the PSA and PNA architectures, but they are missing a lot there.
I don't know off the top of my head which targets (if any) the psa directory has tests for. I haven't looked at it recently, but it would be nice if we can find out and document it a little more precisely here.
| - For eBPF-specific tests: `*_ebpf.p4` or `*_ebpf-kernel.p4`. | ||
| - For graph backend tests: `*graph*.p4`. | ||
| - For UBPF-specific tests: `*_ubpf.p4`. | ||
| - For PSA tests targeting BMv2: `*-bmv2.p4`. |
There was a problem hiding this comment.
There are many tests for the BMv2 back end using the v1model architecture that have file names matching *-bmv2.p4. Do you have examples of specific files with names matching that pattern that use the PSA architecture, too? There might be -- I have not checked.
There was a problem hiding this comment.
Just did a quick check -- yes, there are many that use the PSA architecture, but even more that use the v1model architecture.
This PR updates the documentation for adding tests to the P4C repository.
Pull Request Title:
Update Documentation: Guide for Adding New Tests to P4C Repository
Description:
This PR adds a detailed guide to the documentation for developers looking to contribute tests to the P4C repository. The document outlines:
The guide aims to enhance developer productivity and ensure consistency in test contributions.
Motivation:
Changes Made:
docsdirectory.Testing:
Checklist:
Related Issues:
None directly. This PR aims to improve developer experience and test consistency proactively.