This repository has been archived by the owner on Oct 11, 2024. It is now read-only.
add test coverage table to github summary #368
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
collects code coverage report as JSON to upload to build artifacts, then parses the file to generate a markdown table for presentation in the GitHub summary. Removes Temporarily limits test cases and platforms in nm-remote-push, and skips benchmark, etc in nm-build-test to make testing go faster.
temporarily restrict tests to one file for debugging
may be easier than passing the multi-line table from one action to another.
this time with correction.
this time with correction.
and aligning columns
and revert changes used for testing purposes.
derekk-nm
requested review from
dbarbuzzi,
dhuangnm,
andy-neuma and
robertgshaw2-neuralmagic
andy-neuma
reviewed
Jul 10, 2024
| pip3 install tabulate | ||
| # As a multiline response we cannot pass the table directly to github | ||
| # so redirect it to a file, then cat the file to the output | ||
| python3 ./.github/scripts/coverage_report_breakdown.py ${{ inputs.coverage_json }} > COVERAGE_MD |
There was a problem hiding this comment.
the table ends up looking nice.
| args = parser.parse_args() | ||
| cc = CodeCoverage(Path(args.coverage_json_file)) | ||
|
|
||
| print(cc.to_github_markdown()) |
There was a problem hiding this comment.
i like that you are outputting to a file. this makes it easy to run locally.
| ignore_index=True) | ||
| # clean up the `nan` values for display purposes | ||
| summary_df = summary_df.astype(str) | ||
| summary_df.replace({"nan": None}, inplace=True) |
There was a problem hiding this comment.
quick question, how do we generate "nan"?
There was a problem hiding this comment.
this was "fun" 😉 . to get that empty row and header row between the overall info and the per-sub-directory breakdown I needed to add that empty_row_df and header_row_df. To get the concat to work successfully I needed to populate those with pd.NA, (which is nan). once the concat was done, I could clean up those nans
our test runs won't ever enable it
dbarbuzzi
approved these changes
Jul 15, 2024
|
moved to nm-vllm-ent |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the creation of a JSON form of the test coverage information that has only been generated as HTML up until this point. The JSON content is then parsed by a script to generate a table that is compatible with
github-markdownso that it can be included in the test job summary. The table shows the overall test coverage, and the coverage provided by each sub-directory at the level just below vllm.This nm remote push job demonstrates that the code changes are working (no failures), and shows the expected test coverage summary table (on the summary page, scroll down to the TEST summary), and JSON artifact (on the summary page, scroll down to the artifacts listing and look for the files starting with
cc-vllm-json-*.Also included are some additional functions that will be excluded/omitted from the test coverage. see pyproject.toml