From 6b24ac0cefc8bc5b1e5cd75a7e108dff37ba4e36 Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Mon, 6 Jan 2025 23:52:31 +0000 Subject: [PATCH] Fixes to submission generation docs --- docs/submission/index.md | 201 ++++++++++++++------------------------- 1 file changed, 71 insertions(+), 130 deletions(-) diff --git a/docs/submission/index.md b/docs/submission/index.md index 6fe0ccb4e..f7784426f 100644 --- a/docs/submission/index.md +++ b/docs/submission/index.md @@ -3,10 +3,7 @@ hide: - toc --- - - - -Click [here](https://docs.google.com/presentation/d/1cmbpZUpVr78EIrhzyMBnnWnjJrD-mZ2vmSb-yETkTA8/edit?usp=sharing) to view the prposal slide for Common Automation for MLPerf Inference Submission Generation through CM. +Click [here](https://docs.google.com/presentation/d/1cmbpZUpVr78EIrhzyMBnnWnjJrD-mZ2vmSb-yETkTA8/edit?usp=sharing) to view the proposal slide for Common Automation for MLPerf Inference Submission Generation through CM. === "Custom automation based MLPerf results" If you have not followed the `cm run` commands under the individual model pages in the [benchmarks](../index.md) directory, please make sure that the result directory is structured in the following way. @@ -54,7 +51,7 @@ Click [here](https://docs.google.com/presentation/d/1cmbpZUpVr78EIrhzyMBnnWnjJrD ``` -=== "MLPerf Automation based results" +=== "CM automation based results" If you have followed the `cm run` commands under the individual model pages in the [benchmarks](../index.md) directory, all the valid results will get aggregated to the `cm cache` folder. The following command could be used to browse the structure of inference results folder generated by CM. ### Get results folder structure ```bash @@ -74,13 +71,13 @@ flowchart LR direction TB A[populate system details] --> B[generate submission structure] B --> C[truncate-accuracy-logs] - C --> D{Infer low talency results and/or filter out invalid results} + C --> D{Infer low talency results
and/or
filter out invalid results} D --> yes --> E[preprocess-mlperf-inference-submission] D --> no --> F[run-mlperf-inference-submission-checker] E --> F end - Input((MLPerf Inference Results SUT1)) --> Generation - Generation --> Output((Submission Folder SUT1)) + Input((Results SUT1)) --> Generation + Generation --> Output((Submission Folder
SUT1)) ``` ### Command to generate actual submission folder @@ -95,26 +92,32 @@ cm run script --tags=generate,inference,submission \ --env.CM_DETERMINE_MEMORY_CONFIGURATION=yes \ --quiet ``` -* Use `--hw_name="My system name"` to give a meaningful system name. Examples can be seen [here](https://github.com/mlcommons/inference_results_v3.0/tree/main/open/cTuning/systems) +!!! tip + * Use `--hw_name="My system name"` to give a meaningful system name. Examples can be seen [here](https://github.com/mlcommons/inference_results_v3.0/tree/main/open/cTuning/systems) -* Use `--submitter=` if your organization is an official MLCommons member and would like to submit under your organization + * Use `--submitter=` if your organization is an official MLCommons member and would like to submit under your organization -* Use `--hw_notes_extra` option to add additional notes like `--hw_notes_extra="Result taken by NAME" ` + * Use `--hw_notes_extra` option to add additional notes like `--hw_notes_extra="Result taken by NAME" ` -* Use `--results_dir` option to specify the results folder. It is automatically taken from CM cache for MLPerf automation based runs + * Use `--results_dir` option to specify the results folder. It is automatically taken from CM cache for MLPerf automation based runs -* Use `--submission_dir` option to specify the submission folder. + * Use `--submission_dir` option to specify the submission folder. -* Use `--division=open` for open division submission + * Use `--division=open` for open division submission -* Use `--category` option to specify the category for which submission is generated(datacenter/edge). By default, the category is taken from `system_meta.json` file located in the SUT root directory. + * Use `--category` option to specify the category for which submission is generated(datacenter/edge). By default, the category is taken from `system_meta.json` file located in the SUT root directory. -* Use `--submission_base_dir` to specify the directory to which the outputs from preprocess submission script and final submission is added. No need to provide `--submission_dir` along with this. For `docker run`, use `--submission_base_dir` instead of `--submission_dir`. + * Use `--submission_base_dir` to specify the directory to which the outputs from preprocess submission script and final submission is added. No need to provide `--submission_dir` along with this. For `docker run`, use `--submission_base_dir` instead of `--submission_dir`. -If there are multiple SUTs, the same process needs to be repeated on each of them. One we have Submission folders on all the SUTs, we need to sync them to make a single submission folder +If there are multiple systems where MLPerf results are collected, the same process needs to be repeated on each of them. One we have submission folders on all the SUTs, we need to sync them to make a single submission folder === "Sync Locally" + If you are having results in multiple systems, you need to merge them to one system. You can use `rsync` for this. For example, the below command will sync the submission folder from SUT2 to the one in SUT1. + ``` + rsync -avz username@host1:/ / + ``` + Same needs to be repeated for all other SUTs so that we have the full submissions in SUT1. ```mermaid flowchart LR @@ -127,134 +130,72 @@ If there are multiple SUTs, the same process needs to be repeated on each of the subgraph SUT3 [Submission Generation SUT3] C[Submission Folder SUT3] end + subgraph SUTN [Submission Generation SUTN] + D[Submission Folder SUTN] + end SUT2 --> SUT1 SUT3 --> SUT1 + SUTN --> SUT1 ``` === "Sync via a Github repo" + If you are collecting results across multiple systems you can generate different submissions and aggregate all of them to a GitHub repository (can be private) and use it to generate a single tar ball which can be uploaded to the [MLCommons Submission UI](https://submissions-ui.mlcommons.org/submission). + Run the following command after **replacing `--repo_url` with your GitHub repository URL**. + + ```bash + cm run script --tags=push,github,mlperf,inference,submission \ + --repo_url=https://github.com/mlcommons/mlperf_inference_submissions_v5.0 \ + --commit_message="Results on added by " \ + --quiet + ``` + ```mermaid flowchart LR - classDef hidden fill:none,stroke:none; - - subgraph Generation1 [Submission Generation SUT-1] - direction TB - A3[populate system details] --> B3[generate submission structure] - B3 --> C3[truncate-accuracy-logs] - C3 --> D3{Infer low talency results and/or filter out invalid results} - D3 -- yes --> E3[preprocess-mlperf-inference-submission] - D3 -- no --> F3[run-mlperf-inference-submission-checker] - E3 --> F3 + subgraph SUT1 [Submission Generation SUT1] + A[Submission Folder SUT1] end - - subgraph Generation2 [Submission Generation SUT-2] - direction TB + subgraph SUT2 [Submission Generation SUT2] + B[Submission Folder SUT2] end - - subgraph GenerationN [Submission Generation SUT-N] - direction TB + subgraph SUT3 [Submission Generation SUT3] + C[Submission Folder SUT3] end - - Input1((MLPerf Inference Results folder SUT1)) --> Generation1 --> T1[Submission Tree 1] - Input2((MLPerf Inference Results folder SUT2)) --> Generation2 --> T2[Submission Tree 2] - Input3((MLPerf Inference Results folder SUTN)) --> GenerationN --> TN[Submission Tree N] - - subgraph LargeCircle [ ] - direction TB - Generation1 - Generation2 - GenerationN + subgraph SUTN [Submission Generation SUTN] + D[Submission Folder SUTN] end - - T1 --> Sync((Upload the submission tree to GitHub repo)) - T2 --> Sync - TN --> Sync - - Sync --> clone[Clone the repo to SUT1] --> tar[Submission Tar File] --> upload[Upload result to submission server] --> output((Receive vlidation email)) + SUT2 -- git sync and push --> G[Github Repo] + SUT3 -- git sync and push --> G[Github Repo] + SUTN -- git sync and push --> G[Github Repo] + SUT1 -- git sync and push --> G[Github Repo] + ``` - -## Command to generate actual submission tree -=== "Docker run" - ### Docker run - === "Closed" - ### Closed Submission - ```bash - cm docker script --tags=generate,inference,submission \ - --clean \ - --preprocess_submission=yes \ - --run-checker \ - --submitter=MLCommons \ - --tar=yes \ - --env.CM_TAR_OUTFILE=submission.tar.gz \ - --division=closed \ - --env.CM_DETERMINE_MEMORY_CONFIGURATION=yes \ - --quiet - ``` - - === "Open" - ### Open Submission - ```bash - cm docker script --tags=generate,inference,submission \ - --clean \ - --preprocess_submission=yes \ - --run-checker \ - --submitter=MLCommons \ - --tar=yes \ - --env.CM_TAR_OUTFILE=submission.tar.gz \ - --division=open \ - --env.CM_DETERMINE_MEMORY_CONFIGURATION=yes \ - --quiet - ``` - -=== "Native run" - ### Native run - === "Closed" - ### Closed Submission - ```bash - cm run script --tags=generate,inference,submission \ - --clean \ - --preprocess_submission=yes \ - --run-checker \ - --submitter=MLCommons \ - --tar=yes \ - --env.CM_TAR_OUTFILE=submission.tar.gz \ - --division=closed \ - --env.CM_DETERMINE_MEMORY_CONFIGURATION=yes \ - --quiet - ``` - - === "Open" - ### Open Submission - ```bash - cm run script --tags=generate,inference,submission \ - --clean \ - --preprocess_submission=yes \ - --run-checker \ - --submitter=MLCommons \ - --tar=yes \ - --env.CM_TAR_OUTFILE=submission.tar.gz \ - --division=open \ - --env.CM_DETERMINE_MEMORY_CONFIGURATION=yes \ - --quiet - ``` - - -The above command should generate "submission.tar.gz" if there are no submission checker issues and you can upload it to the [MLCommons Submission UI](https://submissions-ui.mlcommons.org/submission). - -## Aggregate Results in GitHub - -If you are collecting results across multiple systems you can generate different submissions and aggregate all of them to a GitHub repository (can be private) and use it to generate a single tar ball which can be uploaded to the [MLCommons Submission UI](https://submissions-ui.mlcommons.org/submission). - -Run the following command after **replacing `--repo_url` with your GitHub repository URL**. -```bash -cm run script --tags=push,github,mlperf,inference,submission \ - --repo_url=https://github.com/mlcommons/mlperf_inference_submissions_v4.1 \ - --commit_message="Results on added by " \ - --quiet +## Upload the final submission + + +=== "via CLI" + You can do the following command which will run the submission checker and upload the results to the MLCommons submission server + ``` + cm run script --tags=run,submission,checker --submitter_id=<> + ``` +=== "via Browser" + You can do the following command to generate the final submission tar file and then upload to the [MLCommons Submission UI](https://submissions-ui.mlcommons.org/submission). + ``` + cm run script --tags=run,submission,checker --submitter_id=<> --tar=yes --submission_tar_file=mysubmission.tar.gz + ``` + +```mermaid + flowchart LR + subgraph SUT [Combined Submissions] + A[Combined Submission Folder in SUT1] + end + SUT --> B[Run submission checker] + B --> C[Upload to MLC Submission server] + C --> D[Receive validation email] ``` -At the end, you can download the github repo and upload to the [MLCommons Submission UI](https://submissions-ui.mlcommons.org/submission). -Click [here](https://youtu.be/eI1Hoecc3ho) to view the recording of the workshop: Streamlining your MLPerf Inference results using CM. + +