Skip to content

Commit

Permalink
Fixes to submission generation docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@arjunsuresh
arjunsuresh committed Jan 6, 2025
1 parent 8425da6 commit 6b24ac0
Showing 1 changed file with 71 additions and 130 deletions.
201 changes: 71 additions & 130 deletions docs/submission/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,7 @@ hide:
- toc
---




Click [here](https://docs.google.com/presentation/d/1cmbpZUpVr78EIrhzyMBnnWnjJrD-mZ2vmSb-yETkTA8/edit?usp=sharing) to view the prposal slide for Common Automation for MLPerf Inference Submission Generation through CM.
Click [here](https://docs.google.com/presentation/d/1cmbpZUpVr78EIrhzyMBnnWnjJrD-mZ2vmSb-yETkTA8/edit?usp=sharing) to view the proposal slide for Common Automation for MLPerf Inference Submission Generation through CM.

=== "Custom automation based MLPerf results"
If you have not followed the `cm run` commands under the individual model pages in the [benchmarks](../index.md) directory, please make sure that the result directory is structured in the following way.
Expand Down Expand Up @@ -54,7 +51,7 @@ Click [here](https://docs.google.com/presentation/d/1cmbpZUpVr78EIrhzyMBnnWnjJrD
```
</details>

=== "MLPerf Automation based results"
=== "CM automation based results"
If you have followed the `cm run` commands under the individual model pages in the [benchmarks](../index.md) directory, all the valid results will get aggregated to the `cm cache` folder. The following command could be used to browse the structure of inference results folder generated by CM.
### Get results folder structure
```bash
Expand All @@ -74,13 +71,13 @@ flowchart LR
direction TB
A[populate system details] --> B[generate submission structure]
B --> C[truncate-accuracy-logs]
C --> D{Infer low talency results and/or filter out invalid results}
C --> D{Infer low talency results <br>and/or<br> filter out invalid results}
D --> yes --> E[preprocess-mlperf-inference-submission]
D --> no --> F[run-mlperf-inference-submission-checker]
E --> F
end
Input((MLPerf Inference Results SUT1)) --> Generation
Generation --> Output((Submission Folder SUT1))
Input((Results SUT1)) --> Generation
Generation --> Output((Submission Folder <br> SUT1))
```

### Command to generate actual submission folder
Expand All @@ -95,26 +92,32 @@ cm run script --tags=generate,inference,submission \
--env.CM_DETERMINE_MEMORY_CONFIGURATION=yes \
--quiet
```
* Use `--hw_name="My system name"` to give a meaningful system name. Examples can be seen [here](https://github.com/mlcommons/inference_results_v3.0/tree/main/open/cTuning/systems)
!!! tip
* Use `--hw_name="My system name"` to give a meaningful system name. Examples can be seen [here](https://github.com/mlcommons/inference_results_v3.0/tree/main/open/cTuning/systems)

* Use `--submitter=<Your name>` if your organization is an official MLCommons member and would like to submit under your organization
* Use `--submitter=<Your name>` if your organization is an official MLCommons member and would like to submit under your organization

* Use `--hw_notes_extra` option to add additional notes like `--hw_notes_extra="Result taken by NAME" `
* Use `--hw_notes_extra` option to add additional notes like `--hw_notes_extra="Result taken by NAME" `

* Use `--results_dir` option to specify the results folder. It is automatically taken from CM cache for MLPerf automation based runs
* Use `--results_dir` option to specify the results folder. It is automatically taken from CM cache for MLPerf automation based runs

* Use `--submission_dir` option to specify the submission folder.
* Use `--submission_dir` option to specify the submission folder.

* Use `--division=open` for open division submission
* Use `--division=open` for open division submission

* Use `--category` option to specify the category for which submission is generated(datacenter/edge). By default, the category is taken from `system_meta.json` file located in the SUT root directory.
* Use `--category` option to specify the category for which submission is generated(datacenter/edge). By default, the category is taken from `system_meta.json` file located in the SUT root directory.

* Use `--submission_base_dir` to specify the directory to which the outputs from preprocess submission script and final submission is added. No need to provide `--submission_dir` along with this. For `docker run`, use `--submission_base_dir` instead of `--submission_dir`.
* Use `--submission_base_dir` to specify the directory to which the outputs from preprocess submission script and final submission is added. No need to provide `--submission_dir` along with this. For `docker run`, use `--submission_base_dir` instead of `--submission_dir`.


If there are multiple SUTs, the same process needs to be repeated on each of them. One we have Submission folders on all the SUTs, we need to sync them to make a single submission folder
If there are multiple systems where MLPerf results are collected, the same process needs to be repeated on each of them. One we have submission folders on all the SUTs, we need to sync them to make a single submission folder

=== "Sync Locally"
If you are having results in multiple systems, you need to merge them to one system. You can use `rsync` for this. For example, the below command will sync the submission folder from SUT2 to the one in SUT1.
```
rsync -avz username@host1:<path_to_submission_folder2>/ <path_to_submission_folder1>/
```
Same needs to be repeated for all other SUTs so that we have the full submissions in SUT1.

```mermaid
flowchart LR
Expand All @@ -127,134 +130,72 @@ If there are multiple SUTs, the same process needs to be repeated on each of the
subgraph SUT3 [Submission Generation SUT3]
C[Submission Folder SUT3]
end
subgraph SUTN [Submission Generation SUTN]
D[Submission Folder SUTN]
end
SUT2 --> SUT1
SUT3 --> SUT1
SUTN --> SUT1
```

=== "Sync via a Github repo"
If you are collecting results across multiple systems you can generate different submissions and aggregate all of them to a GitHub repository (can be private) and use it to generate a single tar ball which can be uploaded to the [MLCommons Submission UI](https://submissions-ui.mlcommons.org/submission).

Run the following command after **replacing `--repo_url` with your GitHub repository URL**.

```bash
cm run script --tags=push,github,mlperf,inference,submission \
--repo_url=https://github.com/mlcommons/mlperf_inference_submissions_v5.0 \
--commit_message="Results on <HW name> added by <Name>" \
--quiet
```

```mermaid
flowchart LR
classDef hidden fill:none,stroke:none;
subgraph Generation1 [Submission Generation SUT-1]
direction TB
A3[populate system details] --> B3[generate submission structure]
B3 --> C3[truncate-accuracy-logs]
C3 --> D3{Infer low talency results and/or filter out invalid results}
D3 -- yes --> E3[preprocess-mlperf-inference-submission]
D3 -- no --> F3[run-mlperf-inference-submission-checker]
E3 --> F3
subgraph SUT1 [Submission Generation SUT1]
A[Submission Folder SUT1]
end
subgraph Generation2 [Submission Generation SUT-2]
direction TB
subgraph SUT2 [Submission Generation SUT2]
B[Submission Folder SUT2]
end
subgraph GenerationN [Submission Generation SUT-N]
direction TB
subgraph SUT3 [Submission Generation SUT3]
C[Submission Folder SUT3]
end
Input1((MLPerf Inference Results folder SUT1)) --> Generation1 --> T1[Submission Tree 1]
Input2((MLPerf Inference Results folder SUT2)) --> Generation2 --> T2[Submission Tree 2]
Input3((MLPerf Inference Results folder SUTN)) --> GenerationN --> TN[Submission Tree N]
subgraph LargeCircle [ ]
direction TB
Generation1
Generation2
GenerationN
subgraph SUTN [Submission Generation SUTN]
D[Submission Folder SUTN]
end
T1 --> Sync((Upload the submission tree to GitHub repo))
T2 --> Sync
TN --> Sync
Sync --> clone[Clone the repo to SUT1] --> tar[Submission Tar File] --> upload[Upload result to submission server] --> output((Receive vlidation email))
SUT2 -- git sync and push --> G[Github Repo]
SUT3 -- git sync and push --> G[Github Repo]
SUTN -- git sync and push --> G[Github Repo]
SUT1 -- git sync and push --> G[Github Repo]
```
## Command to generate actual submission tree
=== "Docker run"
### Docker run
=== "Closed"
### Closed Submission
```bash
cm docker script --tags=generate,inference,submission \
--clean \
--preprocess_submission=yes \
--run-checker \
--submitter=MLCommons \
--tar=yes \
--env.CM_TAR_OUTFILE=submission.tar.gz \
--division=closed \
--env.CM_DETERMINE_MEMORY_CONFIGURATION=yes \
--quiet
```

=== "Open"
### Open Submission
```bash
cm docker script --tags=generate,inference,submission \
--clean \
--preprocess_submission=yes \
--run-checker \
--submitter=MLCommons \
--tar=yes \
--env.CM_TAR_OUTFILE=submission.tar.gz \
--division=open \
--env.CM_DETERMINE_MEMORY_CONFIGURATION=yes \
--quiet
```

=== "Native run"
### Native run
=== "Closed"
### Closed Submission
```bash
cm run script --tags=generate,inference,submission \
--clean \
--preprocess_submission=yes \
--run-checker \
--submitter=MLCommons \
--tar=yes \
--env.CM_TAR_OUTFILE=submission.tar.gz \
--division=closed \
--env.CM_DETERMINE_MEMORY_CONFIGURATION=yes \
--quiet
```

=== "Open"
### Open Submission
```bash
cm run script --tags=generate,inference,submission \
--clean \
--preprocess_submission=yes \
--run-checker \
--submitter=MLCommons \
--tar=yes \
--env.CM_TAR_OUTFILE=submission.tar.gz \
--division=open \
--env.CM_DETERMINE_MEMORY_CONFIGURATION=yes \
--quiet
```


The above command should generate "submission.tar.gz" if there are no submission checker issues and you can upload it to the [MLCommons Submission UI](https://submissions-ui.mlcommons.org/submission).

## Aggregate Results in GitHub

If you are collecting results across multiple systems you can generate different submissions and aggregate all of them to a GitHub repository (can be private) and use it to generate a single tar ball which can be uploaded to the [MLCommons Submission UI](https://submissions-ui.mlcommons.org/submission).

Run the following command after **replacing `--repo_url` with your GitHub repository URL**.

```bash
cm run script --tags=push,github,mlperf,inference,submission \
--repo_url=https://github.com/mlcommons/mlperf_inference_submissions_v4.1 \
--commit_message="Results on <HW name> added by <Name>" \
--quiet
## Upload the final submission


=== "via CLI"
You can do the following command which will run the submission checker and upload the results to the MLCommons submission server
```
cm run script --tags=run,submission,checker --submitter_id=<>
```
=== "via Browser"
You can do the following command to generate the final submission tar file and then upload to the [MLCommons Submission UI](https://submissions-ui.mlcommons.org/submission).
```
cm run script --tags=run,submission,checker --submitter_id=<> --tar=yes --submission_tar_file=mysubmission.tar.gz
```

```mermaid
flowchart LR
subgraph SUT [Combined Submissions]
A[Combined Submission Folder in SUT1]
end
SUT --> B[Run submission checker]
B --> C[Upload to MLC Submission server]
C --> D[Receive validation email]
```

At the end, you can download the github repo and upload to the [MLCommons Submission UI](https://submissions-ui.mlcommons.org/submission).

Click [here](https://youtu.be/eI1Hoecc3ho) to view the recording of the workshop: Streamlining your MLPerf Inference results using CM.

<!--Click [here](https://youtu.be/eI1Hoecc3ho) to view the recording of the workshop: Streamlining your MLPerf Inference results using CM.-->

0 comments on commit 6b24ac0

Please sign in to comment.