Improved CM commands

language/bert/README.md

@@ -55,7 +55,8 @@ The below CM command will launch the SUT server
 
 ```
 cm run script --tags=generate-run-cmds,inference --model=bert-99 --backend=pytorch  \
---mode=performance --device=cuda --quiet --test_query_count=1000 --network=sut
+--mode=performance --device=cuda --quiet --test_query_count=1000 --network=sut  \
+--outdirname=results/bert-99-performance
 ```
 
 Once the SUT server is launched, the below command can be run on the loadgen node to do issue queries to the SUT nodes. In this command `-sut_servers` has just the localhost address - it can be changed to a comma-separated list of any hostname/IP in the network. 
@@ -64,7 +65,8 @@ Once the SUT server is launched, the below command can be run on the loadgen nod
 ```
 cm run script --tags=generate-run-cmds,inference --model=bert-99 --backend=pytorch  --rerun \
 --mode=performance --device=cuda --quiet --test_query_count=1000  \
---sut_servers,=http://localhost:8000 --network=lon
+--sut_servers,=http://localhost:8000 --network=lon \
+--outdirname=results/bert-99-performance-lon
 ```
 
 If you are not using CM, just add `--network=lon` along with your normal run command on the SUT side.

text_to_image/README.md

@@ -1,9 +1,11 @@
 # MLPerf™ Inference Benchmarks for Text to Image
 
-This is the reference implementation for MLPerf Inference text to image.
+## Automated command to run the benchmark via MLCommons CM
 
 Please see the [new docs site](https://docs.mlcommons.org/inference/benchmarks/text_to_image/sdxl) for an automated way to run this benchmark across different available implementations and do an end-to-end submission with or without docker.
 
+You can also do `pip install cm4mlops` and then use `cm` commands for downloading the model and datasets using the commands given in the later sections.
+
 ## Supported Models
 
 | model | accuracy | dataset | model source | precision | notes |
@@ -53,10 +55,10 @@ We host two checkpoints (fp32 and fp16) that are a snapshot of the [Hugging Face
 The following MLCommons CM commands can be used to programmatically download the model checkpoints.
 
 ```
-pip install cmind
-cm pull repo mlcommons@ck
-cm run script --tags=get,ml-model,sdxl,_fp16,_rclone -j
-cm run script --tags=get,ml-model,sdxl,_fp32,_rclone -j
+cm run script --tags=get,ml-model,sdxl,_fp16,_rclone --outdirname=$MODEL_PATH
+```
+```
+cm run script --tags=get,ml-model,sdxl,_fp32,_rclone --outdirname-$MODEL_PATH
 ```
 #### Manual method
 
@@ -72,30 +74,35 @@ Once Rclone is installed, run the following command to authenticate with the buc
 rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com
 ```
 You can then navigate in the terminal to your desired download directory and run the following commands to download the checkpoints:
+```
+cd $MODEL_PATH
+```
 
 **`fp32`**
 ```
-rclone copy mlc-inference:mlcommons-inference-wg-public/stable_diffusion_fp32 ./stable_diffusion_fp32 -P
+rclone copy mlc-inference:mlcommons-inference-wg-public/stable_diffusion_fp32 $MODEL_PATH -P
 ```
 **`fp16`**
 ```
-rclone copy mlc-inference:mlcommons-inference-wg-public/stable_diffusion_fp16 ./stable_diffusion_fp16 -P
+rclone copy mlc-inference:mlcommons-inference-wg-public/stable_diffusion_fp16 $MODEL_PATH -P
 ```
 
-#### Move to model path
+### Download validation dataset
 
-```bash
-mkdir $MODEL_PATH
-cd $MODEL_PATH
-# For fp32
-mv <path_to_download>/stable_diffusion_fp32.zip .
-unzip stable_diffusion_fp32.zip
-# For fp16
-mv <path_to_download>/stable_diffusion_fp16.zip .
-unzip stable_diffusion_fp16.zip
+#### CM METHOD
+The following MLCommons CM commands can be used to programmatically download the validation dataset.
+
+```
+cm run script --tags=get,dataset,coco2014,_validation,_full --outdirname=coco2014
+```
+
+For debugging you can download only a part of all the images in the dataset
+```
+cm run script --tags=get,dataset,coco2014,_validation,_size.50 --outdirname=coco2014
 ```
 
-### Download dataset
+
+#### MANUAL METHOD
 ```bash
 cd $SD_FOLDER/tools
 ./download-coco-2014.sh -n <number_of_workers>
@@ -107,14 +114,25 @@ cd $SD_FOLDER/tools
 ```
 If the file [captions.tsv](coco2014/captions/captions.tsv) can be found in the script, it will be used to download the target dataset subset, otherwise it will be generated. We recommend you to have this file for consistency.
 
-#### Calibration dataset
+### Download Calibration dataset (only if you are doing quantization)
+
+#### CM METHOD
+The following MLCommons CM commands can be used to programmatically download the calibration dataset.
+
+```
+cm run script --tags=get,dataset,coco2014,_calibration --outdirname=coco2014
+```
+
+
+#### MANUAL METHOD
 
 We provide a script to download the calibration captions and images. To download only the captions:
 ```bash
 cd $SD_FOLDER/tools
-./download-coco-2014-calibration.sh
+./download-coco-2014-calibration.sh -n <number_of_workers>
 ```
-To download only the captions and images:
+
+To download both the captions and images:
 ```bash
 cd $SD_FOLDER/tools
 ./download-coco-2014-calibration.sh -i -n <number_of_workers>

tools/upscale_coco/coco.py

@@ -317,7 +317,7 @@ def showAnns(self, anns):
                     v = kp[2::3]
                     for sk in sks:
                         if np.all(v[sk] > 0):
-                            plt.plot(x[sk], y[sk], linewidth=3, color=c)
+                            plt.plot(x[sk], y[sk], linewidth=3, color=c,label=f"keypoint {sk}")
                     plt.plot(
                         x[v > 0],
                         y[v > 0],
@@ -336,6 +336,10 @@ def showAnns(self, anns):
                         markeredgecolor=c,
                         markeredgewidth=2,
                     )
+                    plt.xlabel("X Coordinate")
+                    plt.ylabel("Y Coordinate") 
+                    print("Script is running correctly!")  
+                    plt.show()             
             p = PatchCollection(
                 polygons,
                 facecolor=color,