Skip to content

Commit

Permalink
Results from self hosted Github actions
Browse files Browse the repository at this point in the history
  • Loading branch information
@arjunsuresh
arjunsuresh committed Jan 9, 2025
1 parent 9f6b3d2 commit f4380aa
Show file tree
Hide file tree
Showing 14 changed files with 444 additions and 444 deletions.
2 changes: 1 addition & 1 deletion ...ns/measurements/gh_action-reference-cpu-pytorch_v2.5.1-default_config/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
| Model | Scenario | Accuracy | Throughput | Latency (in ms) |
|---------------|------------|------------------------------------|--------------|-------------------|
| llama2-70b-99 | offline | (61.7021, 37.9679, 39.3617, 610.0) | 0.362 | - |
| llama2-70b-99 | offline | (61.7021, 37.9679, 39.3617, 610.0) | 0.43 | - |
4 changes: 2 additions & 2 deletions ...ion-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ pip install -U cmind

cm rm cache -f

cm pull repo gateoverflow@mlperf-automations --checkout=ee7be856e5def9e46bc4535b6128ec342f6931db
cm pull repo gateoverflow@mlperf-automations --checkout=0b0073c83f137dabd0f2abb0866cf1a7193c8100

cm run script \
--tags=app,mlperf,inference,generic,_reference,_llama2-70b-99,_pytorch,_cpu,_test,_r5.0-dev_default,_bfloat16,_offline \
Expand Down Expand Up @@ -107,4 +107,4 @@ Model Precision: fp32
`TOKENS_PER_SAMPLE`: `610.0`, Required accuracy for closed division `>= 265.005` and `<= 323.895`

### Performance Results
`Samples per second`: `0.362211`
`Samples per second`: `0.43025`
10 changes: 5 additions & 5 deletions ...on-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline/accuracy_console.out
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
INFO:datasets:PyTorch version 2.5.1+cpu available.
Loading dataset...
Finished loading dataset.
Loading checkpoint shards: 0%| | 0/15 [00:00<?, ?it/s]Loading checkpoint shards: 7%|▋ | 1/15 [00:01<00:27, 2.00s/it]Loading checkpoint shards: 13%|█▎ | 2/15 [00:04<00:26, 2.01s/it]Loading checkpoint shards: 20%|██ | 3/15 [00:05<00:23, 1.98s/it]Loading checkpoint shards: 27%|██▋ | 4/15 [00:07<00:20, 1.90s/it]Loading checkpoint shards: 33%|███▎ | 5/15 [00:09<00:18, 1.86s/it]Loading checkpoint shards: 40%|████ | 6/15 [00:11<00:16, 1.83s/it]Loading checkpoint shards: 47%|████▋ | 7/15 [00:13<00:14, 1.81s/it]Loading checkpoint shards: 53%|█████▎ | 8/15 [00:15<00:13, 1.92s/it]Loading checkpoint shards: 60%|██████ | 9/15 [00:17<00:11, 1.93s/it]Loading checkpoint shards: 67%|██████▋ | 10/15 [00:19<00:10, 2.04s/it]Loading checkpoint shards: 73%|███████▎ | 11/15 [00:21<00:08, 2.10s/it]Loading checkpoint shards: 80%|████████ | 12/15 [00:23<00:06, 2.10s/it]Loading checkpoint shards: 87%|████████▋ | 13/15 [00:25<00:04, 2.02s/it]Loading checkpoint shards: 93%|█████████▎| 14/15 [00:31<00:03, 3.04s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:31<00:00, 2.20s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:31<00:00, 2.08s/it]
Loading checkpoint shards: 0%| | 0/15 [00:00<?, ?it/s]Loading checkpoint shards: 7%|▋ | 1/15 [00:02<00:28, 2.01s/it]Loading checkpoint shards: 13%|█▎ | 2/15 [00:03<00:25, 1.97s/it]Loading checkpoint shards: 20%|██ | 3/15 [00:05<00:23, 1.94s/it]Loading checkpoint shards: 27%|██▋ | 4/15 [00:07<00:20, 1.89s/it]Loading checkpoint shards: 33%|███▎ | 5/15 [00:09<00:18, 1.84s/it]Loading checkpoint shards: 40%|████ | 6/15 [00:11<00:16, 1.82s/it]Loading checkpoint shards: 47%|████▋ | 7/15 [00:12<00:14, 1.81s/it]Loading checkpoint shards: 53%|█████▎ | 8/15 [00:15<00:14, 2.00s/it]Loading checkpoint shards: 60%|██████ | 9/15 [00:17<00:12, 2.03s/it]Loading checkpoint shards: 67%|██████▋ | 10/15 [00:19<00:10, 2.01s/it]Loading checkpoint shards: 73%|███████▎ | 11/15 [00:21<00:07, 1.98s/it]Loading checkpoint shards: 80%|████████ | 12/15 [00:23<00:05, 1.93s/it]Loading checkpoint shards: 87%|████████▋ | 13/15 [00:25<00:03, 1.95s/it]Loading checkpoint shards: 93%|█████████▎| 14/15 [00:29<00:02, 2.78s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:30<00:00, 2.02s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:30<00:00, 2.01s/it]
INFO:Llama-70B-MAIN:Starting Benchmark run
/home/cmuser/venv/cm/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:628: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
warnings.warn(
Expand All @@ -22,7 +22,7 @@ IssueQuery started with 1 samples
IssueQuery done
Saving outputs to run_outputs/q0.pkl
Samples run: 1
BatchMaker time: 0.015237092971801758
Inference time: 789.5984785556793
Postprocess time: 0.07323002815246582
==== Total time: 789.6869456768036
BatchMaker time: 0.008818864822387695
Inference time: 837.1090745925903
Postprocess time: 0.026335716247558594
==== Total time: 837.1442291736603
Expand Down
52 changes: 26 additions & 26 deletions ...s/gh_action-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline/cm-deps.mmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,15 @@ graph TD
get-mlperf-inference-src,4b57186581024797 --> get,git,repo,_branch.master,_repo.https://github.com/mlcommons/inference
get-mlperf-inference-utils,e341e5f86d8342e5 --> get,mlperf,inference,src
app-mlperf-inference,d775cac873ee4231_(_reference,_llama2-70b-99,_pytorch,_cpu,_test,_r5.0-dev_default,_bfloat16,_offline_) --> get,mlperf,inference,utils
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> detect,os
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> detect,cpu
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,sys-utils-cm
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,python
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> detect,cpu
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,sys-utils-cm
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,python
get-generic-python-lib,94b62a682bc44791_(_torch_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_torch
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_torch
get-generic-python-lib,94b62a682bc44791_(_torchvision_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_torchvision
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_torchvision
get-ml-model-huggingface-zoo,53cf8252a443446a_(_clone-repo,_model-stub.meta-llama/Llama-2-70b-chat-hf_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_huggingface_hub_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
Expand All @@ -32,7 +32,7 @@ graph TD
get-ml-model-huggingface-zoo,53cf8252a443446a_(_clone-repo,_model-stub.meta-llama/Llama-2-70b-chat-hf_) --> get,generic-python-lib,_huggingface_hub
get-ml-model-huggingface-zoo,53cf8252a443446a_(_clone-repo,_model-stub.meta-llama/Llama-2-70b-chat-hf_) --> get,git,repo,_lfs,_repo.https://huggingface.co/meta-llama/Llama-2-70b-chat-hf
get-ml-model-llama2,5db97be9f61244c6_(_pytorch_) --> get,ml-model,huggingface,zoo,_clone-repo,_model-stub.meta-llama/Llama-2-70b-chat-hf
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,ml-model,llama2,raw,_pytorch
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,ml-model,llama2,raw,_pytorch
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,sys-utils-cm
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.pyarrow_) --> get,python3
Expand All @@ -42,7 +42,7 @@ graph TD
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> get,python3
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,generic-python-lib,_package.transformers
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> download-and-extract,_rclone,_url.mlc-inference:mlcommons-inference-wg-public/open_orca
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,preprocessed,dataset,openorca,_validation,_mlcommons
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,preprocessed,dataset,openorca,_validation,_mlcommons
generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,cpu
Expand All @@ -53,40 +53,40 @@ graph TD
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,mlcommons,inference,src
get-mlperf-inference-sut-configs,c2fbf72009e2445b --> get,cache,dir,_name.mlperf-inference-sut-configs
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,sut,configs
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> generate,user-conf,mlperf,inference
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,loadgen
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> generate,user-conf,mlperf,inference
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,loadgen
get-mlperf-inference-src,4b57186581024797 --> detect,os
get-mlperf-inference-src,4b57186581024797 --> get,python3
get-mlperf-inference-src,4b57186581024797 --> get,git,repo,_branch.master,_repo.https://github.com/mlcommons/inference
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,mlcommons,inference,src
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,mlcommons,inference,src
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,mlcommons,inference,src
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,mlcommons,inference,src
get-generic-python-lib,94b62a682bc44791_(_package.psutil_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.psutil
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.psutil
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.transformers
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.transformers
get-generic-python-lib,94b62a682bc44791_(_package.datasets_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.datasets
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.datasets
get-generic-python-lib,94b62a682bc44791_(_package.sentencepiece_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.sentencepiece
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.sentencepiece
get-generic-python-lib,94b62a682bc44791_(_package.protobuf_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.protobuf
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.protobuf
get-generic-python-lib,94b62a682bc44791_(_package.accelerate_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.accelerate
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.accelerate
get-generic-python-lib,94b62a682bc44791_(_package.absl-py_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.absl-py
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.absl-py
get-generic-python-lib,94b62a682bc44791_(_package.evaluate_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.evaluate
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.evaluate
get-generic-python-lib,94b62a682bc44791_(_package.nltk_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.nltk
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.nltk
get-generic-python-lib,94b62a682bc44791_(_package.numpy_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.numpy
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.numpy
get-generic-python-lib,94b62a682bc44791_(_package.rouge-score_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.rouge-score
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.rouge-score
get-generic-python-lib,94b62a682bc44791_(_package.more-itertools_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.more-itertools
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.more-itertools
get-generic-python-lib,94b62a682bc44791_(_package.compressed_tensors_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> get,generic-python-lib,_package.compressed_tensors
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.compressed_tensors
detect-cpu,586c8a43320142f7 --> detect,os
benchmark-program,19f369ef47084895 --> detect,cpu
benchmark-program-mlperf,cfff0132a8aa4018 --> benchmark-program,program
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_llama2-70b-99,_cpu,_bfloat16_) --> benchmark-mlperf
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_offline,_cpu,_llama2-70b-99,_bfloat16_) --> benchmark-mlperf
Binary file modified ...n-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline/cm-deps.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit f4380aa

Please sign in to comment.