Update to CUDA 12.4.1 [14.0.x] #9051
Open
+41
−29
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
backport #9046 |
|
enable gpu |
|
A new Pull Request was created by @fwyzard for branch IB/CMSSW_14_0_X/master. @cmsbuild, @iarspider, @aandvalenzuela, @smuzaffar can you please review it and eventually sign? Thanks.
|
|
cms-bot internal usage |
|
please test |
|
-1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14648e/37929/summary.html External BuildI found compilation error when building: [961/1148] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.4.0-db00bd44f20c40655446378926308f3f/bin/nvcc -forward-unknown-to-host-compiler -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DPROTOBUF_USE_DLLS -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -Donnxruntime_providers_cuda_EXPORTS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/protobuf/3.21.9-999e041f1a53b3ff94ee65a9cc8b7a2c/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cudnn/8.8.0.121-acaa18b242f7c97b443d981c456c94c4/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/cutlass-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/cutlass-src/examples -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.4.0-db00bd44f20c40655446378926308f3f/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/mp11-src/include -cudart shared --expt-relaxed-constexpr --Werror default-stream-launch -Xcudafe "--diag_suppress=bad_friend_decl" -Xcudafe "--diag_suppress=unsigned_compare_with_zero" -Xcudafe "--diag_suppress=expr_has_no_effect" -O3 -DNDEBUG --generate-code=arch=compute_60,code=[compute_60,sm_60] --generate-code=arch=compute_70,code=[compute_70,sm_70] --generate-code=arch=compute_75,code=[compute_75,sm_75] --generate-code=arch=compute_80,code=[compute_80,sm_80] --generate-code=arch=compute_89,code=[compute_89,sm_89] -Xcompiler=-fPIC --diag-suppress 554 --compiler-options -Wall --compiler-options -Wno-deprecated-copy --compiler-options -Wno-nonnull-compare -Xcompiler -Wno-nonnull-compare --threads "" -Xcompiler -Wno-reorder -Xcompiler -Wno-error=sign-compare -Werror all-warnings -MD -MT CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/ngram_repeat_block_impl.cu.o -MF CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/ngram_repeat_block_impl.cu.o.d -x cu -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/ngram_repeat_block_impl.cu -o CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/ngram_repeat_block_impl.cu.o
[962/1148] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.4.0-db00bd44f20c40655446378926308f3f/bin/nvcc -forward-unknown-to-host-compiler -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DPROTOBUF_USE_DLLS -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -Donnxruntime_providers_cuda_EXPORTS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/protobuf/3.21.9-999e041f1a53b3ff94ee65a9cc8b7a2c/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cudnn/8.8.0.121-acaa18b242f7c97b443d981c456c94c4/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/cutlass-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/cutlass-src/examples -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.4.0-db00bd44f20c40655446378926308f3f/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/mp11-src/include -cudart shared --expt-relaxed-constexpr --Werror default-stream-launch -Xcudafe "--diag_suppress=bad_friend_decl" -Xcudafe "--diag_suppress=unsigned_compare_with_zero" -Xcudafe "--diag_suppress=expr_has_no_effect" -O3 -DNDEBUG --generate-code=arch=compute_60,code=[compute_60,sm_60] --generate-code=arch=compute_70,code=[compute_70,sm_70] --generate-code=arch=compute_75,code=[compute_75,sm_75] --generate-code=arch=compute_80,code=[compute_80,sm_80] --generate-code=arch=compute_89,code=[compute_89,sm_89] -Xcompiler=-fPIC --diag-suppress 554 --compiler-options -Wall --compiler-options -Wno-deprecated-copy --compiler-options -Wno-nonnull-compare -Xcompiler -Wno-nonnull-compare --threads "" -Xcompiler -Wno-reorder -Xcompiler -Wno-error=sign-compare -Werror all-warnings -MD -MT CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/relative_attn_bias_impl.cu.o -MF CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/relative_attn_bias_impl.cu.o.d -x cu -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/relative_attn_bias_impl.cu -o CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/relative_attn_bias_impl.cu.o
[963/1148] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.4.0-db00bd44f20c40655446378926308f3f/bin/nvcc -forward-unknown-to-host-compiler -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DPROTOBUF_USE_DLLS -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -Donnxruntime_providers_cuda_EXPORTS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/protobuf/3.21.9-999e041f1a53b3ff94ee65a9cc8b7a2c/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cudnn/8.8.0.121-acaa18b242f7c97b443d981c456c94c4/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/cutlass-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/cutlass-src/examples -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.4.0-db00bd44f20c40655446378926308f3f/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/mp11-src/include -cudart shared --expt-relaxed-constexpr --Werror default-stream-launch -Xcudafe "--diag_suppress=bad_friend_decl" -Xcudafe "--diag_suppress=unsigned_compare_with_zero" -Xcudafe "--diag_suppress=expr_has_no_effect" -O3 -DNDEBUG --generate-code=arch=compute_60,code=[compute_60,sm_60] --generate-code=arch=compute_70,code=[compute_70,sm_70] --generate-code=arch=compute_75,code=[compute_75,sm_75] --generate-code=arch=compute_80,code=[compute_80,sm_80] --generate-code=arch=compute_89,code=[compute_89,sm_89] -Xcompiler=-fPIC --diag-suppress 554 --compiler-options -Wall --compiler-options -Wno-deprecated-copy --compiler-options -Wno-nonnull-compare -Xcompiler -Wno-nonnull-compare --threads "" -Xcompiler -Wno-reorder -Xcompiler -Wno-error=sign-compare -Werror all-warnings -MD -MT CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/skip_layer_norm_impl.cu.o -MF CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/skip_layer_norm_impl.cu.o.d -x cu -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/skip_layer_norm_impl.cu -o CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/skip_layer_norm_impl.cu.o
[964/1148] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.4.0-db00bd44f20c40655446378926308f3f/bin/nvcc -forward-unknown-to-host-compiler -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DPROTOBUF_USE_DLLS -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -Donnxruntime_providers_cuda_EXPORTS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/protobuf/3.21.9-999e041f1a53b3ff94ee65a9cc8b7a2c/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cudnn/8.8.0.121-acaa18b242f7c97b443d981c456c94c4/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/cutlass-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/cutlass-src/examples -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.4.0-db00bd44f20c40655446378926308f3f/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/build/_deps/mp11-src/include -cudart shared --expt-relaxed-constexpr --Werror default-stream-launch -Xcudafe "--diag_suppress=bad_friend_decl" -Xcudafe "--diag_suppress=unsigned_compare_with_zero" -Xcudafe "--diag_suppress=expr_has_no_effect" -O3 -DNDEBUG --generate-code=arch=compute_60,code=[compute_60,sm_60] --generate-code=arch=compute_70,code=[compute_70,sm_70] --generate-code=arch=compute_75,code=[compute_75,sm_75] --generate-code=arch=compute_80,code=[compute_80,sm_80] --generate-code=arch=compute_89,code=[compute_89,sm_89] -Xcompiler=-fPIC --diag-suppress 554 --compiler-options -Wall --compiler-options -Wno-deprecated-copy --compiler-options -Wno-nonnull-compare -Xcompiler -Wno-nonnull-compare --threads "" -Xcompiler -Wno-reorder -Xcompiler -Wno-error=sign-compare -Werror all-warnings -MD -MT CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/tensorrt_fused_multihead_attention/mha_runner.cu.o -MF CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/tensorrt_fused_multihead_attention/mha_runner.cu.o.d -x cu -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/tensorrt_fused_multihead_attention/mha_runner.cu -o CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-fcb60177485aaddd6dfade8bf135aeb9/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/bert/tensorrt_fused_multihead_attention/mha_runner.cu.o
ninja: build stopped: subcommand failed.
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.vDonL5 (%build)
RPM build errors:
line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: external+onnxruntime+1.14.1-fcb60177485aaddd6dfade8bf135aeb9
Macro expanded in comment on line 365: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}
|
|
please test |
|
Pull request #9051 was updated. |
|
-1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14648e/37945/summary.html External BuildI found compilation error when building: -- Check if compiler accepts -pthread - yes -- Found Geant4: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/geant4/11.1.2-f1592688174d5183c351e58fb91ca8c3/lib64/cmake/Geant4/Geant4Config.cmake (found version "11.1.2") -- Found nlohmann_json: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/json/3.11.3-627865c7a90221218546ea41a85dfb9a/share/cmake/nlohmann_json/nlohmann_jsonConfig.cmake (found suitable version "3.11.3", minimum required is "3.7.0") -- Found Python: /usr/bin/python3.6 (found suitable version "3.6.8", minimum required is "3.6") found components: Interpreter -- Configuring incomplete, errors occurred! error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Kldb1Y (%build) RPM build errors: line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: external+celeritas+v0.4.1-565cdff813b11a60cde5971b97a7a139 Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Kldb1Y (%build) |
Looking at the log, this seems due to
Who is responsible for "celeritas" ? |
Also, why does it find python 3.6 ? |
|
Pull request #9051 was updated. |
|
please test |
|
Pull request #9051 was updated. |
|
please test with cms-sw/cmssw#44355 |
|
hold
|
|
Pull request has been put on hold by @antoniovilela |
|
I missed the ORP (on vacation), but yes, please hold this. My intent is to use the 14.1.x build and externals to evaluate the impact on the HLT performance, and request the backport only if that check is positive. |
|
please test with cms-sw/cmssw#44355 |
|
-1 Failed Tests: GpuUnitTests GPU Unit TestsI found 1 errors in the following unit tests: ---> test testTFVisibleDevicesCUDA had ERRORS Comparison SummarySummary:
GPU Comparison SummarySummary:
|
Update to CUDA 12.4.1: * CUDA runtime version 12.4.127 * NVIDIA drivers version 550.54.15 See https://docs.nvidia.com/cuda/archive/12.4.1/cuda-toolkit-release-notes/index.html for the full CUDA 12.4.1 release notes and change log.
|
Pull request #9051 was updated. |
fwyzard
changed the title
|
please test with cms-sw/cmssw#44355 |
|
-1 Failed Tests: UnitTests GpuUnitTests Unit TestsI found 1 errors in the following unit tests: ---> test TestGeneratorInterfacePythia8ConcurrentGeneratorFilter had ERRORS GPU Unit TestsI found 1 errors in the following unit tests: ---> test testTFVisibleDevicesCUDA had ERRORS Comparison SummarySummary:
GPU Comparison SummarySummary:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update CMake to version 3.28.3.
Update to CUDA 12.4.1:
See https://docs.nvidia.com/cuda/archive/12.4.1/cuda-toolkit-release-notes/index.html for the full CUDA 12.4.0 release notes and change log.
Force CUDA to consider only the major/minor version of GCC, to work around a problem in the GCC version checks inside
cudafe++, where GCC 12.3.1 is not recognised as equivalent to GCC 12.3.0.Rename the CUDA-related flags used in the spec files to avoid potential problems when adding,
removing or reordering them.
Update cuDNN to version 8.9:
See https://docs.nvidia.com/deeplearning/cudnn/archives/cudnn-897/release-notes/index.html for the release notes and change log.
Update NVIDIA gdrcopy to v2.4.1
Add support for new hardware (Grace Hopper, BlueField 3 NICs and DPUs).
Add support for recent OS (kernel, RHEL, SUSE).
Various bug fixes.
See https://github.com/NVIDIA/gdrcopy/releases/tag/v2.4 and https://github.com/NVIDIA/gdrcopy/releases/tag/v2.4.1 for the release notes and change log.
Update ONNX runtime to version 1.17.1:
Older version of ONNX runtime fail to compile with CUDA 12.4.
On the other hand, ONNX 1.17.1 requires CMAKE 3.26 or later, and cuDNN 8.9 or later.
This version requires two more workarounds:
-Wno-error=maybe-uninitializedis needed to a void a (hopefully false positive) warning about a potentially uninitialised variable with cuDNN 8.9 and 9.0-Donnxruntime_NVCC_THREADS=0is needed because the default isON, causing nvcc to be called asnvcc ... --threads ON ..., which causes an error.This version requires an update inside CMSSW, implemented in cms-sw/cmssw#44355.
Add missing thrust include in PyTorch
Backport of pytorch/pytorch#121421.
Backport status
Backport of #9046 and #9126 to the data taking release.