Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MTD Geometry: update DDD filtering for better memory performance #46531

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

MTD Geometry: update DDD filtering for better memory performance #46531

wants to merge 1 commit into from

Conversation

fabiocos
Copy link
Contributor

PR description:

This PR addresses #46512 , by:

  • remove vetoing no more present volumes (since v2);
  • separate namespace and name filtering, as suggested by @makortel, eliminating the double string creation if only namespace filtering is needed;
  • use locally string_view for comparisons to reduce memory usage.

PR validation:

Geometry/MTD* unit tests successfully run, producing geometry dumps consistent with the reference.

@cmsbuild cmsbuild added this to the CMSSW_14_2_X milestone Oct 28, 2024
@cmsbuild
Copy link
Contributor

cmsbuild commented Oct 28, 2024 •
edited
Loading

cms-bot internal usage

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46531/42394

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @fabiocos for master.

It involves the following packages:

  • Geometry/MTDNumberingBuilder (geometry, upgrade)

@Dr15Jones, @Moanwar, @bsunanda, @civanch, @cmsbuild, @kpedro88, @makortel, @mdhildreth, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks.
@bsunanda, @martinamalberti this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@fabiocos
Copy link
Contributor Author

please test

@makortel makortel mentioned this pull request Oct 28, 2024
Open
@cmsbuild
Copy link
Contributor

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c26682/42427/summary.html
COMMIT: d4baa1c
CMSSW: CMSSW_14_2_X_2024-10-28-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46531/42427/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 12 differences found in the comparisons
  • DQMHistoTests: Total files compared: 46
  • DQMHistoTests: Total histograms compared: 3569177
  • DQMHistoTests: Total failures: 380
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3568777
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
  • Checked 201 log files, 171 edm output root files, 46 DQM output files
  • TriggerResults: no differences found

@fabiocos
Copy link
Contributor Author

enable profiling

@fabiocos
Copy link
Contributor Author

please test

@makortel
Copy link
Contributor

From the logs of workflow 29634.0 step 3, the number of memory allocations goes down from 244090020 to 232478470, i.e. a reduction of 11.6 millions.

@fabiocos
Copy link
Contributor Author

as far as I can see in my standalone unit test on cmsdev45, there is no obvious significant impact on the speed of execution, in any case the caching of the strings is preserved to prevent the previous performance problems, that were fixed one year ago.

@bsunanda
Copy link
Contributor

+geometry

@fabiocos
Copy link
Contributor Author

@smuzaffar are two days normal for a profiling request? As pointed out by @makortel the memory benefit seems anyway visible by the simple memory report in log files, so in case I could abort the test

@smuzaffar
Copy link
Contributor

No @fabiocos , it is not normal. We had only one node to run IB and PR profiling jobs and there were many such jobs waiting for there turn. I now have add couple of more nodes to run profiling jobs

@smuzaffar
Copy link
Contributor

smuzaffar commented Oct 31, 2024 •
edited
Loading

please test

I can not find any trace of previous profiling jobs in jenkins so lets retry the tests

@cmsbuild
Copy link
Contributor

-1

Failed Tests: RelVals-INPUT
Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c26682/42507/summary.html
COMMIT: d4baa1c
CMSSW: CMSSW_14_2_X_2024-10-30-2300/el8_amd64_gcc12
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46531/42507/install.sh to create a dev area with all the needed externals and cmssw changes.

  • DAS Queries: The DAS query tests failed, see the summary page for details.

RelVals-INPUT

  • 2024.0000012024.000001_RunJetMET02024D_10k/step1_dasquery.log
  • 2024.0010012024.001001_RunZeroBias2024D_10k/step1_dasquery.log
  • 2024.1000012024.100001_RunJetMET02024C_10k/step1_dasquery.log
Expand to see more relval errors ...
  • 2024.101001
  • 2024.000001
  • 2024.001001
  • 2024.100001
  • 2024.101001

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 10 differences found in the comparisons
  • DQMHistoTests: Total files compared: 46
  • DQMHistoTests: Total histograms compared: 3569372
  • DQMHistoTests: Total failures: 410
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3568942
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
  • Checked 201 log files, 171 edm output root files, 46 DQM output files
  • TriggerResults: no differences found

@fabiocos
Copy link
Contributor Author

fabiocos commented Nov 5, 2024

input errors appear to be unrelated to this PR, the profiling seems to me inconclusive for the problem addressed by this PR.
@cms-sw/upgrade-l2 please check and comment or sign it

@fabiocos
Copy link
Contributor Author

fabiocos commented Nov 6, 2024

type performance-improvements

@civanch
Copy link
Contributor

civanch commented Nov 7, 2024

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Nov 7, 2024

-1

Failed Tests: UnitTests RelVals-INPUT
Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c26682/42637/summary.html
COMMIT: d4baa1c
CMSSW: CMSSW_14_2_X_2024-11-06-2300/el8_amd64_gcc12
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/46531/42637/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test GeometryMTDNumberingBuilderTestDriver had ERRORS

RelVals-INPUT

  • 2024.1010012024.101001_RunBTagMu2024C_10k/step1_dasquery.log
  • 2024.1020012024.102001_RunJetMET02024C_10k/step1_dasquery.log
  • 2024.2010012024.201001_RunBTagMu2024D_10k/step1_dasquery.log
Expand to see more relval errors ...
  • 2024.202001
  • 2024.101001
  • 2024.102001
  • 2024.201001
  • 2024.202001

Comparison Summary

Summary:

@fabiocos
Copy link
Contributor Author

fabiocos commented Nov 7, 2024

the unit test was broken by other concurrent developments

@fabiocos
Copy link
Contributor Author

fabiocos commented Nov 7, 2024

please test with #46608

@cmsbuild
Copy link
Contributor

cmsbuild commented Nov 7, 2024

-1

Failed Tests: RelVals-INPUT
Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c26682/42650/summary.html
COMMIT: d4baa1c
CMSSW: CMSSW_14_2_X_2024-11-07-1100/el8_amd64_gcc12
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/46531/42650/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

  • 2024.0020012024.002001_RunJetMET02024B_10k/step1_dasquery.log
  • 2024.002001DAS Error

Comparison Summary

Summary:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
code-checks-approved geometry-approved orp-pending pending-signatures performance-improvements requires-external tests-rejected upgrade-pending
Projects
None yet
Milestone
CMSSW_14_2_X
Development

Successfully merging this pull request may close these issues.
6 participants
@fabiocos @cmsbuild @makortel @bsunanda @smuzaffar @civanch