Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MaxMemoryPreload monitoring shows noticeable increase when large number of packages are rebuilt #46359

Open
makortel opened this issue Oct 11, 2024 · 15 comments · May be fixed by #46542
Open

MaxMemoryPreload monitoring shows noticeable increase when large number of packages are rebuilt #46359

makortel opened this issue Oct 11, 2024 · 15 comments · May be fixed by #46542

Comments

@makortel
Copy link
Contributor

Observed in #46200 (comment).

@makortel
Copy link
Contributor Author

assign core

@cmsbuild
Copy link
Contributor

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

@cmsbuild
Copy link
Contributor

cms-bot internal usage

@cmsbuild
Copy link
Contributor

A new Issue was created by @makortel.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor Author

Here is an example from #46204 (comment)
image

@makortel
Copy link
Contributor Author

With @Dr15Jones we hypothesized maybe the FileInPath could lead to this increase?

@Dr15Jones
Copy link
Contributor

Another reason could be the PluginManager cache would also include reading from the workspace's lib directory and therefore increase its working memory.

@makortel
Copy link
Contributor Author

Another example from #46388 (comment)
image

@makortel
Copy link
Contributor Author

makortel commented Oct 18, 2024

Interestingly #46241 (comment), that was a test with full_cmssw = true (I checked both compilation log and unit test report to see that many packages were indeed rebuilt), does not show noticeable increase
image

@makortel
Copy link
Contributor Author

I was able to reproduce locally a "rebuild of most of the universe in a developer area" leading to ~30 MB higher max memory used than "empty developer area".

@makortel
Copy link
Contributor Author

I ran IgProf for both cases, below are the MEM_LIVE after 10 events

and the cause is in edmplugin::CacheParser::read(), see baseline vs rebuild. I believe it is this overload

void CacheParser::read(std::istream& iIn,
const std::filesystem::path& iDirectory,
CacheParser::CategoryToInfos& iOut) {

@makortel
Copy link
Contributor Author

So the cost is that the developer area's .edmplugincache becomes large, and we read that in memory in addition to the .edmplugincache from the release area?

In my "rebuild" developer area the .edmplugincache is 1.3 MB, whereas the $CMSSW_RELEASE_BASE/lib/slc7_amd64_gcc12/.edmplugincache is 1.5 MB.

@makortel
Copy link
Contributor Author

I wonder if there would be any easy way to exclude the contribution from CacheParser::read() (other than starting the monitoring later in the job)

@makortel
Copy link
Contributor Author

Instrumenting some of the PluginManager code, I see (in my rebuild area) that the job reads

  • Developer area biglib/slc7_amd64_gcc12/.edmplugincache
  • Developer area lib/slc7_amd64_gcc12/.edmplugincache
  • Developer area lib/slc7_amd64_gcc12/.poisonededmplugincache
  • Release area biglib/slc7_amd64_gcc12/.edmplugincache
  • Release area lib/slc7_amd64_gcc12/.edmplugincache

So with nearly all of CMSSW build, most of the plugins have 3 entries in the CategoryToInfos: local, local poisoned, and release.

@makortel
Copy link
Contributor Author

#46542 proposes to exclude the PluginManager cache from the MaxMemoryPreload monitoring as a workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants