Run profiling on baseline release and save root output for comparison of RECO process for PRs

[gartung]

I need to set the number of events to 100 and I wanted to avoid running that many for every IB.

Step 1 and step 2 are run for each job so technically the input for step3 RECO is different. On average the results should be the same but not with the limited stats of 10 events.

Saving the output root files and json files from the two workflows used in profiling would allow profiling the PR with the same RECO input as the baseline.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @gartung.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[gartung]

cms-sw/cms-bot#2366

[gartung]

Using the same input step2.root for the RECO step still results in larger differences in module timing than expected.
Perhaps taking difference of the fraction of the total time would be a better metric.

[makortel]

Are the step1 and/or step2 run with multiple threads in the profiling workflow? On a single thread their output should be exactly the same (which is verified in PR test comparisons).

[gartung]

All jobs are run single threaded.

[gartung]

The differences in total job time might be caused by running on different VMs. I am modifying the PR profiling script to run both IB and IB+PR profiling on the same VM.