How to verify an update is actually faster

[krausest]

I'm trying to cover here some suggestions how to benchmark an updated implementation.
The main enemy of this benchmark is noise. Something like +- 0.1 seems inevitable, sometimes even +. 0.2 in the total results might be attributed to noise.

How to reduce noise:

1. Make sure your system has no processes consuming much CPU. Quit all browser instances, vs code or IDEs, e-mail-apps and so only. For the official runs I'm even stopping one drive. You may stop search indexing sudo mdutil -i off / or set up an exception such that the js-framwork-benchmark folder is not indexed.
2. At least on OSX it seems like it currently helps to reboot the system and wait until the high CPU usage of the background processes stops.

How to measure an update:
Let's say you want to improve vanillajs (this would be a very welcome PR 😄 ).

1. Copy the old implementation cp -r frameworks/keyed/vanillajs frameworks/keyed/vanillajs-old.
2. Apply your changes or framework version update
3. Run both implementations to compare the results: npm run bench -- --framework keyed/vanillajs keyed/vanillajs-old
4. Look at the results: npm run results and open http://localhost:8080/webdriver-ts-results/dist/index.html
5. Filter the results to your framework. You can use the compare link below to check if the difference is statistically significant.

How to improve measurement
Let's say you're focused on a specific benchmark like create rows:

1. Run just that benchmark npm run bench -- --count 25 --benchmark 01_ --framework keyed/vanillajs keyed/vanillajs-old with increased number of samples. I can't do that for a full run, but to find out whether an update works this can improve statistical testing.

[mksunny1]

I just copied your command verbatim and this is what I got with Vanillajs-3. This is much better than the full run!

Hoping these tips are present (or visible) in the readme? I did not notice before...

[syduki]

How to reduce noise:
Make sure your system has no processes consuming much CPU.
For the official runs I'm even stopping one drive.

I would also advise to minimize the activity of high priority processes, like network and audio stack, thus it would be better to have the network disabled and no audio output from any applications.