fix: 17331: Improve in-memory virtual map mode to run garbage collection for merged copies

[artemananiev]

Fix summary:

• Reworked VirtualNodeCache.filterMutations(), so it can now handle a case, when there is an old unreleased version of the cache
• Cache mutations are now two-directional lists (prev and next)
• In VirtualPipeline, every virtual root copy is first compacted (if needed), then either flushed or merged. Previously compaction was only applied to copies, which are eligible to flush
• A new unit test is provided. A few changes to existing unit tests

Fixes: #17331
Signed-off-by: Artem Ananev [email protected]

[codecov]

Codecov Report

Attention: Patch coverage is 84.00000% with 16 lines in your changes missing coverage. Please review.

Project coverage is 68.74%. Comparing base (3aac595) to head (e9ffcc0).
Report is 36 commits behind head on main.

Files with missing lines	Patch %	Lines
...ds/virtualmap/internal/cache/VirtualNodeCache.java	83.33%	3 Missing and 7 partials ⚠️
...ds/virtualmap/internal/merkle/VirtualRootNode.java	85.18%	2 Missing and 2 partials ⚠️
.../virtualmap/internal/pipeline/VirtualPipeline.java	84.61%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17405      +/-   ##
============================================
+ Coverage     67.57%   68.74%   +1.16%     
- Complexity    22130    22589     +459     
============================================
  Files          2588     2610      +22     
  Lines         96654    97232     +578     
  Branches      10099    10130      +31     
============================================
+ Hits          65318    66845    +1527     
+ Misses        27589    26565    -1024     
- Partials       3747     3822      +75     

Files with missing lines	Coverage Δ
...rlds/virtualmap/internal/pipeline/VirtualRoot.java	0.00% <ø> (ø)
.../virtualmap/internal/pipeline/VirtualPipeline.java	56.77% <84.61%> (-3.23%)	⬇️
...ds/virtualmap/internal/merkle/VirtualRootNode.java	57.77% <85.18%> (+0.23%)	⬆️
...ds/virtualmap/internal/cache/VirtualNodeCache.java	78.85% <83.33%> (+1.95%)	⬆️

... and 306 files with indirect coverage changes

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +1.37% (target: -1.00%)	✅ 94.00%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (3aac595)	96437	68913	71.46%
Head commit (e9ffcc0)	93366 (-3071)	68001 (-912)	72.83% (+1.37%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#17405)	100	94	94.00%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

[artemananiev]

Here is the core of the fix. Every copy, even if not the latest one, is first compacted if needed (if its estimated size exceeds the threshold). Then it can be either flushed (if the size is still more than the threshold after compaction, and the copy is the oldest) or merged (if the size is under threshold), or nothing (if the size exceeds the threshold, but the copy is not the oldest).