Ensure consistency of system flag on IndexMetadata after diff is applied

[cwperks]

Description

This PR ensures the consistency of the isSystem flag between leader and follower after the IndexMetadataDiff is applied when computing new cluster state.

This inconsistency can appear during rolling upgrades if nodes of a new version know of an index as a system index, but nodes of a previous version treat the same index as a regular index. For example, if a plugin retroactively declares an index to be a system index through SystemIndexPlugin.getSystemIndexDescriptors (Example PR) there can be a discrepancy in the IndexMetadata (for the same version) between the cluster manager node and other nodes in the cluster.

The reason this happens is because when the IndexMetadataDiff is applied, its taking the value from the previous metadata on the node instead of the new metadata that it has received from the incoming publishRequest. As a result, the diff here is being computed incorrectly. It should take the value from the diff instead of from the previous index metadata.

Related Issues

Resolves #16643

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Bukhtawar]

Thanks for the quick fix. Ship it!

[github-actions]

✅ Gradle check result for b526a45: SUCCESS

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.04%. Comparing base (10873f1) to head (7ff0651).
Report is 13 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16644      +/-   ##
============================================
- Coverage     72.15%   72.04%   -0.11%     
- Complexity    65145    65147       +2     
============================================
  Files          5315     5318       +3     
  Lines        303573   303828     +255     
  Branches      43925    43960      +35     
============================================
- Hits         219039   218891     -148     
- Misses        66587    67022     +435     
+ Partials      17947    17915      -32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[owaiskazi19]

@cwperks this change needs a changelog entry

[cwperks]

@cwperks this change needs a changelog entry

Added

[github-actions]

✅ Gradle check result for 7ff0651: SUCCESS