Use pluginSubject for system index interaction and remove usages of stashContext

[cwperks]

Description

This PR removes direct usages of ThreadContext.stashContext and implements a new extension point in core (IdentityAwarePlugin) which assigns this plugin a subject to use for the purposes of executing transport actions outside of the authenticated user context.

With this subject, job-scheduler is permitted to access its own system indices without having to use ThreadContext.stashContext explicitly.

Related Issues

Resolves opensearch-project/opensearch-plugins#238

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 24 lines in your changes missing coverage. Please review.

Project coverage is 36.63%. Comparing base (a659592) to head (6bd012d).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...rch/jobscheduler/transport/RunAsSubjectClient.java	0.00%	19 Missing ⚠️
...rg/opensearch/jobscheduler/JobSchedulerPlugin.java	0.00%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #714      +/-   ##
============================================
- Coverage     37.67%   36.63%   -1.05%     
+ Complexity      135      133       -2     
============================================
  Files            22       23       +1     
  Lines          1189     1212      +23     
  Branches        109      109              
============================================
- Hits            448      444       -4     
- Misses          704      731      +27     
  Partials         37       37              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cwperks]

Not sure why this is necessary here just yet.

[cwperks]

This is needed to resolve a race condition discovered in GetLockMultiNodeRestIT.testGetLockRestAPI. This test will create and release a lock 10 times in a loop. Without adding this RefreshPolicy, when the security plugin is installed it can fail to find the lock the was just created. By setting this RefreshPolicy to IMMEDIATE, we can guarantee that this newly created document is available for search (including GET actions)