Fix init encryption master key

[ylwu-amzn]

Description

1. reduce latch wait timeout from 5 seconds to 1 seconds when init encryption master key
2. create new encryption master key if not found

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 96.29630% with 2 lines in your changes missing coverage. Please review.

Project coverage is 81.26%. Comparing base (f28bb74) to head (724f4a8).
Report is 1 commits behind head on main.

❗ Current head 724f4a8 differs from pull request most recent head 832dde9

Please upload reports for the commit 832dde9 to get more accurate results.

Files	Patch %	Lines
.../opensearch/ml/engine/encryptor/EncryptorImpl.java	96.07%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2554      +/-   ##
============================================
- Coverage     81.31%   81.26%   -0.06%     
+ Complexity     6094     6086       -8     
============================================
  Files           573      573              
  Lines         25268    25306      +38     
  Branches       2666     2665       -1     
============================================
+ Hits          20547    20565      +18     
- Misses         3601     3616      +15     
- Partials       1120     1125       +5     

Flag	Coverage Δ
ml-commons	81.26% <96.29%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zane-neo]

Can we make sure the blocking here is safe? It won't have impact on other threads like event loop? Initiating a master key doesn't seem a long running job(writing index), why we're using a blocking fashion to do this?

[ylwu-amzn]

This is not a perfect fix. Just for quickly unblock query assistant. Did testing on single node and multi-node cluster, looks good. At least this part is same with before, just reduce timeout

[ylwu-amzn]

@rbhavna will have a long-term fix which will remove such await

[zane-neo]

@rbhavna please create an issue to track this.

[opensearch-trigger-bot]

The backport to feature/multi_tenancy failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-feature/multi_tenancy feature/multi_tenancy
# Navigate to the new working tree
cd .worktrees/backport-feature/multi_tenancy
# Create a new branch
git switch --create backport/backport-2554-to-feature/multi_tenancy
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 487f33a2e35e642429e9a3ea1eb0d715d542ea9f
# Push it to GitHub
git push --set-upstream origin backport/backport-2554-to-feature/multi_tenancy
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-feature/multi_tenancy

Then, create a pull request where the base branch is feature/multi_tenancy and the compare/head branch is backport/backport-2554-to-feature/multi_tenancy.