Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated changes to security #11

Open
wants to merge 32 commits into
base: pitsegmentsrest
Choose a base branch
from
Open

Updated changes to security #11

wants to merge 32 commits into from

Conversation

bharath-techie
Copy link
Owner

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dreamer-89 and others added 20 commits August 9, 2022 09:58
@dreamer-89
Add IndexShard#getLatestReplicationCheckpoint behind segrep enable fe…
a120dd2 
…ature flag (opensearch-project#4163)

* Add IndexShard#getLatestReplicationCheckpoint behind segrep enable feature flag

Signed-off-by: Suraj Singh <[email protected]>

* Address review comment. Move tests to SegmentReplicationIndexShardTests

Signed-off-by: Suraj Singh <[email protected]>

* Add segrep enbaled index settings in TargetServiceTests, SourceHandlerTests

Signed-off-by: Suraj Singh <[email protected]>
@navneet1v
Refactor the src and test of GeoBoundsAggregation on GeoPoint from se…
4751f3b 
…rver folder to geo module.

Refactors the GeoBoundsAggregation for geo_point types from the core server 
to the geo module.

Signed-off-by: Navneet Verma <[email protected]>
@sachinpkale
[Remote Store] Add validator that forces segment replication type bef…
1c3f034 
…ore enabling remote store (opensearch-project#4175)

* Add validator that forces segment replication type before enabling remote store

Signed-off-by: Sachin Kale <[email protected]>
@mch2
[Segment Replication] Fix OngoingSegmentReplications to key by alloca…
7dad063 
…tion ID instead of DiscoveryNode. (opensearch-project#4182)

* Fix OngoingSegmentReplications to key by allocation ID instead of DiscoveryNode.

This change fixes segrep to work with multiple shards per node by keying ongoing replications on
allocation ID.  This also updates cancel methods to ensure state is properly cleared on shard cancel.

Signed-off-by: Marc Handalian <[email protected]>

* Clean up cancel methods.

Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>
@sachinpkale
[Remote Store] Change remote_store setting validation message to make…
8ccedd8 
… it more clear (opensearch-project#4199)

* Change remote_store setting validation message to make it more clear

Signed-off-by: Sachin Kale <[email protected]>
@dreamer-89 @Poojita-Raj
[Bug] [Segment Replication] Update store metadata recovery diff logic…
b9a64cb 
… to ignore missing files causing exception (opensearch-project#4185)

* Update Store for segment replication dif

Signed-off-by: Poojita Raj <[email protected]>

Signed-off-by: Poojita Raj <[email protected]>

* Update recoveryDiff logic to ingore missing files causing exception on replica during copy

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments

Signed-off-by: Suraj Singh <[email protected]>

Signed-off-by: Poojita Raj <[email protected]>
Signed-off-by: Suraj Singh <[email protected]>
Co-authored-by: Poojita Raj <[email protected]>
@sachinpkale
[Remote Store] Add RemoteSegmentStoreDirectory to interact with remot…
5f2e66b 
…e segment store (opensearch-project#4020)

* Add RemoteSegmentStoreDirectory to interact with remote segment store

Signed-off-by: Sachin Kale <[email protected]>
@reta
Update Gradle to 7.5.1 (opensearch-project#4211)
0dbbd72 
Signed-off-by: Andriy Redko <[email protected]>

Signed-off-by: Andriy Redko <[email protected]>
@dependabot
Bump com.diffplug.spotless from 6.9.0 to 6.9.1 (opensearch-project#4210)
46d7a47 
Bumps com.diffplug.spotless from 6.9.0 to 6.9.1.

---
updated-dependencies:
- dependency-name: com.diffplug.spotless
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@Rishikesh1159
[Segment Replication] Adding PrimaryMode check before publishing chec…
96bfd1f 
…kpoint and processing a received checkpoint. (opensearch-project#4157)

* Adding PrimaryMode check before publishing checkpoint.

Signed-off-by: Rishikesh1159 <[email protected]>

* Applying spotless check

Signed-off-by: Rishikesh1159 <[email protected]>

* Moving segrep specific tests to SegmentReplicationIndexShardTests.

Signed-off-by: Rishikesh1159 <[email protected]>

* Adding logic and tests for rejecting checkpoints if shard is in PrimaryMode.

Signed-off-by: Rishikesh1159 <[email protected]>

* Applying ./gradlew :server:spotlessApply.

Signed-off-by: Rishikesh1159 <[email protected]>

* Applying ./gradlew :server:spotlessApply

Signed-off-by: Rishikesh1159 <[email protected]>

* Changing log level to warn in shouldProcessCheckpoint() of IndexShard.java class.

Signed-off-by: Rishikesh1159 <[email protected]>

* Removing unnecessary lazy logging in shouldProcessCheckpoint().

Signed-off-by: Rishikesh1159 <[email protected]>

Signed-off-by: Rishikesh1159 <[email protected]>
@yyyogev
Set analyzer to regex query string search (opensearch-project#3967)
ea4cfcc 
Sets analyzer to regex query string search

Signed-off-by: yyyogev <[email protected]>
@nknize
[Upgrade] to lucene-9.4.0-snapshot-ddf0d0a (opensearch-project#4183)
411321e 
Upgrades to lucene-9.4.0-snapshot-ddf0d0a by refactoring to TermOrdValComparator 
as a top level class.

Signed-off-by: Nicholas Walter Knize <[email protected]>
@pgtgrly
Adding documentation about creation of uber-JARs (opensearch-project#…
a081f2f 
…3785)

* Adding documentation about creation of uber-JARs

Signed-off-by: Pranav Garg <[email protected]>

* Fixing linelint error

Signed-off-by: Pranav Garg <[email protected]>

* Comprehensive changes

Signed-off-by: Pranav Garg <[email protected]>

* Adding PR changes

Signed-off-by: Pranav Garg <[email protected]>

* PR changes

Signed-off-by: Pranav Garg <[email protected]>

Signed-off-by: Pranav Garg <[email protected]>
Co-authored-by: Pranav Garg <[email protected]>
@mch2
Segment Replication - Remove unnecessary call to markAllocationIdAsIn…
280b938 
…Sync. (opensearch-project#4224)

This PR Removes an unnecessary call to markAllocationIdAsInSync on the primary shard when replication events complete.
Recovery will manage this initial call.

Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>
@nknize
[MUTE] search with uppercase regex in bwc testing (opensearch-project…
1e47b95 
…#4226)

Prevents "search with uppercase regex" in 190_index_prefix_search.yml from
running in bwc testing until feature is backported.

Signed-off-by: Nicholas Walter Knize <[email protected]>
@Bukhtawar
Introduce TranslogFactory for Local/Remote Translog support (opensear…
551f7c3 
…ch-project#4172)

* Introduce TranslogFactory for Local/Remote Translog support

Signed-off-by: Bukhtawar Khan <[email protected]>
@bharath-techie
Updated changes to security
a625607
@opensearch-trigger-bot @opensearch-ci-bot
Added bwc version 2.2.1 (opensearch-project#4194)
e17af2a 
Co-authored-by: opensearch-ci-bot <[email protected]>
@dreamer-89
[Segment Replication] Wait for documents to replicate to replica shar…
237f1a5 
…ds (opensearch-project#4236)

* [Segment Replication] Add thread sleep to account for replica lag in delete operations test

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments, assertBusy on doc count rather than sleep

Signed-off-by: Suraj Singh <[email protected]>

Signed-off-by: Suraj Singh <[email protected]>
@mch2
Segment Replication - Add additional unit tests for update & delete o…
3a97d4c 
…ps. (opensearch-project#4237)

* Segment Replication - Add additional unit tests for update & delete operations.

Signed-off-by: Marc Handalian <[email protected]>

* Fix spotless.

Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>
@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: a625607

@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: 78984d1

@mch2
Support shard promotion with Segment Replication. (opensearch-project…
f65e02d 
…#4135)

* Support shard promotion with Segment Replication.

This change adds basic failover support with segment replication.  Once selected, a replica will commit and reopen a writeable engine.

Signed-off-by: Marc Handalian <[email protected]>

* Add check to ensure a closed shard does not publish checkpoints.

Signed-off-by: Marc Handalian <[email protected]>

* Clean up in SegmentReplicationIT.

Signed-off-by: Marc Handalian <[email protected]>

* PR feedback.

Signed-off-by: Marc Handalian <[email protected]>

* Fix merge conflict.

Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>
@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: f10730dbcf969ad1b7ff89285882bec7d736113c

kartg and others added 4 commits August 17, 2022 14:27
@kartg
Added timing data and more granular stages to SegmentReplicationState (…
a2ba3a8 
…opensearch-project#4222)

* Added timing data and more granular stages to SegmentReplicationState

This change introduces instrumentation logging that measures the latency of the various stages of segment replication as seen by each replica. Logs have also been added to the source node for checkpoint publishing and checkpoint metadata responses. All logging is currently at the TRACE level.

Signed-off-by: Kartik Ganesh <[email protected]>

* Fixing SegmentReplicationTarget tests

Signed-off-by: Kartik Ganesh <[email protected]>

* Incorporated PR feedback

Signed-off-by: Kartik Ganesh <[email protected]>

* Fixing SegmentReplicationTargetService tests

Signed-off-by: Kartik Ganesh <[email protected]>

Signed-off-by: Kartik Ganesh <[email protected]>
@owaiskazi19
Handles the status code for . properties (opensearch-project#4246)
36f1d77 
* Return 400 status code for array out of bound

Signed-off-by: Owais Kazi <[email protected]>

* Spotless apply

Signed-off-by: Owais Kazi <[email protected]>

* PR comments

Signed-off-by: Owais Kazi <[email protected]>

Signed-off-by: Owais Kazi <[email protected]>
@dreamer-89
[Segment Replication] Update PrimaryShardAllocator to prefer replicas…
d308a29 
… with higher replication checkpoint (opensearch-project#4041)

* [Segment Replication] Update PrimaryShardAllocator to prefer replicas having higher replication checkpoint

Signed-off-by: Suraj Singh <[email protected]>

* Use empty replication checkpoint to avoid NPE

Signed-off-by: Suraj Singh <[email protected]>

* Update NodeGatewayStartedShards to optionally wire in/out ReplicationCheckpoint field

Signed-off-by: Suraj Singh <[email protected]>

* Use default replication checkpoint causing EOF errors on empty checkpoint

* Add indexSettings to GatewayAllocator to allow ReplicationCheckpoint comparator only for segrep enabled indices

* Add unit tests for primary term first replica promotion & comparator fix

* Fix NPE on empty IndexMetadata

* Remove settings from AllocationService and directly inject in GatewayAllocator

* Add more unit tests and minor code clean up

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments & integration test

Signed-off-by: Suraj Singh <[email protected]>

* Fix comparator on null ReplicationCheckpoint

Signed-off-by: Suraj Singh <[email protected]>

Signed-off-by: Suraj Singh <[email protected]>
@bharath-techie
addressing comments
96c8b3e 
Signed-off-by: Bharathwaj G <[email protected]>
@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: 96c8b3e

reta and others added 5 commits August 18, 2022 15:51
@reta
Fixing NodeGatewayStartedShards bwc (de)serialization issues (opensea…
ee26e01 
…rch-project#4258)

Signed-off-by: Andriy Redko <[email protected]>

Signed-off-by: Andriy Redko <[email protected]>
@dbwiddis
Fix AbstractStringFieldDataTestCase tests to account for TotalHits lo…
4643620 
…wer bound (opensearch-project#4270)

Fixes tests to account for TotalHits uncertainty as of Lucene 9.

Signed-off-by: Daniel Widdis <[email protected]>
@dependabot
Bump com.gradle.enterprise from 3.10.3 to 3.11.1 (opensearch-project#…
6629b09 
…4273)

Bumps com.gradle.enterprise from 3.10.3 to 3.11.1.

---
updated-dependencies:
- dependency-name: com.gradle.enterprise
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@navneet1v
Refactored the src and test of GeoHashGrid and GeoTileGrid Aggregatio…
5433056 
…ns on GeoPoint from server folder to geo module.(opensearch-project#4071) (opensearch-project#4072) (opensearch-project#4180)

The changes also includes:
   * Updated Search plugin to provide the interface so that plugins can also register the composite aggregations
   * Added YAML test for the geo_grid, geo_tile and composite aggregation

Signed-off-by: Navneet Verma <[email protected]>
@bharath-techie
Merge branch 'main' of https://github.com/opensearch-project/OpenSearch…
3c8116c 
… into pitsegmentsrest1
@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: 3c8116c

@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: 00daa5f66b5b9cea70f8652bdb39f88379b9884f

@bharath-techie
security changes
fa5940f 
Signed-off-by: Bharathwaj G <[email protected]>
@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: fa5940f

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.