Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert 422 backport/backport 413 2.17 #430

Closed
wants to merge 121 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
121 commits
Select commit Hold shift + click to select a range
ea5d4f0
Merge pull request #13 from zane-neo/2.x
zane-neo Dec 14, 2023
ec1bd44
Include ML zip dependency in integTest and run tasks (#36)
ohltyler Dec 18, 2023
d588b78
add security policy for accessDeclaredMembers (#35) (#37)
opensearch-trigger-bot[bot] Dec 18, 2023
ff17f3d
fix version
zhichao-aws Dec 21, 2023
dc6180a
Merge pull request #42 from zhichao-aws/fix_2.x
dbwiddis Dec 21, 2023
c931029
[Backport 2.x] Add basic search detectors tool; pull plugin deps in g…
ohltyler Dec 22, 2023
c6e18c6
[Backport 2.x] feature: Add AbstractRetriverTool, VectorDBTool, Neura…
zhichao-aws Dec 25, 2023
e75e082
add ut and parser (#64)
xinyual Dec 26, 2023
0ccb019
Add a search AD results tool (#52) (#67)
opensearch-trigger-bot[bot] Dec 27, 2023
8f746aa
increase AbstractRetrieverToolTests code coverage (#65)
mingshl Dec 27, 2023
8aadce4
[Backport 2.x] Fix backport branch name is not correct (#71)
opensearch-trigger-bot[bot] Dec 27, 2023
ebd330f
change plugin name add opensearch prefix (#72) (#73)
opensearch-trigger-bot[bot] Dec 27, 2023
cd54a22
Change job-scheduler plugin group to point to correct location (#56) …
opensearch-trigger-bot[bot] Dec 27, 2023
b0db422
Add visualization tool (#41) (#68)
opensearch-trigger-bot[bot] Dec 27, 2023
0d0c1c1
Migrate alerting tools (#66) (#76)
opensearch-trigger-bot[bot] Dec 28, 2023
813c55e
Add RAGTool (#78) (#79)
opensearch-trigger-bot[bot] Dec 29, 2023
d027d60
feat: search index tool (#61) (#77)
opensearch-trigger-bot[bot] Dec 29, 2023
2c86116
Add tools to ToolPlugin (#81) (#82)
opensearch-trigger-bot[bot] Dec 29, 2023
64c3804
Fix IT run error by adding job-scheduler back to zipArchive dependenc…
zane-neo Dec 29, 2023
1de3b93
[Backport 2.x] Base class for Integ test; add integ test for NeuralSp…
opensearch-trigger-bot[bot] Jan 2, 2024
3e73b7b
fix: some parsing exception from search index tool. (#93) (#95)
opensearch-trigger-bot[bot] Jan 2, 2024
acc1d32
[Backport 2.x] Integrate search detectors tool with profile API (#88)…
ohltyler Jan 3, 2024
156f54c
provide PPL for querying system indices (#98) (#99)
opensearch-trigger-bot[bot] Jan 3, 2024
d69b45a
ignore alias field (#101) (#103)
opensearch-trigger-bot[bot] Jan 3, 2024
39e4361
Mock http server for LLM; Integration test for visualization tool (#9…
opensearch-trigger-bot[bot] Jan 4, 2024
0fc9f19
copy zip to output folder (#89) (#105)
opensearch-trigger-bot[bot] Jan 4, 2024
f5e15ac
Add ppl tool it (#104) (#106)
opensearch-trigger-bot[bot] Jan 5, 2024
c73fc7a
fix: update search index tool parse logic (#100) (#107)
opensearch-trigger-bot[bot] Jan 5, 2024
943650e
test: integration test for search index tool (#108) (#109)
opensearch-trigger-bot[bot] Jan 5, 2024
415c9e2
Set default index pattern for search AD results tool (#111) (#112)
opensearch-trigger-bot[bot] Jan 8, 2024
6656a51
[Backport 2.x] Various AD tool improvements; add corresponding IT (#…
ohltyler Jan 10, 2024
04693a6
fix compile error (#114) (#115)
opensearch-trigger-bot[bot] Jan 12, 2024
bd72f5b
feature: tune description on monitor and anomaly detection (#120) (#126)
opensearch-trigger-bot[bot] Jan 12, 2024
6860ae9
Add default prompt to ppl tool (#125) (#129)
opensearch-trigger-bot[bot] Jan 16, 2024
097d11c
fix prompt (#133) (#134)
opensearch-trigger-bot[bot] Jan 18, 2024
b24a621
Add alerting tools IT; fix missing system index bug of SearchMonitors…
opensearch-trigger-bot[bot] Jan 23, 2024
952cdd7
fix name bug (#139) (#142)
opensearch-trigger-bot[bot] Jan 23, 2024
9f28f4c
[Backport 2.x] Include more return values for SearchAnomalyDetectorsT…
ohltyler Jan 24, 2024
18e31ae
Change default anomaly grade to nonzero (#148) (#149)
opensearch-trigger-bot[bot] Jan 25, 2024
bafff96
add execute field (#146) (#147)
opensearch-trigger-bot[bot] Jan 26, 2024
1194d06
add description for VectorDBTool and remove json parsig for RAGTool (…
opensearch-trigger-bot[bot] Jan 26, 2024
361e0a1
Enhance RagTool to choose neural sparse query type (#140) (#152)
opensearch-trigger-bot[bot] Jan 26, 2024
2bd56ed
Fix SearchMonitorsTool bugs; add corresponding ITs (#151) (#153)
opensearch-trigger-bot[bot] Jan 27, 2024
53932eb
Tune descriptions; improve monitor type (#154) (#155)
opensearch-trigger-bot[bot] Jan 29, 2024
7ac1fb5
Fix boolean parse (#157) (#158)
opensearch-trigger-bot[bot] Jan 30, 2024
4d88e09
Onboarding skills (#163)
zane-neo Jan 31, 2024
55ef7be
Fix the flaky test due to m_l_limit_exceeded_exception (#150) (#164)
opensearch-trigger-bot[bot] Jan 31, 2024
eb70c18
Change 2.x maven-publish repo name to be skills and sync with main br…
peterzhuamazon Jan 31, 2024
bc56d78
Improve IT coverage of all AD & Alerting tools (#165) (#167)
opensearch-trigger-bot[bot] Jan 31, 2024
e3d6711
fix UT bug (#172) (#175)
opensearch-trigger-bot[bot] Feb 2, 2024
41606a8
Add agent framework configuration to make IT pass (#179) (#180)
opensearch-trigger-bot[bot] Feb 4, 2024
005b4af
fixPPLAllowedFields (#181) (#182)
opensearch-trigger-bot[bot] Feb 4, 2024
8cb2318
backport from #131 (#183)
xinyual Feb 5, 2024
b8e1802
trim index name in parameters for PPL tool (#185) (#186)
opensearch-trigger-bot[bot] Feb 6, 2024
b40255e
Fix alert constructor (#191) (#193)
opensearch-trigger-bot[bot] Feb 7, 2024
1a514c6
Increase wait time to avoid flaky test (#173) (#176)
opensearch-trigger-bot[bot] Feb 7, 2024
d6f6cde
(PPLTool) update claude and openai prompts (#184) (#192)
opensearch-trigger-bot[bot] Feb 7, 2024
d87cd09
Fix failure ITs caused by ml-common error handling change (#195) (#196)
opensearch-trigger-bot[bot] Feb 7, 2024
1ee884e
Update mockito monorepo to v5.10.0 (#128) (#197)
opensearch-trigger-bot[bot] Feb 7, 2024
69843eb
Fixed POM to include inception year (#190) (#200)
opensearch-trigger-bot[bot] Feb 7, 2024
59204b7
Fix json parsing exception for NeuralSparseSearchTool and VectorDBToo…
opensearch-trigger-bot[bot] Feb 7, 2024
e5c9c73
Fixed build script to run assemble (#207) (#208)
opensearch-trigger-bot[bot] Feb 8, 2024
4dfdf9a
add truncate For PPL Tool (#206) (#211)
opensearch-trigger-bot[bot] Feb 8, 2024
ddf8523
Added release notes for 2.12 (#213) (#214)
opensearch-trigger-bot[bot] Feb 9, 2024
69ccd7f
Fixed build (#215)
owaiskazi19 Feb 9, 2024
eb2fa17
Excludes integration test classes that attempt to delete system indic…
joshpalis Feb 9, 2024
706fa85
Onboard search AD and search monitor tools to security IT (#225) (#226)
opensearch-trigger-bot[bot] Feb 13, 2024
d1877c9
fix VisualizationsToolIT (#229) (#230)
opensearch-trigger-bot[bot] Feb 15, 2024
0d4bdd9
Fix SearchAnomalyDetectorsTool indices param bug; add more IT (#233) …
opensearch-trigger-bot[bot] Feb 20, 2024
f3f2317
[Backport 2.x] Fix detector state params in SearchAnomalyDetectorsToo…
ohltyler Feb 26, 2024
7889975
Updated release notes file name (#241) (#242)
opensearch-trigger-bot[bot] Feb 28, 2024
c506f24
Update dependency org.apache.commons:commons-lang3 to v3.14.0 (#47) (…
opensearch-trigger-bot[bot] Mar 1, 2024
cfb419f
Add lebeler.yml to 2.x branch (#247)
zane-neo Mar 1, 2024
df9820e
Update dependency org.apache.commons:commons-text to v1.11.0 (#62) (#…
opensearch-trigger-bot[bot] Mar 1, 2024
cfd0ede
Update plugin io.freefair.lombok to v8.6 (#245) (#249)
opensearch-trigger-bot[bot] Mar 1, 2024
e9b46ff
Update plugin de.undercouch.download to v5.6.0 (#239) (#250)
opensearch-trigger-bot[bot] Mar 1, 2024
bed81dd
Update plugin com.diffplug.spotless to v6.25.0 (#127) (#252)
opensearch-trigger-bot[bot] Mar 1, 2024
9f4e831
Update dependency org.json:json to v20240205 (#246) (#251)
opensearch-trigger-bot[bot] Mar 1, 2024
63f0e5c
Add IT for VectorDBTool and NeuralSparseTool (#177) (#255)
opensearch-trigger-bot[bot] Mar 6, 2024
5088170
update ppl tool claude prompts to use `<ppl>` tags (#260) (#261)
opensearch-trigger-bot[bot] Mar 16, 2024
64c5e2e
add parameter validate for PPL tool (#259) (#262)
opensearch-trigger-bot[bot] Mar 18, 2024
357be37
change release date (#267)
zane-neo Mar 21, 2024
9f1bd35
update 2.13 release note add dependency (#269) (#270)
opensearch-trigger-bot[bot] Mar 21, 2024
76dcbcb
fix filter fields, adding geo point and date_nanos (#285) (#286)
opensearch-trigger-bot[bot] Apr 18, 2024
de2e2d7
Increment version to 2.13; change ad plugin jar dependency; increment…
zane-neo Apr 18, 2024
eeb0d89
remove logic about replace quota for finetuning model (#289) (#291)
opensearch-trigger-bot[bot] Apr 18, 2024
1ef9502
[Backport 2.x] feat: remove search index tool (#297)
yuye-aws Apr 26, 2024
0f6c1bc
Move visualization tool to ml-commons (#296) (#298)
opensearch-trigger-bot[bot] Apr 29, 2024
fc352a9
Add 2.14 release notes (#299)
zane-neo Apr 30, 2024
1ddf2cd
add error message for NPE (#301) (#304)
opensearch-trigger-bot[bot] Apr 30, 2024
37beb3b
Increment version to 2.14.0-SNAPSHOT
opensearch-ci-bot Apr 30, 2024
8ee946b
Change AD jar name for 2.14 compatibility (#308)
dbwiddis May 1, 2024
e2da02d
Increment version to 2.15.0-SNAPSHOT (#310)
opensearch-trigger-bot[bot] Jun 6, 2024
bb32961
Fix build error due to upstream change (#316) (#317)
Hailong-am Jun 6, 2024
a544e37
add release note (#320)
zane-neo Jun 11, 2024
9e572a8
Fix it failure (#321) (#322)
opensearch-trigger-bot[bot] Jun 12, 2024
51e58c5
Fix search alert IT failure (#329)
zane-neo Jun 14, 2024
30bd989
Fix flaky ITs (#332)
zane-neo Jun 14, 2024
4039bf7
change log level to error (#339)
zane-neo Jun 21, 2024
08f41b8
fix compilation error and change gradle version to compatible with jd…
zane-neo Jul 4, 2024
5340662
backport (#352)
zhichao-aws Jul 16, 2024
858e1e3
[Feature] support nested query in neural sparse tool, vectorDB tool a…
opensearch-trigger-bot[bot] Jul 17, 2024
32382ab
Add cluster setting to control ppl execution (#344) (#356)
opensearch-trigger-bot[bot] Jul 17, 2024
d5af2e1
[Backport 2.x] Add CreateAnomalyDetectorTool (#355)
gaobinlong Jul 18, 2024
dc7bd7a
Increment version to 2.16.0-SNAPSHOT (#324)
opensearch-trigger-bot[bot] Jul 19, 2024
4c32da7
fix(deps): update dependency lombok to v1.18.34 (#266) (#365)
opensearch-trigger-bot[bot] Jul 25, 2024
c0d3aff
chore(deps): update dependency gradle to v8.9 (#275) (#366)
zane-neo Jul 25, 2024
0fbce60
Add 2.16 release note (#373)
zane-neo Jul 29, 2024
c52dbea
Add includeFields parameter to the method extractFieldNamesTypes (#37…
opensearch-trigger-bot[bot] Aug 2, 2024
fc9ae93
[Backport] Remove ppl tool execution setting (#383)
zane-neo Aug 3, 2024
fc23773
Fix 2.17 compilation error since AD change (#388)
zane-neo Aug 28, 2024
10b7f8b
backport #256 to 2.x (#367)
zane-neo Aug 28, 2024
69aa50e
backport #258 to 2.x (#370)
zane-neo Aug 28, 2024
3cbb1f4
Upgrade apache common lang version to 3.16 (#390)
zane-neo Aug 28, 2024
6b399d0
chore(deps): update dependency gradle to v8.10 (#389) (#392)
opensearch-trigger-bot[bot] Aug 29, 2024
76d65e3
chore(deps): update plugin io.freefair.lombok to v8.10 (#393) (#394)
opensearch-trigger-bot[bot] Aug 29, 2024
5e0fdf5
Add 2.17 release note (#401) (#402)
opensearch-trigger-bot[bot] Sep 7, 2024
09dc7da
Fix 2.17 test compilation issue (#406) (#407)
opensearch-trigger-bot[bot] Sep 10, 2024
e526ede
Increment version to 2.17.1-SNAPSHOT (#409)
opensearch-trigger-bot[bot] Sep 24, 2024
7cd2c22
Add LogPatternTool (#413) (#418) (#422)
qianheng-aws Oct 10, 2024
ec87ad6
Revert "Add LogPatternTool (#413) (#418) (#422)"
xinyual Oct 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion .github/workflows/backport.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,17 @@ on:
jobs:
backport:
runs-on: ubuntu-latest
# Only react to merged PRs for security reasons.
# See https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target.
if: >
github.event.pull_request.merged
&& (
github.event.action == 'closed'
|| (
github.event.action == 'labeled'
&& contains(github.event.label.name, 'backport')
)
)
permissions:
contents: write
pull-requests: write
Expand All @@ -26,6 +37,6 @@ jobs:
uses: VachaShah/[email protected]
with:
github_token: ${{ steps.github_app_token.outputs.token }}
branch_name: backport/backport-${{ github.event.number }}
head_template: backport/backport-<%= number %>-to-<%= base %>
labels_template: "<%= JSON.stringify([...labels, 'autocut']) %>"
failure_labels: "failed backport"
65 changes: 52 additions & 13 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,36 @@ on:

jobs:
Get-CI-Image-Tag:
uses: opensearch-project/opensearch-build/.github/workflows/get-ci-image-tag.yml@main
with:
product: opensearch
runs-on: ubuntu-latest
outputs:
ci-image-version-linux: ${{ steps.step-ci-image-version-linux.outputs.ci-image-version-linux }}
steps:
- name: Install crane
uses: iarekylew00t/crane-installer@v1
with:
crane-release: v0.15.2
- name: Checkout opensearch-build repository
uses: actions/checkout@v2
with:
repository: 'opensearch-project/opensearch-build'
ref: 'main'
path: 'opensearch-build'
- name: Get ci image version from opensearch-build repository scripts
id: step-ci-image-version-linux
run: |
crane version
CI_IMAGE_VERSION=`opensearch-build/docker/ci/get-ci-images.sh -p centos7 -u opensearch -t build | head -1`
echo $CI_IMAGE_VERSION
echo "ci-image-version-linux=$CI_IMAGE_VERSION" >> $GITHUB_OUTPUT

build-linux:
needs: Get-CI-Image-Tag
strategy:
matrix:
java:
- 11
- 17
- 21
name: Build and Test
java: [11, 17, 21]
env:
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
name: Build and Test skills plugin on Linux
runs-on: ubuntu-latest
container:
# using the same image which is used by opensearch-build team to build the OpenSearch Distribution
Expand Down Expand Up @@ -50,14 +67,36 @@ jobs:
with:
token: ${{ secrets.CODECOV_TOKEN }}

build-MacOS:
strategy:
matrix:
java: [11, 17, 21]

name: Build and Test skills Plugin on MacOS
needs: Get-CI-Image-Tag
runs-on: macos-latest

steps:
- name: Checkout skills
uses: actions/checkout@v1
- name: Setup Java ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}
- name: Install dependencies on macos
run: |
brew reinstall gcc
export FC=/usr/local/Cellar/gcc/12.2.0/bin/gfortran
- name: Run build
run: |
./gradlew build -Dos.arch=x86_64

build-windows:
strategy:
matrix:
java:
- 11
- 17
- 21
name: Build and Test
java: [11, 17, 21]
name: Build and Test skills plugin on Windows
needs: Get-CI-Image-Tag
runs-on: windows-latest

steps:
Expand Down
26 changes: 26 additions & 0 deletions .github/workflows/labeler.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: "Pull Request Labeler"
on:
pull_request_target:
branches:
- main
types:
- opened

jobs:
label:
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
steps:
- name: GitHub App token
id: github_app_token
uses: tibdex/[email protected]
with:
app_id: ${{ secrets.APP_ID }}
private_key: ${{ secrets.APP_PRIVATE_KEY }}
installation_id: 22958780
- name: Label
uses: actions/labeler@v5
with:
repo-token: ${{ steps.github_app_token.outputs.token }}
8 changes: 4 additions & 4 deletions .github/workflows/maven-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
build-and-publish-snapshots:
strategy:
fail-fast: false
if: github.repository == 'opensearch-project/agent-tools'
if: github.repository == 'opensearch-project/skills'
runs-on: ubuntu-latest

permissions:
Expand All @@ -22,9 +22,9 @@ jobs:
- uses: actions/setup-java@v3
with:
distribution: temurin # Temurin is a distribution of adoptium
java-version: 17
java-version: 21
- uses: actions/checkout@v3
- uses: aws-actions/configure-aws-credentials@v1
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.PUBLISH_SNAPSHOTS_ROLE }}
aws-region: us-east-1
Expand All @@ -34,4 +34,4 @@ jobs:
export SONATYPE_PASSWORD=$(aws secretsmanager get-secret-value --secret-id maven-snapshots-password --query SecretString --output text)
echo "::add-mask::$SONATYPE_USERNAME"
echo "::add-mask::$SONATYPE_PASSWORD"
./gradlew publishShadowPublicationToSnapshotsRepository
./gradlew publishPluginZipPublicationToSnapshotsRepository
44 changes: 44 additions & 0 deletions .github/workflows/test_security.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: Run Security tests
on:
push:
branches-ignore:
- 'whitesource-remediate/**'
- 'backport/**'
pull_request:
types: [opened, synchronize, reopened]

jobs:
Get-CI-Image-Tag:
uses: opensearch-project/opensearch-build/.github/workflows/get-ci-image-tag.yml@main
with:
product: opensearch

integ-test-with-security-linux:
strategy:
matrix:
java: [11, 17, 21]
env:
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
name: Run Security Integration Tests on Linux
runs-on: ubuntu-latest
needs: Get-CI-Image-Tag
container:
# using the same image which is used by opensearch-build team to build the OpenSearch Distribution
# this image tag is subject to change as more dependencies and updates will arrive over time
image: ${{ needs.Get-CI-Image-Tag.outputs.ci-image-version-linux }}
# need to switch to root so that github actions can install runner binary on container without permission issues.
options: --user root

steps:
- name: Checkout Skills
uses: actions/checkout@v3
- name: Setup Java ${{ matrix.java }}
uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: ${{ matrix.java }}
- name: Run tests
# switching the user, as OpenSearch cluster can only be started as root/Administrator on linux-deb/linux-rpm/windows-zip.
run: |
chown -R 1000:1000 `pwd`
su `id -un 1000` -c "whoami && java -version && ./gradlew integTest -Dsecurity.enabled=true"
Loading
Loading