[Discussion] Modify input manfiest yml schema and assemble workflow to support pre-installation of native plugins #2849
Comments
joshpalis
added
enhancement
|
Thanks for writing this up @joshpalis. How about including a flag named - name: anomaly-detection
repository: https://github.com/opensearch-project/anomaly-detection.git
depends_on: ['Job-Scheduler']
ref: '2.4'
platforms:
- linux
- windows
checks:
- gradle:properties:version
- gradle:dependencies:opensearch.versionWhile reading the input manifest we have to take into account the dependency and based on the flag we can either install or not the JS native plugin. |
|
Hey @owaiskazi19 something similar to what proposed here ? |
|
Hey @prudhvigodithi. Looks like we are on the same line with the scheme here. For the above schema, I don't have any strong opinion(totally fine with it). Only just that it will create a little complex manifest file which can be simplified by using the flag |
|
Thanks @owaiskazi19 and @prudhvigodithi for your contributions for this discussion. I am partial to Owais' proposal for the modifications to the input manifest yml schema. The modifications would be simplified and it places the onus of responsibility for components to define the plugins that they are dependent on, making it clear which components are dependent on which native plugin. |
prudhvigodithi
added
feature
|
[Triage] Hey @joshpalis what is the targeted release with job-scheduler as native plugin, we should time this right to unblock 3.0.0 development. |
|
@prudhvigodithi Currently the targeted release is 2.5.0 |
|
@joshpalis I would recommend to target 2.6.0 release since this is technically a breaking change for the build and development process so we need to synchronize the effort between multiple teams. CC: @CEHENKLE @peterzhuamazon @minalsha |
|
Sure, I am fine with targeting 2.6.0 release. I have no strong opinions against this. @minalsha is this alright with you? |
|
@bbarani we are working closely with folks across different teams. I don't see why we need to push it out. Lets target for 2.5.0 and see where we land on 01/02/2023. cc @CEHENKLE @prudhvigodithi @joshpalis @dagneyb |
|
@minalsha Is there a need to rush this change soon? This is not a new feature for users rather more of a change to existing plugin installation process. This change needs to be synchronized across multiple versions (3.x and 2.x). The available resources are currently working on 1.3.7 release along with Windows support hence I would not prioritize this for 2.5.0 release. @CEHENKLE @prudhvigodithi @joshpalis @dagneyb @peterzhuamazon @gaiksaya @saratvemulapalli Will this change be done only for 2.x and 3.x version. I assume 1.x versions are not affected by this change. Can you please confirm? |
@bbarani you are right. This change will only go out for 2.x and above. This will not impact any 1.x releases. Job Scheduler will still be supported with the same maven co-ordinates for 1.x. |
|
@bbarani @prudhvigodithi The relocation of Job Scheduler to native plugins has been determined to be a breaking change due to the necessary Moving forward, modifications to the input manifest yml schema will still be necessary. Support for the pre-installation of native plugins will only be needed for OpenSearch 3.x and onwards. |
|
[Proposal] Posting one of the approach here: In order to add JS as a native plugin, a new schema needs to be introduced. OpenSearch consist of a number of native plugins. In this case, we choose to install just one as a part of the distribution which is job scheduler (maybe more in future). Hence the manifest needs to document it. One way is to have a
Each type will define how the component is installed. The functionality to install components that are of type core-plugin needs to be added to the build system.
From offline discussion looks like the move will be happen from 2.x version but enforced from 3.x due to breaking changes. The new input manifest can look like below: ---
schema-version: '1.1'
build:
name: OpenSearch
version: 3.0.0
ci:
image:
name: opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v2
args: -e JAVA_HOME=/opt/java/openjdk-17
components:
- name: OpenSearch
repository: https://github.com/opensearch-project/OpenSearch.git
ref: main
type: min
checks:
- gradle:publish
- gradle:properties:version
- name: common-utils
repository: https://github.com/opensearch-project/common-utils.git
type: lib
ref: main
platforms:
- linux
- windows
checks:
- gradle:publish
- gradle:properties:version
- name: job-scheduler
repository: https://github.com/opensearch-project/job-scheduler.git
type: core-plugin
ref: main
platforms:
- linux
- windows
checks:
- gradle:properties:version
- gradle:dependencies:opensearch.version
- name: ml-commons
repository: https://github.com/opensearch-project/ml-commons.git
type: plugin
ref: main
platforms:
- linux
- windows
checks:
- gradle:properties:version
- gradle:dependencies:opensearch.version: opensearch-ml-plugin
@saratvemulapalli @joshpalis @owaiskazi19 @dblock @opensearch-project/engineering-effectiveness let me know what do you think about this. More approaches and recommendations are welcomed. |
|
Generally adding a type is a clutch, I don't recommend it. It forks a lot of classes and makes switching on type instead of switching on features.
|
|
@dblock Are you suggesting to integrate JS native plugin installation process outside of distribution build process? In that case, we will always assume that -min contains the job-scheduler (JS) pre-installed and install other plugins? @saratvemulapalli @minalsha @joshpalis can you add your inputs here? |
|
This manifest is used for build and eventually assemble the distribution. It changes with installation, the way they are installed. As long as we have right maven co-ordinates to install JS (as a native plugin), wouldn't that solve the problem? @gaiksaya The proposal also talks about order of execution which is a problem by itself. |
bbarani
moved this from Not started
to In Progress
in OpenSearch Engineering Effectiveness
bbarani
added
enhancement
|
Hey all just circling back to this issue, thanks for multiple proposals added, @joshpalis just to refresh can you please add the timeline? Like what would be the next step for 2.6? the best chosen proposal has to be pushed before 2.6? As far as I know now the JS plugin is being tested as both as native and as a regular plugin, hence no change needed from build side. Will this strategy be same for 2.6 as well? |
|
Hey @joshpalis just following up, can you please add your thoughts based on my previous message? |
|
The change is targeted for 3.0.0 release. To support, the build process has to be modified to ensure the job-scheduler is installed during assemble workflow. This can be done by installing the zip file from the core-plugins folder (code) and finally having the job-scheduler removed as a component from the manifest file. The native plugin zips (under core-plugins folder), should also be considered publishing to maven. This gives more options for a user to install the native plugin, currently it pulls form But this can be an enhancement later and will not impact the installation of JS as a native plugin during assemble as the assemble workflow use the build manifest (the output of the build workflow) and installs the plugins using the zips from the local workspace. Thank you |
|
Do we need to change the manifest at all? Would it be simpler to install whichever native plugins we want as part of |
|
Hey @dblock with the existing flow the install.sh is called during install_plugin that has the logic just to run some So looks to me like Tested with Inside the final I'm open for any other solutions. Thank you |
|
Another approach (first approach in previous comment) we can go with is having a gradle task from OpenSearch core repo that can handle the desired native plugin installation code, and the script install.sh can have this gradle task called which ensures the native plugins are installed. |
I am ok with this approach, tho the input manifest needs changes to know the type of the plugins, per @gaiksaya suggestions. |
This sounds like a good idea! |
|
Hey @joshpalis coming form your comment opensearch-project/OpenSearch#5310 (comment), is the JS native plugin migration targeted to 3.0.0 release? |
I support this approach, as it negates the need to modify the input manifest to add a type to differentiate plugins and native plugins. As @saratvemulapalli stated, they should be treated the same. |
|
Thanks Josh, in that case can you please coordinate from the core side to on board a task that can install the native plugins and then we can come back to the build side and update the install.sh file with that gradle task ? |
|
@prudhvigodithi will look in this issue and see if it can be integrated to Gradle build for 3.x version. |
|
This issue remains on hold as the proposal and implementation for migrating job-scheduler as native plugin is still under review, also this migration is a breaking change targeted to 3.0.0 release which is moved to Feb 18 2025 based on the release schedule opensearch-project/.github#186. Its also worth exploring using of core Thanks |
Is your feature request related to a problem? Please describe
Now that the Job Scheduler project ownership has been transferred to the OpenSearch Core Team, we and the community have made a decision to relocate Job Scheduler to native plugins [1]. This change will have an effect on how the full bundle of OpenSearch is assembled, since multiple plugins (ISM, AD, Reporting) have a dependency on Job Scheduler.
Currently, the input build manifest includes Job Scheduler as a component[2], and this will be removed upon relocation. However, unless there is a mechanism to detect and pre-install native plugin dependencies, assembly of the aforementioned dependent plugins will fail.
[1] opensearch-project/OpenSearch#4218
[2] https://github.com/opensearch-project/opensearch-build/blob/main/manifests/2.4.0/opensearch-2.4.0.yml#L26
Describe the solution you'd like
In order to support the assembly of the Job Scheduler dependent components, I propose modifications to the input manifest yml schema such that native plugins that other components depend on will also be included. Consequently, changes to the input manifest yml schema will necessitate changes to the bundle_OpenSearch assemble workflow, such that native plugins listed within the input manifest are pre-installed prior to the dependent components.
CC : @prudhvigodithi @peterzhuamazon Open to discussion on how best to handle these changes
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: