[BUG] Generic HTTP Actions in Java Client does not work with AwsSdk2Transport

[anubhav3008]

What is the bug?

This bug is in feature: #377 where the generic Http actions are enabled. While the generic Http actions work fine on local open search instances, they give 403 when working with AWS open search.

One reason can be, as I see is that in the PR: https://github.com/opensearch-project/opensearch-java/pull/910/files , the changes are done to RestClientTransport.java and ApacheHttpClient5Transport.java , but the changes to AwsSdk2Transport is not done.
cc: @reta

How can one reproduce the bug?

Try calling the AWS open search. The below opensearch client gives 200 when working with non generic flow, but fails for generic flow.

below is sample code:

private String searchWithGenericClient(OpenSearchClient openSearchClient) throws IOException {
        Response response = openSearchClient.generic()
                .execute(
                        Requests.builder()
                                .endpoint("/" + "notifications" + "/_search?typed_keys=true")
                                .method("GET")
                                .json("{"
                                        + "    \"query\": {"
                                        + "        \"match_all\": {}"
                                        + "    }"
                                        + "}"
                                )
                                .build()
                );
            return response.getBody().get().bodyAsString();
    }

open search version: 2.10.1

 <dependency>
            <groupId>org.opensearch.client</groupId>
            <artifactId>opensearch-java</artifactId>
            <version>2.10.1</version>
        </dependency>

Output:

Exception in thread "main" org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [security_exception] 403 Forbidden
	at org.opensearch.client.transport.aws.AwsSdk2Transport.parseResponse(AwsSdk2Transport.java:477)
	at org.opensearch.client.transport.aws.AwsSdk2Transport.executeSync(AwsSdk2Transport.java:396)
	at org.opensearch.client.transport.aws.AwsSdk2Transport.performRequest(AwsSdk2Transport.java:193)
	at org.opensearch.client.opensearch.generic.OpenSearchGenericClient.execute(OpenSearchGenericClient.java:168)
	at com.anubhav.dao.OpenSearchDao.searchWithGenericClient(OpenSearchDao.java:52)
	at com.anubhav.dao.OpenSearchDao.main(OpenSearchDao.java:37)

What is the expected behavior?

We need the generic flow to work for AWS opensearch as well. We are in the design stage of our product where, we need to decide generic flow vs object based flow. We want to use generic flow, but due to this error, we are stuck.

What is your host/environment?

mac os 14.4.1 (23E224)

Do you have any screenshots?

No screenshot. Only code snippet already attached.

Do you have any additional context?

Looks like the change is not done for all flows for #377 and PR: https://github.com/opensearch-project/opensearch-java/pull/910/files

[anubhav3008]

Thanks @reta for assigning to yourself.

Complete code:

        software.amazon.awssdk.http.SdkHttpClient httpClient = ApacheHttpClient.builder().build();
        org.opensearch.client.opensearch.OpenSearchClient client = new OpenSearchClient(
                new AwsSdk2Transport(
                        httpClient,
                        // replace with endpoint in aws opensearch
                        HttpHost.create("https://xxxxxxxxxx.us-east-1.aoss.amazonaws.com").getHostName(),
                        "aoss",
                        Region.of("us-east-1"),
                        AwsSdk2TransportOptions.builder().build()));

        Response response = client.generic()
                .execute(
                        Requests.builder()
                                .endpoint("/" + "notifications" + "/_search?typed_keys=true")
                                .method("GET")
                                .json("{"
                                        + "    \"query\": {"
                                        + "        \"match_all\": {}"
                                        + "    }"
                                        + "}"
                                )
                                .build()
                );
        System.out.println(response.getBody().get().bodyAsString());

Setup:

Use AWS open search server-less: https://aws.amazon.com/opensearch-service/features/serverless/

[reta]

@anubhav3008 do you wanna take it? thank you! 🙏

[anubhav3008]

@anubhav3008 do you wanna take it? thank you! 🙏
@reta , I can take it up. But i not sure how to fix it. I am trying to understand the code flow as of now.

[BrendonFaleiro]

Which version would this fix be in?

[dblock]

@BrendonFaleiro https://github.com/opensearch-project/opensearch-java/blob/main/CHANGELOG.md, 2.10.3. If you want a release quickly open an issue "Release v. Next" and we can do it soon.