Check on the HTTP performance issue with Java 21 migration

[TharmiganK]

Description

A load test for the following scenario with 500 concurrent users seems to be hanging:

H1(client - jmeter) <-> H2(passthrough service - Ballerina) <-> H1(backend - netty echo server)

Version

Ballerina SwanLake Update 11

[TharmiganK]

Passthrough service:

import ballerina/http;
import ballerina/log;

configurable string epKeyPath = "./ballerinaKeystore.p12";
configurable string epTrustStorePath = "./ballerinaTruststore.p12";
configurable string epKeyPassword = "ballerina";

listener http:Listener securedEP = new (9090,
    secureSocket = {
        key: {
            path: epKeyPath,
            password: epKeyPassword
        }
    }
);

final http:Client nettyEP = check new ("https://localhost:8688",
    secureSocket = {
        cert: {
            path: epTrustStorePath,
            password: epKeyPassword
        },
        verifyHostName: false
    }
);

service /passthrough on securedEP {
    isolated resource function post .(http:Request clientRequest) returns http:Response {
        log:printInfo("request received");
        http:Response|http:ClientError response = nettyEP->forward("/service/EchoService", clientRequest);
        if (response is http:Response) {
            return response;
        } else {
            log:printError("Error at h1_h1_passthrough", 'error = response);
            http:Response res = new;
            res.statusCode = 500;
            res.setPayload(response.message());
            return res;
        }
    }
}

Backend - Docker command:

docker run -d -p 8688:8688 -e "SSL=true" ldclakmal/netty-echo-backend:latest

JMeter client - JMX file: https://gist.github.com/TharmiganK/af4ba00cc1771dc4db2271b6a763c7bd

[TharmiganK]

Thread dump during the hanging issue: (Generated from jcmd command since jstack does not show the virtual threads)
threaddump.txt
fastthreadreport.pdf

The issue seems to be with this synchronized block. It seems the synchronized blocks are not loom-friendly

[TharmiganK]

Fixed with this PR: ballerina-platform/module-ballerina-http#2285