download performance comparision about s3crtclient and transfermanager

[zzyyllrr]

https://github.com/aws/aws-sdk-cpp/wiki/Improving-S3-Throughput-with-AWS-SDK-for-CPP-v1.9
In this article, s3crtclient achieves better performance than s3client. Is there any comparision about s3crtclient and transfermanager?
I tested downloading 1000 files sequentially and each file is 20MB, the result shows transfermanager costs half time of s3crtclient. During the test, transfermanager reaches 1Gbps bandwidth while s3crtclient reaches 600Mbps at max.

[jmklix]

We don't have any direct comparison between s3crtclient and transfermanager. It can depend highly on how you have each configured. You can increase the s3crtclient bandwidth to something more than the default. It defaults to 2Gbps max, so increasing this could possibly improve the async speeds (For this sdk Gbps means Gigabits per second, and not Gigabytes per second). You can set this with throughputTargetGbps:

auto config = Aws::S3Crt::ClientConfiguration();
config.throughputTargetGbps = targetgbps;

Currently the transfer manager uses the s3 client under the hood, but we have plans in the future to allow customers to choose if they want to use s3 or s3crt. This external PR makes some of the necessary changes, but there are still a few other changes needed before we merge this into main.

[zzyyllrr]

Thansk.
I have already set throughputTargetGbps to 10Gbps，and the speed of NIC is 25000Mb/s in the test.
From the debug log, I do see the downloading seperated into multiple Range request just like tansfermanager. So What is difference between s3crtclient and transfermanager(s3client based) in processing the download?

[jmklix]

The main difference is that transfermanager provides some extra utility functions (e.i. UploadDirectory, upload status, etc.). The s3crt was written as a replacement for transfermanager. It was designed for high throughput cases but it is not strictly better than transfermanager for all use cases. A big difference in the backend is that s3crt is written in c while s3 (and transfermanager) is written in c++.

[zzyyllrr]

ok.
Another question, I use s3crtclient's getobjectasync API in my project in multiple thread, but the GetObjectResponseReceivedHandler always called in the same thread(one of the aws_event_loop_thread). I see 30+ aws_event_loop_thread in gdb. I want to know if any configuration i can set to Trigger the handler evenly on those aws_event_loop_thread?

[jmklix]

The GetObjectResponseReceivedHandler will always deliver the callback to one single thread, but the actually getting of different objects will take place across those 30+ threads. There is no way to configure the handler to run on multiple threads with this sdk.

[DmitriyMusatkin]

Also note: that crt ensures that all the events for GetObject are delivered sequentially. So events for a single GetObject will all be delivered on one thread. But in case where you are running several GetObjects in parallel, crt will spread out which threads will be used to deliver objects, so its very likely that callbacks will be made on different threads for different requests.