New HTTP demo to generate a pre-signed URL to an S3 object file

[giuspen]

New HTTP demo to generate a pre-signed URL to an S3 object file.

The setup is identical to the demo HTTP S3 download and in fact I implemented this based on that demo / starting from a duplication of demos/http/http_demo_s3_download.

By submitting this pull request, I confirm that my contribution is made under the terms of the MIT license.

[kstribrnAmzn]

Thank you for this contribution! I will review your PR today and send this out to the team for further review.

[kstribrnAmzn]

Do you intend for this demo to only generate the pre-signed URL? Or would you also like it to download the item using the pre-signed URL?

[kstribrnAmzn]

There is a lot of duplicated code between your demo and the download demo (which you based it off of). Would you mind refactoring some of these functions to a common location like this?

[giuspen]

Thanks for your review @kstribrnAmzn , I will push new commits with the requested changes.
I didn't think about also downloading the object because there is already another demo that is doing exactly that - the http download from S3 multi threaded takes a presigned URL as an input

[kstribrnAmzn]

Thank you so much @giuspen! Looking forward to reviewing when you push the changes. Keeping pre-signed URL creation and download as separate demos works for me. It should be far simpler for a new user to understand.

[giuspen]

I addressed all the points of the review except for the refactoring to share code with the other demo, will do that in one of the next few days...

[giuspen]

@kstribrnAmzn I refactored most of the common code in a shared .h/.c please let me know your thoughts

[kstribrnAmzn]

I'll take a look later today. Thank you for taking the time to refactor this :)

[kstribrnAmzn]

On first pass I don't see anything troubling. I will make a more thorough review tomorrow morning with fresh eyes and see if I can fix the failing workflows (or can make suggestions for how you may).

Thank you for your hard work on this! Its really awesome to see how the refactor you did shrunk both demos by over 1k lines combined. I'd says that's a huge win :)

[kstribrnAmzn]

/bot run uncrustify

[kstribrnAmzn]

Should these instructions be there? i'm not seeing how they are used in the step

[giuspen]

Correct

[kstribrnAmzn]

Why set these values to a local variable here when generateS3ObjectFilePresignedURL sets them to a global?

[kstribrnAmzn]

I've pushed a commit applying some fixes from the workflow failures. I doubt I've got them all so I may be applying a further commit.

[kstribrnAmzn]

Did a bit of testing and found all of these lines (and their corresponding globals) to be unneeded. Could you remove these?

[giuspen]

Thanks for your review @kstribrnAmzn I'm pretty busy for a couple of days but then I'll process your comments!

[giuspen]

@kstribrnAmzn thanks again for your work on my PR, I removed the unneeded code that you spotted and the leftover from copy/paste in the README

[kstribrnAmzn]

Merged. Thank you for your hard work on this one @giuspen 😄

[giuspen]

Cool, thanks to you @kstribrnAmzn take care ;)

[yinlonglyl]

@giuspen First of all, thanks for your contribution of 'http_demo_s3_generate_presigned_url'. I had a problem using it:

1. The 'http_demo_s3_download' already working in my embeded env, I can do the s3 download with STS token.
2. I ran into 'SignatureDoesNotMatch' error when using 'http_demo_s3_generate_presigned_url' (For both http 'Get' and 'Put' method). I have been blocked for 3 days, can you help me. Please check the error log of 'curl presignd_url' for detailed info:

curl "https://luaplugin.s3.us-east-1.amazonaws.com/upload/Hello_0918.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAUK2JWNPWOOD4JUBL%2F20240920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240920T065555Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEDcaCXVzLXdlc3QtMiJHMEUCIQDQl17z9nEZypd5QpWg04rwKTjBWrvnMC26zTwlt6VAuAIgHKY2IIZDFXy90aNKGEDr65dCj0lN%2FzUQ83zO69JFAgwqvgMIcBABGgwyOTgxMTc5MTc2NzYiDDmzg837AviFVJUkbCqbAzgT0Wn1heZWmXiU2AjY9kHDfMJRQ4J5heZO95Ist%2FYHtvtLqwC%2B2tz%2FhQ3Z2k4dVc3K6uuUUzD6qPC39OlDtUdX3coT%2Fjz3LqP2Tvm2FmCJ%2FiLl4OZHyyrSiauKX90DwO6NIhl3dnxfI3l%2BWGOTqKnxazhNv0f7zGp9D7WkXiWu8JuVKa1TioxQPx49wQoKvw3GVXIQyT%2BhCGPrsvi%2BuCLF2TmnP16uRDxNc8fc9uc38tm%2BxNKeALYkxmgnpRE8vTpldTzlP2zX73cnh%2FQaJgN04f6MFakflE6Ul8heA0etOrLjPgKxS2G93rgmwRJrTXlEm%2BQFfXxnkr1ucO%2Fl80XoVtNifN%2BR0wQi8dLobmvFOERxxO7k8Xv3nYTlzgoC%2F%2FmdMeYRzLhV2QURP7IZJBY0WbOJ713m9TIGIKzU7pn1jpwFOSkkVMQmT3b02Lzo9nknOS6cwDQZ44C1wR1YMGb%2FHzBQaj1rytXL9EnUh380kjEa%2BFnkqo60Pj%2BNnM0hSoLybNPDmprYldRb1S1uVtJKmtu6ZtkVpTuwZzD7uLS3BjqWAdgZj%2FI6oEwWj0%2BNN7QzGM2Z8Zx6GGn5SvUCrcuXRLQvO1VRlJjYMP52YRPHDAm8JE4L%2BRjmEthuhnmZ9aPgiFHNaQyNV6xXEf21jhEF1CDh2aNLK9xm1M%2BYTOr5QMxS%2BBf27WeUWCVpPpAWlZAOevkThPgUVNKuelOtWUbOE%2BdVRTmcG2pZi5L1R3cbJRE09kFmL6wq1g%3D%3D&X-Amz-Signature=3247218b9665e08bc8fa9752202c5f906132cf51b86b645aad4254b1e66f8f89"

SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method.ASIAUK2JWNPWOOD4JUBLAWS4-HMAC-SHA256
20240920T065555Z
20240920/us-east-1/s3/aws4_request
ab08562475a1a73ebf7e3892093403a3e0492da2cd5376b625ce5f95fc4355263247218b9665e08bc8fa9752202c5f906132cf51b86b645aad4254b1e66f8f8941 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 34 30 39 32 30 54 30 36 35 35 35 35 5a 0a 32 30 32 34 30 39 32 30 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 61 62 30 38 35 36 32 34 37 35 61 31 61 37 33 65 62 66 37 65 33 38 39 32 30 39 33 34 30 33 61 33 65 30 34 39 32 64 61 32 63 64 35 33 37 36 62 36 32 35 63 65 35 66 39 35 66 63 34 33 35 35 32 36GET
/upload/Hello_0918.txt
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAUK2JWNPWOOD4JUBL%2F20240920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240920T065555Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEDcaCXVzLXdlc3QtMiJHMEUCIQDQl17z9nEZypd5QpWg04rwKTjBWrvnMC26zTwlt6VAuAIgHKY2IIZDFXy90aNKGEDr65dCj0lN%2FzUQ83zO69JFAgwqvgMIcBABGgwyOTgxMTc5MTc2NzYiDDmzg837AviFVJUkbCqbAzgT0Wn1heZWmXiU2AjY9kHDfMJRQ4J5heZO95Ist%2FYHtvtLqwC%2B2tz%2FhQ3Z2k4dVc3K6uuUUzD6qPC39OlDtUdX3coT%2Fjz3LqP2Tvm2FmCJ%2FiLl4OZHyyrSiauKX90DwO6NIhl3dnxfI3l%2BWGOTqKnxazhNv0f7zGp9D7WkXiWu8JuVKa1TioxQPx49wQoKvw3GVXIQyT%2BhCGPrsvi%2BuCLF2TmnP16uRDxNc8fc9uc38tm%2BxNKeALYkxmgnpRE8vTpldTzlP2zX73cnh%2FQaJgN04f6MFakflE6Ul8heA0etOrLjPgKxS2G93rgmwRJrTXlEm%2BQFfXxnkr1ucO%2Fl80XoVtNifN%2BR0wQi8dLobmvFOERxxO7k8Xv3nYTlzgoC%2F%2FmdMeYRzLhV2QURP7IZJBY0WbOJ713m9TIGIKzU7pn1jpwFOSkkVMQmT3b02Lzo9nknOS6cwDQZ44C1wR1YMGb%2FHzBQaj1rytXL9EnUh380kjEa%2BFnkqo60Pj%2BNnM0hSoLybNPDmprYldRb1S1uVtJKmtu6ZtkVpTuwZzD7uLS3BjqWAdgZj%2FI6oEwWj0%2BNN7QzGM2Z8Zx6GGn5SvUCrcuXRLQvO1VRlJjYMP52YRPHDAm8JE4L%2BRjmEthuhnmZ9aPgiFHNaQyNV6xXEf21jhEF1CDh2aNLK9xm1M%2BYTOr5QMxS%2BBf27WeUWCVpPpAWlZAOevkThPgUVNKuelOtWUbOE%2BdVRTmcG2pZi5L1R3cbJRE09kFmL6wq1g%3D%3D&X-Amz-SignedHeaders=host
host:luaplugin.s3.us-east-1.amazonaws.com

host
UNSIGNED-PAYLOAD47 45 54 0a 2f 75 70 6c 6f 61 64 2f 48 65 6c 6c 6f 5f 30 39 31 38 2e 74 78 74 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 53 49 41 55 4b 32 4a 57 4e 50 57 4f 4f 44 34 4a 55 42 4c 25 32 46 32 30 32 34 30 39 32 30 25 32 46 75 73 2d 65 61 73 74 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 34 30 39 32 30 54 30 36 35 35 35 35 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 36 30 30 26 58 2d 41 6d 7a 2d 53 65 63 75 72 69 74 79 2d 54 6f 6b 65 6e 3d 49 51 6f 4a 62 33 4a 70 5a 32 6c 75 58 32 56 6a 45 44 63 61 43 58 56 7a 4c 58 64 6c 63 33 51 74 4d 69 4a 48 4d 45 55 43 49 51 44 51 6c 31 37 7a 39 6e 45 5a 79 70 64 35 51 70 57 67 30 34 72 77 4b 54 6a 42 57 72 76 6e 4d 43 32 36 7a 54 77 6c 74 36 56 41 75 41 49 67 48 4b 59 32 49 49 5a 44 46 58 79 39 30 61 4e 4b 47 45 44 72 36 35 64 43 6a 30 6c 4e 25 32 46 7a 55 51 38 33 7a 4f 36 39 4a 46 41 67 77 71 76 67 4d 49 63 42 41 42 47 67 77 79 4f 54 67 78 4d 54 63 35 4d 54 63 32 4e 7a 59 69 44 44 6d 7a 67 38 33 37 41 76 69 46 56 4a 55 6b 62 43 71 62 41 7a 67 54 30 57 6e 31 68 65 5a 57 6d 58 69 55 32 41 6a 59 39 6b 48 44 66 4d 4a 52 51 34 4a 35 68 65 5a 4f 39 35 49 73 74 25 32 46 59 48 74 76 74 4c 71 77 43 25 32 42 32 74 7a 25 32 46 68 51 33 5a 32 6b 34 64 56 63 33 4b 36 75 75 55 55 7a 44 36 71 50 43 33 39 4f 6c 44 74 55 64 58 33 63 6f 54 25 32 46 6a 7a 33 4c 71 50 32 54 76 6d 32 46 6d 43 4a 25 32 46 69 4c 6c 34 4f 5a 48 79 79 72 53 69 61 75 4b 58 39 30 44 77 4f 36 4e 49 68 6c 33 64 6e 78 66 49 33 6c 25 32 42 57 47 4f 54 71 4b 6e 78 61 7a 68 4e 76 30 66 37 7a 47 70 39 44 37 57 6b 58 69 57 75 38 4a 75 56 4b 61 31 54 69 6f 78 51 50 78 34 39 77 51 6f 4b 76 77 33 47 56 58 49 51 79 54 25 32 42 68 43 47 50 72 73 76 69 25 32 42 75 43 4c 46 32 54 6d 6e 50 31 36 75 52 44 78 4e 63 38 66 63 39 75 63 33 38 74 6d 25 32 42 78 4e 4b 65 41 4c 59 6b 78 6d 67 6e 70 52 45 38 76 54 70 6c 64 54 7a 6c 50 32 7a 58 37 33 63 6e 68 25 32 46 51 61 4a 67 4e 30 34 66 36 4d 46 61 6b 66 6c 45 36 55 6c 38 68 65 41 30 65 74 4f 72 4c 6a 50 67 4b 78 53 32 47 39 33 72 67 6d 77 52 4a 72 54 58 6c 45 6d 25 32 42 51 46 66 58 78 6e 6b 72 31 75 63 4f 25 32 46 6c 38 30 58 6f 56 74 4e 69 66 4e 25 32 42 52 30 77 51 69 38 64 4c 6f 62 6d 76 46 4f 45 52 78 78 4f 37 6b 38 58 76 33 6e 59 54 6c 7a 67 6f 43 25 32 46 25 32 46 6d 64 4d 65 59 52 7a 4c 68 56 32 51 55 52 50 37 49 5a 4a 42 59 30 57 62 4f 4a 37 31 33 6d 39 54 49 47 49 4b 7a 55 37 70 6e 31 6a 70 77 46 4f 53 6b 6b 56 4d 51 6d 54 33 62 30 32 4c 7a 6f 39 6e 6b 6e 4f 53 36 63 77 44 51 5a 34 34 43 31 77 52 31 59 4d 47 62 25 32 46 48 7a 42 51 61 6a 31 72 79 74 58 4c 39 45 6e 55 68 33 38 30 6b 6a 45 61 25 32 42 46 6e 6b 71 6f 36 30 50 6a 25 32 42 4e 6e 4d 30 68 53 6f 4c 79 62 4e 50 44 6d 70 72 59 6c 64 52 62 31 53 31 75 56 74 4a 4b 6d 74 75 36 5a 74 6b 56 70 54 75 77 5a 7a 44 37 75 4c 53 33 42 6a 71 57 41 64 67 5a 6a 25 32 46 49 36 6f 45 77 57 6a 30 25 32 42 4e 4e 37 51 7a 47 4d 32 5a 38 5a 78 36 47 47 6e 35 53 76 55 43 72 63 75 58 52 4c 51 76 4f 31 56 52 6c 4a 6a 59 4d 50 35 32 59 52 50 48 44 41 6d 38 4a 45 34 4c 25 32 42 52 6a 6d 45 74 68 75 68 6e 6d 5a 39 61 50 67 69 46 48 4e 61 51 79 4e 56 36 78 58 45 66 32 31 6a 68 45 46 31 43 44 68 32 61 4e 4c 4b 39 78 6d 31 4d 25 32 42 59 54 4f 72 35 51 4d 78 53 25 32 42 42 66 32 37 57 65 55 57 43 56 70 50 70 41 57 6c 5a 41 4f 65 76 6b 54 68 50 67 55 56 4e 4b 75 65 6c 4f 74 57 55 62 4f 45 25 32 42 64 56 52 54 6d 63 47 32 70 5a 69 35 4c 31 52 33 63 62 4a 52 45 30 39 6b 46 6d 4c 36 77 71 31 67 25 33 44 25 33 44 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 0a 68 6f 73 74 3a 6c 75 61 70 6c 75 67 69 6e 2e 73 33 2e 75 73 2d 65 61 73 74 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 4443X35CP2B3C31VDDtmw9dpD21ycJl2v35dSjxau6oETwMty+uyeEz9696y/DNqv6Dr9XTGsLRnrdHoRya8GmTVJOwdA=

[giuspen]

Hey @yinlonglyl my advice is to first have the demo working with your parameters / valid keys and certificate in a linux desktop.
I just tried the following myself and it is still working perfectly (I have my keys and certificates in /opt/aws)

cd ~/git/aws-iot-device-sdk-embedded-C
rm -rf build
mkdir build && cd build
cmake .. -DAWS_IOT_ENDPOINT="***your endpoint here***" \
         -DROOT_CA_CERT_PATH="/opt/aws/AmazonRootCA1.pem" \
         -DCLIENT_CERT_PATH="/opt/aws/device.pem.crt" \
         -DCLIENT_PRIVATE_KEY_PATH="/opt/aws/private.pem.key" \
         -DAWS_IOT_THING_NAME="***your thing name***" \
         -DAWS_IOT_CREDENTIAL_PROVIDER_ENDPOINT="***your credential provider endpoint (see s3 download setup)***" \
         -DAWS_IOT_CREDENTIAL_PROVIDER_ROLE="your credential provider role (see s3 download setup)***" \
         -DAWS_S3_BUCKET_NAME="***your s3 bucket name***" \
         -DAWS_S3_BUCKET_REGION="***your s3 bucket region***" \
         -DAWS_S3_OBJECT_NAME="***your file name in the bucket***"
make -j
./bin/http_demo_s3_generate_presigned_url

And you get the URL printed out. If you copy and paste the URL in a browser and it works / you can download, then your bug is in porting the code from the demo to your embedded system.

[yinlonglyl]

@giuspen Thanks very much!
I find the the root cause was that my coreHTTP library version was too low(v2.1.0), it does not support 'HTTP_REQUEST_NO_USER_AGENT_FLAG', need to update to v3.0.0.
Thanks again! That was really helpful!

[giuspen]

I'm glad you sorted out @yinlonglyl ;)