Socket hang up while uploading sourcemaps #1129
Comments
|
Hi @mychaelgo. How often are you experiencing this issue? It looks like this happens because the upload takes too much time. It might be a temporary connectivity issue on your side, for example on a network with very low bandwidth. |
|
@BenoitZugmeyer we're experiencing this issue as well, we can reproduce it consistently when trying to upload thousands of sourcemaps, it's fast for the first minute and then everything starts failing and starts retrying. I suspect the DataDog backend is either rate limiting our CI or the backend is failing due to the volume of requests being made to upload all sourcemaps for our application. In the logs we see a combination of "socket hang up" and "Request failed with status code 408". |
|
Thank you for your feedback. I investigated a bit and found something that might help improve the situation. Stay tuned |
|
@mattlewis92 is this an issue you started experiencing recently, or did you always experienced it? Do your CI runs on Azure/GCP/AWS? |
It's only started happening to us since January 5th, but that build also increased the amount of sourcemaps we were uploading, so it's possible the issue has always been there and we've just hit an upper limit where it starts to trigger. This would make sense as the problem only starts to occur towards the last bunch of sourcemaps uploaded. Our CI runs on github actions, so would be from the Azure data center under the hood, as we are not using self hosted runners. |
|
Sure thing! Our next prod release isn't for another week, so will check in then and let you know if the issue is gone! |
|
I took a look at our logs today after upgrading |
|
Ok, thank you for trying it out. Maybe with less concurrency, requests will end more quicky, preventing timeouts? Could you try to run the upload command with the |
Sure thing, will try and that and let you know how it goes! |
|
Then I tried deleting a few hundred sourcemaps to bring the total count under 3000 and everything gets fast, and all of them are uploaded in under a minute: The problem only started after we crossed the 3000 count for sourcemaps, so I'm reasonably confident there is a hard limit somewhere on the datadog side, either connections are not being closed after upload by the client, or maybe the backend API that accepts the uploads is having a firewall rule triggered that starts blocking the requests as it thinks github actions is trying to ddos the endpoint. |
|
Let's keep this issue open until we have confirmation that it is indeed fixed. |
|
Each request has a 1 minute timeout on our side, this is why we are seeing some HTTP requests failing with status 408. When the request was retried, an implementation bug was causing the request to never end, causing the We released a fix for retrying uploads in v2.30.0. We won't increase the 1 minute timeout for now as it has security implications. We might revisit later. As your source maps files are relatively small, retrying the upload after a timeout should work now that it is fixed. If it's still unreliable and you are still seeing failing HTTP requests, using a lower concurrency might help uploading individual source maps faster. Could you try it again and let me know if it improves the situation? Sorry for the inconvenience and the back and forth! |
Thanks for digging into this! After upgrading to When compared to the run where I deleted a few hundred sourcemaps, it's still quite slower: Will try lowering Edit: setting |
Bug description
Sourcemaps upload error, even the file size < 500K
Describe what you expected
All sourcemaps uploaded without any errors
Steps to reproduce the issue
Additional context
Command
sourcemaps
The text was updated successfully, but these errors were encountered: