The client reported an error when copying the request body

[gswartz777]

Describe the bug

Yarp was working fine recently and for some unknown reason it stopped working yesterday for me. It appears to be isolated to my machine though as it works on our other developer's machines. When trying to proxy an ajax json post it's saying "The client reported an error when copying the request body."

Exceptions

info: Yarp.ReverseProxy.Forwarder.HttpForwarder[9]
      Proxying to https://localhost:44301/MVC/Login/Verify HTTP/2 RequestVersionOrLower
warn: Yarp.ReverseProxy.Forwarder.HttpForwarder[48]
      RequestBodyClient: The client reported an error when copying the request body.
      System.AggregateException: One or more errors occurred. (Sent 0 request content bytes, but Content-Length promised 1255.) (An error occurred while sending the request.)
       ---> System.InvalidOperationException: Sent 0 request content bytes, but Content-Length promised 1255.
         --- End of inner exception stack trace ---
       ---> (Inner Exception #1) System.Net.Http.HttpRequestException: An error occurred while sending the request.
       ---> System.IO.IOException: An error occurred when reading the request body from the client.
       ---> System.InvalidOperationException: Sent 0 request content bytes, but Content-Length promised 1255.
         --- End of inner exception stack trace ---
         at Yarp.ReverseProxy.Forwarder.StreamCopyHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
         at Yarp.ReverseProxy.Forwarder.StreamCopyHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
         at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken)
         at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer, CancellationToken cancellationToken)<---

Further technical details

• Started while using 2.0.1, upgraded to 2.1.0 in hopes that might fix it, but it didn't.
• The platform Windows

[MihaZupan]

Sent 0 request content bytes, but Content-Length promised 1255.

This indicates that something's wrong with the body/header logic. Do you have any custom transforms/middleware that are reading/modifying the request body?

[gswartz777]

Not that I am aware of. I've gone back through commit logs to see what might have changed but I don't see anything. Would this reading/modifying the request body typically be happening on the .net 6 side where yarp is being used or on the old .net framework app side? I'm assuming the .net 6 side, but just confirming.

I don't know if it could be related, but a few days ago I updated my JetBrains Rider install to the latest version 2024.1.6 and also installed Visual Studio Community 2022. However, things were working fine after those updates until yesterday, so I don't think it's related. But again, this same code works fine on other developer machines.

Here's the ajax post code.

$.ajax({
    type: 'POST',
    url: '/MVC/Login/Verify',
    headers: {
        'X-Requested-With': 'XMLHttpRequest'
    },
    contentType: "application/json",
    success: function (result) {
        $("#errorMsg").text(result);
        if (result === "User logged in.") {
            window.location.href = "/default.aspx";
        }
    },
    processData: false,
    dataType: "json",
    data: JSON.stringify(packet)
});

[MihaZupan]

That'd be on the app where you're using YARP.
E.g. it could be caused by logging being implemented incorrectly.

The IDE used is very unlikely to be related here.

You could try getting a network capture without https to see what content the client is sending the proxy.

[gswartz777]

Here's what I see. From what I can tell, it looks like it's successfully posting the body .

[MihaZupan]

Are you able to create a minimal runnable repro?

[gswartz777]

I will try but I just noticed this. When the app using yarp first spins up it's throwing an error about "System.Threading.Tasks.TaskCanceledException: A task was canceled. at Yarp.ReverseProxy.Forwarder.StreamCopier.CopyAsync" and I wonder if this is something in the startup program that's failing that is later used to copy the body of the request? When I look at line 145 in the program file it's just the await next() portion of this block of code. I've confirmed with the other developers that they aren't getting this yarp error message.

app.Use(async (context, next) =>
{
    context.Request.EnableBuffering();
    await next();
});

info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: D:\Code\Contract\src\FW\
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[9]
      Proxying to https://localhost:44301/ HTTP/2 RequestVersionOrLower no-streaming
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[56]
      Received HTTP/2.0 response 302.
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[48]
      ResponseBodyCanceled: Copying the response body was canceled.
      System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Yarp.ReverseProxy.Forwarder.StreamCopier.CopyAsync(Stream input, Stream output, Int64 promisedContentLength, StreamCopierTelemetry telemetry, ActivityCancellationTokenSource activityToken, Boolean autoFlush, CancellationToken cancellation)
fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.InvalidOperationException: StatusCode cannot be set because the response has already started.
         at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ThrowResponseAlreadyStartedException(String name)
         at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.set_StatusCode(Int32 value)
         at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.Microsoft.AspNetCore.Http.Features.IHttpResponseFeature.set_StatusCode(Int32 value)
         at Microsoft.AspNetCore.Http.DefaultHttpResponse.Redirect(String location, Boolean permanent)
         at Microsoft.AspNetCore.Http.HttpResponse.Redirect(String location)
         at FH.Infrastructure.Web.Errors.WebErrorHandler.DisplayPublicError() in D:\Code\Contract\src\FH.Infrastructure\Web\Errors\WebErrorHandler.cs:line 110
         at FH.Infrastructure.Web.Middleware.ErrorHandlingMiddleware.Invoke(HttpContext context) in D:\Code\Contract\src\FH.Infrastructure\Web\Middleware\ErrorHandlingMiddleware.cs:line 33
         at FH.Infrastructure.Web.Middleware.ErrorHandlingMiddleware.handleException(Exception ex, HttpContext context) in D:\Code\Contract\src\FH.Infrastructure\Web\Middleware\ErrorHandlingMiddleware.cs:line 63 
         at Program.<>c.<<<Main>$>b__0_12>d.MoveNext() in D:\Code\Contract\src\FW\Program.cs:line 145
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
warn: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[2]
      The response has already started, the error page middleware will not be executed.
fail: Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer[2]
      Connection ID "18230571309580943364", Request ID "40000005-0004-fd00-b63f-84710c7967bb": An unhandled exception was thrown by the application.
      System.InvalidOperationException: StatusCode cannot be set because the response has already started.
         at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ThrowResponseAlreadyStartedException(String name)
         at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.set_StatusCode(Int32 value)
         at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.Microsoft.AspNetCore.Http.Features.IHttpResponseFeature.set_StatusCode(Int32 value)
         at Microsoft.AspNetCore.Http.DefaultHttpResponse.Redirect(String location, Boolean permanent)
         at Microsoft.AspNetCore.Http.HttpResponse.Redirect(String location)
         at FH.Infrastructure.Web.Errors.WebErrorHandler.DisplayPublicError() in D:\Code\Contract\src\FH.Infrastructure\Web\Errors\WebErrorHandler.cs:line 110
         at FH.Infrastructure.Web.Middleware.ErrorHandlingMiddleware.handleException(Exception ex, HttpContext context) in D:\Code\Contract\src\FH.Infrastructure\Web\Middleware\ErrorHandlingMiddleware.cs:line 63 
         at FH.Infrastructure.Web.Middleware.ErrorHandlingMiddleware.Invoke(HttpContext context) in D:\Code\Contract\src\FH.Infrastructure\Web\Middleware\ErrorHandlingMiddleware.cs:line 33
         at Program.<>c.<<<Main>$>b__0_12>d.MoveNext() in D:\Code\Contract\src\FW\Program.cs:line 145
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
         at Microsoft.Extensions.DependencyInjection.RegisterAdapterFeaturesMiddleware.InvokeAsync(HttpContext context)
         at Microsoft.Extensions.DependencyInjection.RegisterAdapterFeaturesMiddleware.InvokeAsync(HttpContext context)
         at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()
fail: Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer[2]
      Connection ID "18230571309580943364", Request ID "40000005-0004-fd00-b63f-84710c7967bb": An unhandled exception was thrown by the application.
      System.Runtime.InteropServices.COMException (0x800704CD): An operation was attempted on a nonexistent network connection. (0x800704CD)
         at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()

[MihaZupan]

Hey, any luck with getting this working / creating a repro?