A ReadableStream branch was created but never consumed

[HaidongPang]

I'm using ky in cloudflare worker with following code:

import ky from "ky";
export default {
    async fetch (): Promise<Response> {
        const result = await ky.get("https://www.github.com").text();
        return new Response(result);
    }
} satisfies ExportedHandler;

and when I turn off local mode, requests raise the following warning:

A ReadableStream branch was created but never consumed. Such branches can be created, for instance, by calling the tee() method on a ReadableStream, or by calling the clone() method on a Request or Response object. If a branch is created but never consumed, it can force the runtime to buffer the entire body of the stream in memory, which may cause the Worker to exceed its memory limit and be terminated. To avoid this, ensure that all branches created are consumed.

 * Unused stream created:
    at result.<computed> [as text] (file:///....../node_modules/ky/source/core/Ky.ts:92:36)
......

Locate the following code snippet:

source/core/Ky.ts

export class Ky {
	static create(input: Input, options: Options): ResponsePromise {
		const ky = new Ky(input, options);

		const function_ = async (): Promise<Response> => {
                         ......
		};

		......

		for (const [type, mimeType] of Object.entries(responseTypes) as ObjectEntries<typeof responseTypes>) {
			result[type] = async () => {
				......
				const awaitedResult = await result;
				const response = awaitedResult.clone();

				if (type === 'json') {
					if (response.status === 204) {
						return '';
					}

					const arrayBuffer = await response.clone().arrayBuffer();
					const responseSize = arrayBuffer.byteLength;
					if (responseSize === 0) {
						return '';
					}

					if (options.parseJson) {
						return options.parseJson(await response.text());
					}
				}

				return response[type]();
			};
		}

		return result;
	}

The awaitedResult is a variable of type Response. Since it is declared, it has been cloned only once, and its ReadableStream has never been consumed.
After awaitedResult has finished cloning, its ReadableStream should be closed to ensure memory safety.
Seems like:

const awaitedResult = await result;
const response = awaitedResult.clone();
await awaitedResult.body.cancel();

[sholladay]

PR welcome to fix this issue. If I'm not mistaken, canceling the stream could lead to lost data even though we've cloned the stream. We'd need to test it thoroughly to make sure that's not the case.

[HaidongPang]

I‘m willing to do so, and a PR have already been submitted .
Regarding your concerns, here is a simple verification.

export default {
    async fetch (): Promise<Response> {
        const originalResponse = new Response("Some Data");
        const clonedResponse = originalResponse.clone();
        await originalResponse.body?.cancel();
        console.log("Original Stream Data:", await originalResponse.body?.getReader().read());
        console.log("Cloned Stream Data:", await clonedResponse.body?.getReader().read());
        return new Response("");
    }
} satisfies ExportedHandler;

I cloned an unconsumed response and canceled the ReadableStream of the original response before consuming the cloned response.

The result is as follows:

Original Stream Data: Object {
  done: true
}
Cloned Stream Data: Object {
  value: Uint8Array(9),
  done: false
}

The original ReadableStream has already been canceled, but the cloned response has an independent ReadableStream object that can be consumed.

[HaidongPang]

There is also a similar issue about orphan ReadableStream remaining.

Related to :#650

[TheHolyWaffle]

I'm not sure if this applies to Cloudflare Workers, but the undici team released version 6.19.8 which will automatically garbage collect response streams if the response object is out of scope.

See nodejs/undici#2856

[work-in-progress-danny]

#661 might not have fixed this. I'm on v1.7.4 and getting this error

const response = await ky.post("...url...", {
  throwHttpErrors: false,
  retry: { limit: 0 },
  headers: {
    "Content-Type": "application/json",
    Authorization: `Bearer ${ctx.env.API_KEY}`,
  },
  json: <...data...>,
})

if (!response.ok) {
  console.error(response.statusText)
  return
}

const jsonResponse = await response.json()

ReadableStream branch was created but never consumed. Such branches can be created, for instance, by calling the tee() method on a ReadableStream, or by calling the clone() method on a Request or Response object. 
If a branch is created but never consumed, it can force the runtime to buffer the entire body of the stream in memory, which may cause the Worker to exceed its memory limit and be terminated.
To avoid this, ensure that all branches created are consumed.

 * Unused stream created:
    at Ky._fetch (file://..../node_modules/ky/source/core/Ky.ts:305:30)
    at function_ (file://..../node_modules/ky/source/core/Ky.ts:38:28)
    at async <function> (file://..../function.ts:30:20)

[sholladay]

@work-in-progress-danny you are probably encountering #650, which is technically a separate but similar problem that leads to the same outcome. Think of it as a "Part 2", if you will.

There is already a PR to fix it, see #651. I was hoping to release these together but something has been causing CI to fail on that PR and I'm not sure why yet. It passes locally for me. The failure might be related to usage of httpbin.org in the tests, which would make it unrelated to that PR, but that's just a guess. Once that is solved, we'll get that PR merged and released quickly. Feel free to help investigate the CI failure if you have time.

[work-in-progress-danny]

Awesome, thanks so much @sholladay! Love this package 🙏