[Bug] HTTP Streaming doesn't work on Premium plans

[hallvictoria]

Expected Behavior

Setting "PYTHON_ENABLE_INIT_INDEXING": "1" as an app setting, including the azurefunctions-extensions-http-fastapi streaming library, and deploying an app to a Premium or Dedicated plan should be successful, and the function should show in the portal and execute as expected.

Actual Behavior

The worker fails indexing, and the function doesn't load or show in the portal.

Steps to Reproduce

1. Follow the steps listed in Azure Functions Support for Http Streams in Python.
2. Deploy the app to a Premium or Dedicated plan

Relevant code being tried

import time
import azure.functions as func
from azurefunctions.extensions.http.fastapi import Request, StreamingResponse

app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS)


def generate_count():
    """Generate a stream of chronological numbers."""
    count = 0
    while True:
        yield f"counting, {count}\n\n"
        count += 1

@app.route(route="stream", methods=[func.HttpMethod.GET])
async def stream_count(req: Request) -> StreamingResponse:
    """Endpoint to stream of chronological numbers."""
    return StreamingResponse(generate_count(), media_type="text/event-stream")

Relevant log output

Worker failed to index functions

Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure Exception: FunctionLoadError: cannot load the stream_count function: 'req' binding type 'httpTrigger' and dataType 'None' in function.json do not match the corresponding function parameter's Python type annotation 'Request'

requirements.txt file

azure-functions
azurefunctions-extensions-http-fastapi

Where are you facing this problem?

Production Environment (explain below)

Function app name

No response

Additional Information

The team is aware of this issue and is actively working to address it.

As a workaround for now, this feature works as expected following the steps listed in Azure Functions Support for Http Streams in Python for the Consumption and Flex Consumption plans.

Thank you for your patience, and we will provide updates here when the issue has been resolved.

[mgburnsy]

Love that you are working on this, thank you so much!!!

[alan-czysz]

We had this same problem (deploying to Premium app), but never saw this log output. Where did you find it?

RE: Worker failed to index functions

Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure Exception: FunctionLoadError: cannot load the stream_count function: 'req' binding type 'httpTrigger' and dataType 'None' in function.json do not match the corresponding function parameter's Python type annotation 'Request'

[hallvictoria]

Hi @alan-czysz, this is part of our internal logs, but you can also see it when running the function locally (func host start --verbose) if the PYTHON_ENABLE_INIT_INDEXING flag is missing.

[hallvictoria]

Thanks everyone for your patience. This issue has been addressed. Creating a new app using streaming on the Premium / Dedicated plans following the above documentation will now work as expected. If you have already created and deployed an app, please restart it.

Please let us know if you're still facing this issue!

[alan-czysz]

@hallvictoria - has something changed in Premium apps since last week? As of Friday, all of our function apps using azurefunctions.extensions.http.fastapi and StreamingResponse were working on a Premium function app deployment slot. As of yesterday (6/10/2024) the apps started failing with messages in app-insights like: "No initialized language worker channel found for runtime: python."

Attempting to redeploy from VSCode fails with either "Encountered an error (BadGateway) from host runtime." or "Encountered an error (InternalServerError) from host runtime." after the following sequence executes:
9:49:05 AM : PreDeployment: context.OutputPath /home/site/wwwroot
9:49:06 AM : Repository path is /tmp/zipdeploy/extracted
9:49:06 AM : Running oryx build...
9:49:06 AM : Command: oryx build /tmp/zipdeploy/extracted -o /tmp/build/expressbuild --platform python --platform-version 3.11 -i /tmp/8dc8a1d421d4b8b -p packagedir=.python_packages/lib/site-packages
9:49:17 AM : Syncing triggers...
9:49:42 AM : Syncing triggers (Attempt 2/6)...
9:49:53 AM : Syncing triggers (Attempt 3/6)...
9:50:13 AM : Syncing triggers (Attempt 4/6)...
9:51:44 AM : Syncing triggers (Attempt 5/6)...
9:53:55 AM : Syncing triggers (Attempt 6/6)...
9:54:13 AM: Error: Encountered an error (BadGateway) from host runtime.

I get the same message when I try to execute the code snippet provided in the original comment for this issue. I've tried spinning up a new premium app and a new deployment slot on our current premium app; both result in the same error. Lastly, the code I'm attempting to deploy to Premium is deploying without issue to a consumption app and a flex consumption app via a VSCode deployment.

Thanks!

[hallvictoria]

Hi @alan-czysz, would you be willing to share your function app name? (sharing your function app name privately)

Also, is this error happening for all your function apps, or just those using streaming?

[alan-czysz]

@hallvictoria - of course!

1. https://stratsimpleai-test.azurewebsites.net is the function app running on a flex consumption plan. All of the http triggers in this app are using azurefunctions.extensions.http.fastapi, only one end-point actually uses streaming. This function app contains the code we were trying to run on the EP1 plan.
2. https://stratsimpleai.azurewebsites.net is the EP1 plan function app. It is currently running http triggers that were built without azurefunctions.extensions.http.fastapi and there are no streaming implementations. Prior to Monday, the non-production deployment slot in https://stratsimpleai.azurewebsites.net was running the same code that currently sits on https://stratsimpleai-test.azurewebsites.net.

The above error was occurring anytime we tried to deploy the package of http triggers that included azurefunctions.extensions.http.fastapi to the https://stratsimpleai.azurewebsites.net function app via VSCode.

Build & Deploy to https://stratsimpleai.azurewebsites.net/ from our source code workflow would succeed... then generate errors like "No initialized language worker channel found for runtime: python." in app-insights when we restarted the function app.

If you need more information, happy to send you anything you need via email.

[hallvictoria]

Thanks for this! It looks like there are two different issues here -- failure to deploy to a premium app through VSCode, and the "No initialized language worker" error message.

I wasn't able to repro the VSCode deployment error. If it's consistently failing, please file an issue in this repo so we can take a look at it. As a mitigation for now, please try using remote build (func azure functionapp publish <function name>).

The "No initialized language worker" error is being caused by a timeout when indexing the function app. We currently have a limit of 10s, so function apps with a lot of functions or large package requirements will time out (this is a known issue, and we're working on it!). Indexing times can also vary.

I saw this timeout error occur in the stratsimpleai function app even before 6/7, so it looks like the app is right on the edge of the timeout limit. Since the indexing times can have transient variations, that's why sometimes it would load and sometimes it wouldn't. Removing the azurefunctions.extensions.http.fastapi library / converting the http functions might've been enough to meet the timeout limit.

Consumption and flex SKUs have faster indexing times, so as a mitigation please use those for now! I apologize for this issue and confusion. We're working to address timeout issues for large function apps that want to utilize streaming, but I don't have an ETA for that right now.