Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ModelAuthProvider restricted to @Blocking #1214

Open
flyinfish opened this issue Jan 10, 2025 · 8 comments · May be fixed by #1215
Open

ModelAuthProvider restricted to @Blocking #1214

flyinfish opened this issue Jan 10, 2025 · 8 comments · May be fixed by #1215

Comments

@flyinfish
Copy link

ISSUE

actually i am not 100% shure if bug,issue or i just missed something...
i am using ModelAuthProvider as described in https://docs.quarkiverse.io/quarkus-langchain4j/dev/openai.html#_using_authprovider.

everything works fine except that it forces me to use @Blocking when using SSE

  @GET
  @Path("/stream")
  @Produces(MediaType.SERVER_SENT_EVENTS)
  //@Blocking
  public Multi<String> helloStream(@QueryParam("message") Optional<String> message) {
    return helloAI.chatStream(message.orElse("Hi"));
  }

@ApplicationScoped
public class AzureOpenAiTokenProvider implements ModelAuthProvider {
  ...

  @Override
  public String getAuthorization(Input input) {
    var token = tokenCache.getTokenSync(TOKEN_REQUEST_CONTEXT, false).getToken();
    return format("Bearer %s", token);
  }
}

: io.vertx.core.VertxException: Thread blocked
        at java.base/java.io.FileInputStream.readBytes(Native Method)
....
        at com.azure.identity.implementation.IdentityClientBase.getTokenFromAzureCLIAuthentication(IdentityClientBase.java:681)
        at com.azure.identity.implementation.IdentitySyncClient.authenticateWithAzureCli(IdentitySyncClient.java:393)
        at com.azure.identity.AzureCliCredential.getTokenSync(AzureCliCredential.java:97)
        at com.azure.identity.ChainedTokenCredential.getTokenSync(ChainedTokenCredential.java:145)
        at com.azure.core.implementation.AccessTokenCache.lambda$new$2(AccessTokenCache.java:65)
        at com.azure.core.implementation.AccessTokenCache.lambda$retrieveTokenSync$11(AccessTokenCache.java:230)
        at com.azure.core.implementation.AccessTokenCache.getTokenSync(AccessTokenCache.java:93)
        at io.quarkiverse.langchain4j.sample.hello.AzureOpenAiTokenProvider.getAuthorization(AzureOpenAiTokenProvider.java:30)

Expected Behaviour

wouldn't it be nice to use non-blocking auth which at least azure would provide

Workaround

using blocking communication for whole llm-call

Reproduce

git clone https://github.com/flyinfish/quarkus-langchain4j--samples-azure-openai.git
cd hellomodelauthprovider/
quarkus dev
http://localhost:8080/hello/stream
@geoand geoand linked a pull request Jan 13, 2025 that will close this issue
Draft
@geoand
Copy link
Collaborator

geoand commented Jan 13, 2025

We probably want to provide both a sync and async version instead of just dropping the sync one completely...

@flyinfish
Copy link
Author

We probably want to provide both a sync and async version instead of just dropping the sync one completely...

this would mean 2 interfaces ModelAuthProvider ModelAuthProviderAsync?

by the way there seems to be another approach in gemini to work around the issue:
https://github.com/flyinfish/quarkus-langchain4j/blob/942c8306e43ff3e82e7ef2ea3f61769e1da1de18/model-providers/vertex-ai-gemini/runtime/src/main/java/io/quarkiverse/langchain4j/vertexai/runtime/gemini/VertxAiGeminiRestApi.java#L143

@geoand
Copy link
Collaborator

geoand commented Jan 13, 2025 •
edited
Loading

this would mean 2 interfaces ModelAuthProvider ModelAuthProviderAsync?

Probably just a single interface with one async and one sync method

by the way there seems to be another approach in gemini to work around the issue:
https://github.com/flyinfish/quarkus-langchain4j/blob/942c8306e43ff3e82e7ef2ea3f61769e1da1de18/model-providers/vertex-ai-gemini/runtime/src/main/java/io/quarkiverse/langchain4j/vertexai/runtime/gemini/VertxAiGeminiRestApi.java#L143

Right, I need to look at this in a more systematic way

@sberyozkin
Copy link
Contributor

Right, I need to look at this in a more systematic way

Sure, it will likely be best if having 2 methods can be avoided, I recall Quarkus REST or REST Client can also decide if it is blocking given the response type of the filter

@flyinfish
Copy link
Author

@geoand so you guys take over from here - is there anything you expect from me?

@geoand
Copy link
Collaborator

geoand commented Jan 14, 2025

Sure yeah

@flyinfish
Copy link
Author

the next step in context of "azure-openai" would the be to provide a default-implementation somehow like https://github.com/flyinfish/quarkus-langchain4j/blob/942c8306e43ff3e82e7ef2ea3f61769e1da1de18/model-providers/vertex-ai-gemini/runtime/src/main/java/io/quarkiverse/langchain4j/vertexai/runtime/gemini/VertxAiGeminiRestApi.java#L143 does.

but one step after the other

@geoand
Copy link
Collaborator

geoand commented Jan 14, 2025

@sberyozkin do you plan to look into this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Draft: refactor ModelAuthProvider to async flyinfish/quarkus-langchain4j
3 participants
@sberyozkin @flyinfish @geoand