Handle tool calls in playground #758

andresgutgon · 2025-01-02T11:41:14Z

What?

We have a task to allow users to give a response on one or more tool calls that the AI can request in our playground. The thing is that the work necessary to make this work is also essential for making tool calling work on our SDK and Gateway. So this PR is a spike to have a picture of what are the moving parts necessary to make this happen.

Related issue

#598

Possible affected clients

These are the places I think these changes affect

Web app playground
Running prompts in batch (from playground)
Users using our SDK clients
Users using our API

TODO

Tool calls when running prompt in Batch in Playground 🧠 🧠 🧠 🤯

How we can do this?

In SDK

Talked with @csansoon this is how it would look to have tool calls in SDK

const response = sdk.run(
  'weather_experet', 
  { location: "Barcelona" }, 
  { 
    tools: [
      // Same name as in schema
      get_weather: (location: string) => {
         // Call some API or DB and get weather in "loc"
         return { temperature: 23 }
      }
    ]
  }
)

Considerations

AI can respond with multiple tool calls at the same time. So maybe we need to handle it at the SDK level as a Promise.all and wait for all the calls to be finished before continuing.
What happens if the work to be done in one or more of the tool calls is heavy and needs to be put in a queue in the user's system? Do you think we should pass the reference to the cached partial response (serialized chain) in the tool calls?

andresgutgon · 2025-01-02T11:42:54Z

apps/web/src/actions/sdk/runDocumentAction.ts

@@ -28,6 +32,7 @@ export async function runDocumentAction({
  parameters,
 }: RunDocumentActionProps) {
  const { workspace, user } = await getCurrentUserOrError()
+  let tools = []


Requests to SDK in our playground are stateless. We need to implement a cache to remember where the user stopped because there were one o more tool call requested by the AI

andresgutgon · 2025-01-02T11:43:57Z

apps/web/src/actions/sdk/runDocumentAction.ts

+        (type === ChainEventTypes.StepComplete ||
+          type === ChainEventTypes.Complete) &&
+        data.response.streamType === 'text'
+      ) {


I think this is the combination of AI response that produce tool calls

Step completed of type text

Chain Completed of type text

Am I missing something?

andresgutgon · 2025-01-02T11:45:35Z

packages/core/src/services/chains/run.ts

@@ -305,6 +313,8 @@ async function runStep({
    } else {
      streamConsumer.stepCompleted(response)

+      // TODO: Here instead of running the state if the step has
+      // toolCalls we serialize the chain up until this point


When the AI is cached or when is fresh in both cases the step response can be stepCompleted or chainCompleted and we're doing the same.

Now we also want to handle tool calling and chain serialization so I think we need to DRY this in to a new unified method that handle all the cases

apps/gateway/package.json

apps/web/next.config.mjs

apps/web/package.json

geclos · 2025-01-07T11:36:06Z

This API does not follow the pattern of other methods in the SDK:

const response = sdk.run(
  'weather_experet', 
  { location: "Barcelona" }, 
  { 
    tools: [
      // Same name as in schema
      get_weather: (location: string) => {
         // Call some API or DB and get weather in "loc"
         return { temperature: 23 }
      }
    ]
  }
)

Keep in mind sdk format follows this format: required arguments as positional + last options argument. This means that all required arguments are positional and then any optional arguments go in a last options arguments that is an object. In this case I imagine this would look something like:

const response = sdk.run(
  'weather_experet', 
  { 
   parameters: { location: 'barcelona'},
    tools: [
      get_weather: (location: string) => {
         return { temperature: 23 }
      }
    ]
  }
)

geclos · 2025-01-07T11:46:28Z

When stopping the chain because there are tool calls. Emit a new event or emit chainCompleted? ask @geclos and @csansoon for opinions. Alex voted for a new, event

Isn't this the same event than when a step is completed? Since a tool call request wouldbe the response from a step and providing the response would start a new step in the chain.

andresgutgon · 2025-01-07T11:52:24Z

Isn't this the same event than when a step is completed?

If we emit stepCompleted the stream is not stopped. I'm talking in our playground. For that I send chainCompleted to stop execution in the UI.

This works but maybe is not a good idea

andresgutgon · 2025-01-07T11:53:42Z

This API does not follow the pattern of other methods in the SDK:

So we have to use positional arguments?

In OpenAI playground they show arguments as an object

geclos · 2025-01-07T14:55:03Z

This API does not follow the pattern of other methods in the SDK:

So we have to use positional arguments?

In OpenAI playground they show arguments as an object

Let's maintain consistency across methods in our SDK, parameters is an optional parameter

andresgutgon · 2025-01-08T17:29:29Z

packages/core/src/services/chains/chainCache/index.ts

+  documentLogUuid: string
+}) {
+  const key = generateCacheKey({ documentLogUuid, workspace })
+  const serialized = chain.serialize()


chain.serialize() is a new method introduced in that PR

andresgutgon · 2025-01-08T17:34:03Z

apps/web/src/actions/sdk/runDocumentAction.ts

+  // and add more messages to the same stream session.
+  //
+  // For doing this we need a new gateway endpoint to send the tool calls to the server and run the chain
+  // also the SDK method to be called here


I'll remove this comment. For now I opted for a different action. Let's see how it feels.

neoxelox · 2025-01-14T10:44:18Z

packages/core/src/helpers.ts

+  const text = data.text
+  const object = type === 'object' ? data.object : undefined
+  const toolCalls = type === 'text' ? (data.toolCalls ?? []) : []
+  let content: MessageContent[] = []


nit: you can just avoid the content variable and push directly to the message.content

andresgutgon · 2025-01-16T10:40:26Z

packages/core/src/services/chains/ChainValidator/index.ts

    const { completed, messages, config } = await (chain as PromptlChain).step(
-      prevText,
+      prevContent as PromptlMessage[] | undefined | string,


We now can add to promptl an array of messages as the previous response.

andresgutgon · 2025-01-16T10:43:07Z

packages/core/src/helpers.ts

+          typeof toolCallResponse.result === 'string'
+            ? JSON.parse(toolCallResponse.result)
+            : toolCallResponse.result,
+        isError: toolCallResponse.isError || false,


I don't like the fact that we're building response message with the types of old compiler. But I don't want to change this now.

We should do a PR killing all compiler types and start using prompt ASAP

csansoon · 2025-01-20T08:20:50Z

...itUuid]/documents/[documentUuid]/_components/DocumentEditor/Editor/Playground/useMessages.ts

+  // FIXME: Kill compiler please. I made this module compatible with
+  // both old compiler and promptl. But because everything is typed with
+  // old compiler prompts in promptl version are also formatted as old compiler


I second this.

csansoon · 2025-01-20T08:24:52Z

packages/web-ui/src/ds/molecules/Chat/ChatTextArea/ToolCallForm/index.tsx

+        <Text.H6>
+          You have{' '}
+          {totalToolRequests <= 1 ? (
+            <>
+              <Badge variant='muted'>{totalToolRequests}</Badge> tool to be
+              responded
+            </>
+          ) : (
+            <>
+              <Badge variant='muted'>{currentToolRequest}</Badge> of{' '}
+              <Badge variant='muted'>{totalToolRequests}</Badge> tools to be
+              responded
+            </>
+          )}
+        </Text.H6>


Now that this is the first thing you read on the Tool Response input, since it is the top text, I'd move the (?) Tooltip here instead, to explain what do "tools to be responded" mean.

csansoon · 2025-01-20T08:26:02Z

packages/web-ui/src/ds/molecules/DocumentTextEditor/Editor/RegularEditor.tsx

-  const [isEditorMounted, setIsEditorMounted] = useState(false) // to avoid race conditions
+  // to avoid race conditions
+  const [isEditorMounted, setIsEditorMounted] = useState(false)


csansoon

lgtm!

We have a task to allow users give a response on one or more tool calls that the AI can request in our playground. The thing is that the work necessary to make this work is also necessary for making tool call work on our SDK so this PR is a spike to have a picture of what are the moving parts necessary to make this happen

andresgutgon added 🚧 wip Work in progress 🧩 compiler labels Jan 2, 2025

andresgutgon commented Jan 2, 2025

View reviewed changes

andresgutgon mentioned this pull request Jan 2, 2025

Serialize and deserialize chains latitude-dev/promptl#11

Merged

9 tasks

andresgutgon commented Jan 3, 2025

View reviewed changes

apps/gateway/package.json Outdated Show resolved Hide resolved

andresgutgon commented Jan 3, 2025

View reviewed changes

apps/web/next.config.mjs Outdated Show resolved Hide resolved

andresgutgon changed the title ~~Handle tool calls in SDK and playground~~ Handle tool calls in playground Jan 3, 2025

andresgutgon commented Jan 3, 2025

View reviewed changes

apps/web/package.json Outdated Show resolved Hide resolved

andresgutgon commented Jan 8, 2025

View reviewed changes

andresgutgon force-pushed the spike/tool-calls branch 4 times, most recently from c6b81db to 5f11b4e Compare January 14, 2025 09:54

neoxelox reviewed Jan 14, 2025

View reviewed changes

andresgutgon force-pushed the spike/tool-calls branch 3 times, most recently from 5f622ed to 1afb6cd Compare January 16, 2025 10:37

andresgutgon commented Jan 16, 2025

View reviewed changes

andresgutgon force-pushed the spike/tool-calls branch from 4b1da6b to c3d363d Compare January 17, 2025 15:00

andresgutgon removed the 🚧 wip Work in progress label Jan 17, 2025

andresgutgon force-pushed the spike/tool-calls branch from c3d363d to 5043e6f Compare January 17, 2025 18:16

csansoon reviewed Jan 20, 2025

View reviewed changes

andresgutgon force-pushed the spike/tool-calls branch from 5043e6f to 0bae63b Compare January 20, 2025 08:21

csansoon reviewed Jan 20, 2025

View reviewed changes

csansoon previously approved these changes Jan 20, 2025

View reviewed changes

andresgutgon dismissed csansoon’s stale review via 97f0894 January 20, 2025 08:47

andresgutgon force-pushed the spike/tool-calls branch from 0bae63b to 97f0894 Compare January 20, 2025 08:47

andresgutgon force-pushed the spike/tool-calls branch from 97f0894 to c8bede9 Compare January 20, 2025 09:22

andresgutgon merged commit 4b30a94 into main Jan 20, 2025
4 checks passed

andresgutgon deleted the spike/tool-calls branch January 20, 2025 09:44

github-actions bot locked and limited conversation to collaborators Jan 20, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle tool calls in playground #758

Handle tool calls in playground #758

andresgutgon commented Jan 2, 2025 •

edited

Loading

andresgutgon Jan 2, 2025

andresgutgon Jan 2, 2025

andresgutgon Jan 2, 2025

geclos commented Jan 7, 2025

geclos commented Jan 7, 2025

andresgutgon commented Jan 7, 2025

andresgutgon commented Jan 7, 2025

geclos commented Jan 7, 2025

andresgutgon Jan 8, 2025

andresgutgon Jan 8, 2025

neoxelox Jan 14, 2025

andresgutgon Jan 16, 2025

andresgutgon Jan 16, 2025

csansoon Jan 20, 2025

csansoon Jan 20, 2025

csansoon Jan 20, 2025

csansoon left a comment

Handle tool calls in playground #758

Handle tool calls in playground #758

Conversation

andresgutgon commented Jan 2, 2025 • edited Loading

What?

Related issue

Possible affected clients

TODO

Tool calls when running prompt in Batch in Playground 🧠 🧠 🧠 🤯

In SDK

Considerations

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

geclos commented Jan 7, 2025

geclos commented Jan 7, 2025

andresgutgon commented Jan 7, 2025

andresgutgon commented Jan 7, 2025

geclos commented Jan 7, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csansoon left a comment

Choose a reason for hiding this comment

andresgutgon commented Jan 2, 2025 •

edited

Loading