Vertex AI Agent not returning toolCall in response

Question

I am trying to use Function Tools in Agent Builder. According to the Vertex AI documentation here, I should get back toolCall in the response when the agent thinks that I am calling a tool.

{
  "queryResult": {
    "text": "what is the weather in Mountain View",
    "languageCode": "en",
    "responseMessages": [
      {
        "source": "VIRTUAL_AGENT",
        "toolCall": {
          "tool": "<tool-resource-name>",
          "action": "get-weather-tool",
          "inputParameters": {
            "location": "Mountain View"
          }
        }
      }
    ]
  }}

I have tried this the following way:In agent builder playbook, I type in how is the weather. I can see in the network request from dialogflow.clients6.google.com/v3alpha1/ that it contains toolCall and tells me which function is called.The response with playbook:

  "queryResult": {
    "text": "how is the weather in Stockholm?",
    "languageCode": "en",
    "responseMessages": [
      {
        "source": "VIRTUAL_AGENT",
        "toolCall": {
          "tool": "projects/geoconxaiservices/locations/eu/agents/2e495876-b080-4755-9f98-9962ab0cc873/tools/a11594cb-50f5-42cf-9eba-06728ee91e92",
          "action": "getWeather",
          "inputParameters": {
            "location": "Stockholm, Sweden",
            "unit": "celsius"
          }
        }
      }
    ],

When I use the integration with Dialogflow Messenger, it calls dialogflow.googleapis.com/v3/ and I don't get the toolCall in the response.

  "queryResult": {
    "text": "how is the weather in Stockholm?",
    "languageCode": "en",
    "responseMessages": [
      {}
    ],

But it will also fail on following queryInputs since it is waiting for the output from my client side code execution. I get "message": "com.google.apps.framework.request.BadRequestException: Session is waiting for tool call result of tool, and action, getWeather.". I have tested the same flow in both playbook and df messenger (with the same agent) and it doesn't seem that they work the same way. Does anyone know why there is a difference in behavior and why isn't toolCall returned to df messenger?

Don't have a solution but I have created an issue on github: github.com/googleapis/google-cloud-python/issues/12749 — BinarSkugga
– BinarSkugga, Commented May 29, 2024 at 19:53

Dev Bitcot · Accepted Answer · 2024-06-05 10:30:25Z

0

first of all this is a demo function which you're trying to use. this function has no server. if you want to use the function tool then first deploy your api in any server and put your API's YAML schema in openapi tool.

answered Jun 5, 2024 at 10:30

Dev Bitcot

1

Sign up to request clarification or add additional context in comments.

Comments

Dan Bush · Accepted Answer · 2024-09-14 14:40:10Z

0

It works in the v3 beta version of the Java client.

import com.google.cloud.dialogflow.cx.v3beta1.*;

answered Sep 14, 2024 at 14:40

Dan Bush

11 bronze badge

2 Comments

Brian A. Henning Over a year ago

Please include example usage and response, so that the community can see how your answer solves the OP's problem.

Community Over a year ago

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Collectives™ on Stack Overflow

Vertex AI Agent not returning toolCall in response

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related