omarcevi.dev / workshops / deploy-agents / 5 · Observe

Step 5 · Observe

5 min

Follow one leave request in Cloud Trace, from the concierge to the HRIS. Open a dashboard that shows the blocked calls.

checkpoint branch step-5-observe

How it works

One leave request as a trace waterfall. The span invoke_workflow concierge contains the span invoke_agent concierge, with a model call and transfer_to_agent. Then the span invoke_agent leave_agent contains an MCP tools/list request, a model call, execute_tool get_leave_balance, a model call, execute_tool submit_leave_request and a last model call. Tool calls take milliseconds. Model calls take seconds. On the right, the diagram shows where the telemetry goes. Spans go to Cloud Trace through the telemetry API. OpenTelemetry log records go to Cloud Logging, with the model input and output events and the workplace.tool_blocked audit event. Agent Runtime metrics, ADK metrics, logs-based metrics and the dashboard are in Cloud Monitoring. The gateway writes a log with the allow or deny decision for each request.
One turn from start to end. Each agent, model call and tool call is a span.

You get traces without extra code. agents-cli deploy sets GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true, and ADK sends OpenTelemetry spans to Cloud Trace. There is an invoke_workflow span for the turn and an invoke_agent span for each agent. There is a call_llm/generate_content span for each model call and an execute_tool span for each tool call. A failed tool call has the attribute error.type.

On Agent Runtime, logs need OpenTelemetry. Output from print() and from Python logging in the container does not get to Cloud Logging. OpenTelemetry log records get to Cloud Logging. The agent writes its audit event (workplace.tool_blocked) as an OpenTelemetry log record, and it also sends Python warnings this way. Setup also enabled prompt and answer events (EVENT_ONLY) and the instructions of the agent on the call_llm spans (ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=true; agents-cli disables it by default). The Online Monitor of step 6 reads the two.

Metrics: Agent Runtime publishes the request count and the latency. ADK publishes the durations of tools and agents. Two logs-based metrics count the blocked calls, from the audit event of the agent and from the log of the gateway.

Run

If you did not book leave in step 3, make one leave request. Wait one minute for the trace. Then run the script:

agents-cli run --url "$AGENT_URL" --mode adk "Book next Monday off as a half day and tell me how many leave days I have left."
./scripts/observe.sh       # creates the dashboard, prints the newest leave-request trace

Open the two links that the script shows: the trace in Trace Explorer and the Workplace Assistant (workshop) dashboard. The console also shows the traces of each session in Deployments → workplace-assistant → Traces.

Expected result

0.00s 17.20s invoke_workflow concierge 0.05s 2.70s invoke_agent concierge 0.10s 2.50s call_llm 0.10s 2.40s generate_content gemini-3.5-flash 2.60s 0.01s execute_tool transfer_to_agent 2.70s 14.40s invoke_agent leave_agent 2.75s 0.90s MCP send tools/list 3.70s 3.10s call_llm 6.90s 0.13s execute_tool get_leave_balance 7.20s 4.00s call_llm 11.30s 0.15s execute_tool submit_leave_request 11.50s 5.60s call_llm https://console.cloud.google.com/traces/explorer;traceId=…

Read the trace as a profile. The tool calls take milliseconds, and the model calls take seconds. Thus, most of the latency and cost comes from the model calls. The dashboard has five tiles: requests per minute, turn latency (p50 and p95), tool error rate, blocked calls and gateway denials. It shows the tool error rate for each tool and the blocked calls for each reason.

Check

./scripts/check.sh 5
✔ sub-agent spans: ['invoke_agent leave_agent'] ✔ tool spans: ['execute_tool get_leave_balance', 'execute_tool submit_leave_request'] ✔ dashboard exists ✔ logs-based metric workplace_tool_blocked exists ==> Step 5 checks passed.
If this fails
No trace matching …
A trace gets to Cloud Trace one to two minutes after the turn. If you booked leave more than three hours ago, book leave again.
The dashboard tiles show “No data”
The tiles show data when traffic arrives. A logs-based metric counts only the log entries that come after you create the metric. For this reason, setup created the metrics early.
There are no workplace.tool_blocked entries
These entries occur only after a refused call (step 4). Ask for a salary band again.
The p95 request latency is very high
The request latency of Agent Runtime includes all of the streamed response. The turn latency tile from ADK gives a better measurement.