omarcevi.dev / workshops / deploy-agents / Production checklist

Production readiness checklist

Agree on these items with the customer before the agent goes into production. Each item refers to the step where you saw it.

Identity and access

  • Each agent has its own Agent Identity. You grant roles to its principal://, not to a shared service account (step 2).
  • Deployments run from CI with a dedicated deployer identity. Developers do not have setIamPolicy on production projects (step 2).
  • Only a trusted, authenticated front end can call query/streamQuery. The user id that it sends separates the sessions and memories of each user (steps 2, 3).
  • Tools that act for a person use the token of that person (3-legged OAuth through Auth Manager). The tool server verifies the token (step 4).
  • Secrets are in Secret Manager. They go to the agent through --secrets, never through .env (step 2).

Governance

  • Each destination has an entry in Agent Registry. The owner of each tool examines its tool annotations (steps 1, 4).
  • An egress gateway controls all calls from the agent. The security team owns the policies: organization guardrails, project rules for each agent, and deny rules for sensitive systems (step 4).
  • New policies run in DRY_RUN mode first. You read the denial logs before you enforce the policies (step 4).
  • Model Armor and semantic governance policies (where available) stop prompt injection and requests for data that the model must not get (step 4).
  • You examined the organization policies: iam.managed.disableAccessPolicyBindings, iam.allowedPolicyMemberDomains and region restrictions.

Data

  • You chose the region for data at rest. The data protection officer of the customer agreed to the model endpoint: global, multi-region or regional (steps 1, 2).
  • Memory topics are narrow and memories have a TTL. Requests for deletion (KVKK, GDPR) go to Memory Bank too (step 3).
  • The customer agreed to record prompts and answers in logs. These logs go to a restricted log bucket with a retention period (step 5).

Operations

  • A dashboard and alerts show turn latency, tool error rate and blocked calls. Refused calls return HTTP 200, so set alerts on logs-based metrics (step 5).
  • Model calls have timeouts and retries. If traffic is high or latency is important, you have a plan for capacity (Provisioned Throughput).
  • The minimum instances, maximum instances and concurrency come from a load test, not from the default values.
  • There is a rollback path. Agent Runtime keeps revisions, and you know how to deploy the previous version again.

Quality

  • The business and the engineers own the eval set together. They agree on a threshold for each metric, and a gate in CI stops deployments that fail (step 6).
  • An Online Monitor runs with a sampling rate and a limit for each run that fit the budget (step 6).
  • The monitor scores your traffic. To make sure, count the INSUFFICIENT_DATA entries in its log. Today the monitor skips turns that have more than one agent (step 6).
  • Each incident and each low monitor score becomes an eval case before you release the fix (step 6).