Bedrock publishes runtime metrics to Amazon CloudWatch per model, including Invocations, InvocationLatency, InputTokenCount, OutputTokenCount, InvocationThrottles, InvocationClientErrors and InvocationServerErrors. Alarms on throttles and errors catch trouble early; token counts show cost trends. Guardrails, agents and knowledge bases publish their own metrics too.
Add your own metrics for what matters to the business: answers rated helpful, hallucinations caught by grounding checks, tasks an agent finished, escalations to a person, cost per conversation. CloudWatch dashboards (or Amazon Managed Grafana) put operational and business metrics side by side.
Model invocation logging writes each request and response, with token counts and metadata, to CloudWatch Logs, S3 or both. CloudWatch Logs Insights can query them: the slowest prompts, the longest replies, which prompt version produced bad output. Your app should log its own context too: user or session id, prompt template and version, retrieved document ids, guardrail result.
Trace across services. AWS X-Ray follows one request through API Gateway, Lambda, retrieval and the model call, so you can see which step is slow or failing. For agents, record each tool call: which tool, how long it took, whether it failed, and how many steps the whole task needed. A sudden jump in steps or a tool's error rate is an early sign that something changed.
Watch for problems unique to generative AI. Response drift (answers changing in length, tone or content after a model or prompt update), token bursts (a loop or abuse driving usage up), rising hallucination rates, and retrieval going stale. Run a golden dataset (questions with known good answers) on a schedule and alarm when scores fall. CloudWatch anomaly detection can learn normal token usage and flag spikes; AWS Cost Anomaly Detection flags unusual spending.
Watch the vector store as well: query latency, index size, failed ingestion jobs and how fresh the data is. A knowledge base sync that quietly failed last week looks like the model getting dumber.
Which CloudWatch metric shows Bedrock is rejecting calls because the account is over its quota?
| Invocations / InvocationLatency | How many calls / how long they take. |
|---|---|
| Input/OutputTokenCount | Tokens in and out: usage and cost. |
| InvocationThrottles | Calls rejected for quota. |
| Logs Insights | Query invocation and app logs. |
| X-Ray | Traces a request across services. |
| Golden dataset | Known questions and answers run on a schedule. |
| Cost Anomaly Detection | Flags unusual spending. |