Evaluating models, RAG and agents

Generative output can't be scored with a simple accuracy number. Judge it on several measures: relevance (does it answer the question), factual accuracy or correctness, faithfulness to the sources, completeness, consistency (same question, same answer), fluency and tone, and harmfulness. Pick the ones that matter for the use case and score them on a fixed test set.

Amazon Bedrock Evaluations runs these for you. Automatic evaluation uses built-in tasks and metrics on your dataset or a built-in one. LLM-as-a-judge uses a second model to grade each answer against criteria (correctness, helpfulness, harmfulness and others, or your own), which scales far beyond human review at a fraction of the cost. Human evaluation sends answers to your own reviewers through a work team for things only people can judge. You can compare several models or prompt versions side by side.

RAG needs two kinds of measurement. Retrieval: did the right chunks come back (context relevance, context coverage, and how long it took)? Generation: is the answer correct, complete and faithful to what was retrieved? Bedrock can evaluate a knowledge base on both. If retrieval scores are low, fix chunking, embeddings or search; if retrieval is good but answers are bad, fix the prompt or model.

Agents are judged on outcomes and paths: the share of tasks completed, whether the right tools were called with the right parameters, how many steps it took, cost and time per task, and whether its reasoning holds up when you read the trace.

Ship changes carefully. A/B testing sends part of the traffic to a new model or prompt and compares results; a canary sends a small slice first and widens it if metrics hold. Regression tests run the full test set on every change, and an automated quality gate in the deploy pipeline blocks a release when scores fall. After launch, keep collecting real feedback (thumbs up and down, ratings, annotations through tools such as SageMaker Ground Truth) and feed it back into the test set.