Cost and performance

You pay for tokens in and tokens out, so cost follows prompt size, reply size and the model's price. Cut input: send only the retrieved chunks that matter (fewer, better ones after reranking), trim or summarize old conversation turns, and remove boilerplate from system prompts. Cut output: set maxTokens, and ask for short answers or just the JSON fields you need. Track tokens per request with CloudWatch metrics so you see where they go.

Prompt caching saves on repeated prefixes. If many requests start with the same long text (a big system prompt, a manual, tool definitions), Bedrock can cache that prefix for supported models: you mark a cache checkpoint, and later requests that share the prefix read it from the cache at a lower cost and latency. It suits long, stable context followed by a short, changing question. Check which models support it and how long the cache lasts.

Response caching skips the model entirely. Exact-match caching hashes the normalized request and returns a stored answer for an identical one (Amazon ElastiCache or DynamoDB). Semantic caching embeds the question and returns a stored answer when a previous question was close enough in meaning; it catches rephrasings but needs a careful similarity threshold. Pre-compute answers to predictable questions ahead of time, and cache at the edge with CloudFront where answers are public.

Pick the cheapest model that's good enough. Measure quality and cost per request for a few models on your own test set, route routine traffic to a small model, and keep the large one for hard requests. Use batch inference for offline work, and Provisioned Throughput only where steady use makes the reservation pay off.

For latency: stream replies, run independent calls in parallel (retrieval and a classification at once, say), keep prompts short, use smaller models, use latency-optimized inference where Bedrock offers it for your model, and put the app in the same Region as the model. Benchmark time to first token and total time separately; streaming improves the first, smaller prompts and models improve both.

Tune the model's parameters for the job: low temperature and a tight maxTokens for extraction, more room for creative writing. Set limits on throughput with planning: estimate tokens per minute at peak from requests times tokens, compare with your quotas, and request increases or add cross-Region inference before launch, not after.