Deploying and serving models

Bedrock has several ways to pay for and run inference. On-demand charges per input and output token with no commitment, which suits most apps and spiky traffic. Batch inference takes a file of prompts in S3 and writes results back to S3 later, for work nobody waits on, like tagging a million product reviews overnight; for many models it costs less than on-demand. Provisioned Throughput reserves a set amount of capacity (in model units) for a model, for steady high volume that needs guaranteed throughput, and is also the usual way to serve some customized models. Bedrock has added service tiers too (such as Priority and Flex) that trade price against how quickly requests are served; check which apply to your model.

Amazon SageMaker AI hosts models you bring or fine-tune, including open-weight models from SageMaker JumpStart. Its endpoint types: real-time (always on, low latency), serverless (scales to zero, CPU only, so not for large language models that need GPUs), asynchronous (requests queue and results land in S3, for big payloads and long processing, and it can scale to zero), and batch transform (whole datasets offline).

Large language models differ from classic ML models. The weights can be tens of gigabytes, so they need GPU instances with enough memory, and loading takes minutes. Serving containers built for LLMs (such as SageMaker's large model inference containers) shard a model across GPUs, batch requests together and stream tokens. Smaller models or quantized versions need less memory and start faster, which is often the better trade.

Lambda is good glue in front of Bedrock: on-demand, no servers, and fine for calls that finish within its 15-minute limit. Long jobs belong in Step Functions, SQS-driven workers, Bedrock batch inference or SageMaker asynchronous endpoints.

Choose by the constraint. Unpredictable traffic and no ops: Bedrock on-demand. Overnight millions: batch inference. Steady peak needing guaranteed capacity: Provisioned Throughput. Custom open-weight model with special serving needs: a SageMaker AI endpoint. Large inputs that take minutes: SageMaker asynchronous inference.