Cloud AI platforms

So far the lessons have talked to "the model" as if it were one API. In companies the model is often reached through a managed service in the cloud they already use. AWS Bedrock, Azure OpenAI (part of Microsoft's Azure AI services) and Google Vertex AI are the best-known examples. The names and features change often, so this lesson sticks to the ideas they share; check your provider's docs for the details.

A managed model service runs models for you inside a cloud provider. You turn on access to the models you want in your account, then call them with the cloud's own SDK and credentials. Many offer models from several companies behind one API, so trying another model can be a change of model id rather than a new integration.

# the shape of a call through a cloud platform (names vary by provider; check its SDK docs)
client = cloud_sdk.client("model-runtime", region="eu-west-1")   # credentials come from the app's role, not a pasted key
reply = client.invoke(
    model="a-model-id-you-have-enabled",
    messages=[{"role": "user", "content": "Summarise this clause in one sentence: ..."}],
    max_tokens=200,
)

Why go through a cloud platform rather than sign up with a model company directly? Mostly it's the rest of the cloud. Data handling: requests are processed under the cloud contract and in a region you choose, which matters for data-residency rules (read the provider's data terms to see exactly what's stored, logged or used for training). Access: the cloud's identity and access management (IAM) decides which apps and people may call which models, with short-lived credentials instead of an API key pasted into config. Networking: private endpoints keep traffic on the cloud's private network instead of the public internet. Billing and audit: usage lands on the same bill, budgets and audit logs as everything else.

There are costs too. A new model can reach a cloud platform later than its maker's own API, or not at all; some features and models are only in some regions; and the more managed pieces you use, the harder it is to move.

Managed knowledge bases do RAG for you: point one at a storage bucket of documents, and it splits them into chunks, embeds them, keeps the vectors in a vector store and gives you a retrieve (or retrieve-and-answer) call. These are the same steps as the RAG course, run by someone else. You still choose the chunking and the sources, and you still have to evaluate whether the right passages come back.

Guardrails are configurable filters on what goes in and out: block topics you don't serve, filter harmful content, mask personal data such as account numbers, and on some platforms check that an answer is supported by the retrieved text. They catch whole classes of problems, but they don't make answers correct. Your evals still decide that.

Agent runtimes host the agent loop from the agents course: you describe the tools (often as API definitions or functions), and the platform runs the loop, keeps the session's history and calls your tools. Tracing each step is what makes these debuggable, so turn it on.

Capacity comes two ways. On-demand: pay per token, share capacity with other customers, and live with rate limits (quotas) per account and region. Provisioned (reserved) throughput: pay for dedicated capacity by the hour or month, usually with a commitment, for steady volume or when you need predictable speed. Many platforms also run batch jobs at a lower price when an answer can wait hours.

tokens = 3_000_000_000             # a month's input and output tokens (made-up)
price_per_million = 2.00           # on-demand, dollars (made-up)
provisioned = 5_000                # reserved capacity, dollars a month (made-up)

on_demand = tokens / 1_000_000 * price_per_million
print(on_demand, on_demand > provisioned)

The arithmetic is only part of it. Provisioned capacity is paid for whether you use it or not, so it suits steady, heavy traffic; spiky or small workloads are usually cheaper on demand. And the commitment ties you to one model for its term, so check that the model will still be the one you want.