Retries and failure handling

Things fail: a request times out, the provider says you're sending too many requests (HTTP 429), a server has a hiccup (500, 502, 503), or the model's reply doesn't parse. The first question is whether trying again could help.

Transient failures (timeouts, 429, most 5xx errors) often succeed on a second try. Permanent ones (a malformed request, a 400, a missing permission) will fail the same way every time, so retrying just wastes time and money. Retry the first kind, a limited number of times, and fail the second kind straight away.

Back off between retries, doubling the wait each time (exponential backoff), and add a random amount (jitter) so a thousand clients that failed together don't all retry in the same instant. If the response says how long to wait (a Retry-After header), wait that long.

import random

random.seed(1)
base = 1.0
for attempt in range(4):
    wait = base * 2 ** attempt
    jittered = wait * random.uniform(0.5, 1.0)
    print(f"attempt {attempt + 1}: wait up to {wait:.0f}s, this time {jittered:.2f}s")

Retrying an action is dangerous when the first try might have worked and only the reply was lost: retry a payment and you might charge twice. Give each action an idempotency key (a unique id for this one operation) and have the receiving system ignore a second request with the same key. Many payment and messaging APIs support this.

When the model's output fails validation, retry once or twice with the error included ("The JSON was missing due_date"). If it still fails, try a stronger model or send it to a person; don't loop.

Whatever can't be done automatically goes somewhere visible: a review queue or a dead-letter queue with the input, the error and the attempts so far. Silent failures are the worst kind.