Cost and latency

Models are priced per token, usually quoted per million tokens, with input (what you send) and output (what the model writes) priced separately. Output usually costs several times more per token than input. So both long prompts and long answers cost money, and a big document resent on every turn adds up fast.

input_tokens = 3000
output_tokens = 500
price_in = 3.00    # dollars per million input tokens (a made-up example)
price_out = 15.00  # dollars per million output tokens (also made up)

cost = input_tokens / 1e6 * price_in + output_tokens / 1e6 * price_out
print(round(cost, 4))
print(round(cost * 100_000, 2), "for 100,000 requests")

Waiting time has two parts: the time to the first token (sending, queueing and reading the prompt) and then the time to write each output token. For a long answer the second part dominates, so asking for shorter answers is one of the best speed-ups. Streaming shows the answer as it's written: it takes as long overall, but people see progress straight away.

The levers, roughly in the order to try them:

Use the smallest model that passes your evals for the task, and route harder requests to a bigger one. Trim the prompt to what the task needs. Cap and shape the output ("three bullet points"). Cache: many providers discount a prompt whose beginning is the same as a recent one (prompt caching), so put the fixed parts first; and you can store whole answers to questions you see again. Send jobs that don't need an answer now to a batch interface where one exists, which is usually cheaper. Run independent calls in parallel instead of one after another.

Quality, cost and speed pull against each other, and the trade-off is different for every task. Measure all three on your own examples before and after a change, instead of guessing.