← All lessons
Agents and workflows · Lesson 5 of 7

Keeping a human in the loop

Automation doesn't have to be all or nothing. The best systems decide, item by item, whether to act alone, ask for approval, or hand the whole thing to a person.

Require approval for actions that are irreversible, expensive or sensitive: sending money, emailing many people, deleting data, anything with legal weight. The system prepares the action; a person clicks to approve or edits it first.

Send items for review when the system is unsure in a way you can measure: validation failed, sources disagree, a value falls outside normal ranges, the document type wasn't recognised, or the model abstained. A model's own confidence score is a weak signal on its own; checks that can fail are better.

Make the reviewer fast. Show the proposed action or extracted values next to the evidence (the highlighted part of the page, the cited passage), what checks failed and why, and one-click approve, edit and reject. A review screen that makes people dig is a review screen people rubber-stamp.

items = [
    {"id": 1, "checks_failed": 0, "amount": 120},
    {"id": 2, "checks_failed": 1, "amount": 80},
    {"id": 3, "checks_failed": 0, "amount": 25_000},
]
for it in items:
    if it["checks_failed"]:
        route = "review"
    elif it["amount"] > 10_000:
        route = "approve"
    else:
        route = "auto"
    print(it["id"], route)
Prints
1 auto
2 review
3 approve

Check the automatic path too. Sample a share of the items that went through without review, have a person look, and track the error rate. If it creeps up, tighten the rules; if it stays low for months, you may be able to automate more.

Every correction is data. Store what the system proposed and what the person changed, and add the interesting cases to your eval set.

Practice

Using the routing rules from the lesson, what does this print?

it = {"checks_failed": 0, "amount": 9_999}
print("review" if it["checks_failed"] else "approve" if it["amount"] > 10_000 else "auto")
More info