Generating memos and reports

Credit memos, investment memos and portfolio reports follow a set structure, and readers expect it: the same sections in the same order, every time. So don't ask a model to "write a memo". Build it from a template, one section at a time.

For each section, the template says what goes in it and which facts it needs. The system retrieves or looks up those facts (extracted fields with their provenance, passages from the documents), then asks the model to write that section from them and only them, citing each fact it uses. A short section from the right ten facts beats a whole memo from a pile of pages.

Compute the numbers in code. Ratios, totals, growth rates and anything else that's arithmetic should be worked out by your program from the extracted values and handed to the model as facts. Models are good at prose and unreliable at arithmetic, and a memo's numbers are what readers act on.

noi = 1_840_000
debt_service = 1_404_580
dscr = round(noi / debt_service, 2)
print(dscr)

Then check the draft against the facts. Every number in the text should match one in the facts it was given; anything else was made up or mangled. A simple check finds the numbers in the draft and looks each one up:

import re
facts = {"noi": "1,840,000", "dscr": "1.31x", "ltv": "68%"}
draft = "NOI of $1,840,000 supports a DSCR of 1.35x at a 68% LTV."
numbers = re.findall(r"\d[\d,.]*\d|\d", draft)
unsupported = [n for n in numbers if not any(n in v for v in facts.values())]
print(unsupported)

Reviewers then edit. Show the draft section by section with sources beside it; record each edit (who, what, when) rather than overwriting; and let a reviewer lock a section they've approved so that regenerating other sections never changes it. The edits are useful data too: a reviewer who keeps rewriting the same kind of sentence is telling you what to fix in the template or the prompt.