01The idea
When an AI feature changes its behaviour, it's usually because someone edited a prompt. We wanted a way to see what that edit did to real answers before it reached a single user.
Prompt diff runs a fixed set of questions against the old and the new prompt and shows the answers side by side, with the differences marked like a code review.
02How it's built
A small Python CLI reads a test set, calls the model with both prompts and scores each pair against simple checks you write yourself: must mention, must not mention, length, format. The report is a static HTML page you can attach to a pull request.
- Test sets live in the repo, next to the prompts
- Checks are plain Python functions, not a new language
- Every run is cached, so re-runs are cheap
- An HTML report that fits into normal code review
1# a check is just a function that passes or fails2@check3def cites_a_source(answer: str) -> bool:4 return "[source:" in answer56@check7def stays_short(answer: str) -> bool:8 return len(answer.split()) < 120910run("prompts/support_v3.txt", "prompts/support_v4.txt",11 cases="tests/support.yaml")03What we learned
Most prompt regressions are boring: an answer got longer, a disclaimer disappeared, a format broke. Simple checks catch those. The genuinely hard cases still need a person to read the diff.
- Version prompts like code
- Cheap checks first, human review second
- Keep the test set small enough to read
Prompt diff is now a required step before any AI change we ship for a client.
