Which tool to choose
To test prompts you don't need anything special: the assistant you use and somewhere to note cases and results (a spreadsheet, a note) are enough. If you make prompts for a repeated task — descriptions, customer replies, summaries — the investment of half an hour in a testing method pays you back every day. If it's a throwaway prompt, testing it thoroughly isn't worth it: you tweak it on the fly and move on. The method is for the prompts you'll reuse.
How to do it
The problem with improving prompts by eye: you change something, the example in front of you gets better, you conclude that the prompt is better. But maybe you made it worse on three cases you didn't try. Without fixed test cases, you're not improving, you're guessing.
- Gather five or six representative real cases, including the hard ones and the edge cases.
- For each, write what a good answer would be: your yardstick for judgment.
- Run the current prompt on all the cases and note where it fails.
- Change one single thing in the prompt, try again on the same cases, compare.
- Keep the version that does better on the whole set, not the one that wins on a single case. Repeat until it holds on all of them.
The operational syntax to get the AI to help you with the comparison:
I'm giving you a task, two versions of the prompt and three test cases with the expected answer. Run each version of the prompt on each case, then tell me which version comes closest to the expected answers and on which cases each one fails.
Task: [...]
Prompt A: [...]
Prompt B: [...]
Test cases with expected answer:
1. [input] -> [right answer]
2. [input] -> [right answer]
3. [input] -> [right answer]
After the comparison, change one more single thing on the winning version and try again. The single-change rule is what tells you which modification worked; if you change three at once, you'll never know which one made the difference.
A concrete example
Valentina writes a prompt to generate replies to customer messages. The first version was fine on the easy complaint she had in mind, but when she actually used it, it stumbled on the thorny cases. She builds six test cases: two easy, two hard, two edge (an angry customer, a request outside the catalog). She tries the current version: it passes four out of six. She changes one line, tries again: five out of six. She changes again: six out of six. Now she knows the prompt holds up even on the cases that previously made her look bad in front of real customers.
When it does NOT work (and how to fix it)
If every modification fixes one case and breaks another
You're oscillating because the test cases pull in opposite directions and the prompt is trying to satisfy them all. Check whether the cases are compatible: sometimes two requests really are in conflict and you have to decide which takes priority. Other times you need to add a rule that distinguishes the cases instead of one that treats them the same.
If you can't say which answer is "right"
Without a yardstick, you can't test. Before touching the prompt, write for each case what aspect a good answer must have, even just in two lines. Defining the target is half the work: often the prompt wasn't improving because you didn't even know yourself what you wanted.
If the improvements are minimal and not worth the effort
Maybe the prompt is already good enough for the purpose. Systematic testing is for the prompts you use a lot and where errors cost; for the others, "it works" is enough. Recognizing when to stop is part of the method: the endless optimization of a rarely used prompt is wasted time.
A tip from someone who actually uses it
The cases that teach the most are the hard ones, not the easy ones. It's comfortable to test the prompt on the example that always works, but it's useless: any version passes there. Fill your test collection with the thorny cases, the ones where the first version of the prompt looked bad. A prompt that holds up on the hard cases also holds up on the easy ones; the reverse isn't true.
Frequently asked questions
How many test cases do I need?
Few and well chosen beat many and similar. Five or six cases that cover the easy, the hard and the edges say more than twenty variants of the same scenario. The quality of the coverage matters more than the number: each case must put something different to the test.
Can I use the AI itself to judge which prompt is better?
Yes for the rough comparison (which version comes closest to the expected answers), but the yardstick of "right" stays yours, because only you know the real purpose and the audience. Use the AI to run and compare quickly; keep the final verdict on the cases that matter for yourself.
Isn't it faster to tweak the prompt on the fly when it gets something wrong?
For throwaway prompts, yes. For the ones you reuse, no, and it's the misunderstanding that wastes the most time: tweaking on the fly fixes the symptom of the moment and silently breaks the other cases, until you have a prompt that works "sometimes" without you knowing when. Test cases turn "it seems better to me" into a verifiable fact.