Free guide · Comparing AI tools on the same job
AI Tool Comparison Worksheet
Use this worksheet to compare AI tools on one real job instead of comparing feature lists or impressions.
1. Define the test
TEST JOB Job to be done: Audience: Decision or action the output supports: Required output: Verified facts supplied: Source files supplied: Known missing information: Prohibited claims: Time limit per tool: Review checklist:
Do not use confidential or customer data for an early comparison. Use a fictional or approved test case.
2. Use the same prompt
Create the requested output using only the supplied brief and sources. Before drafting: - list material missing information; - separate verified facts from assumptions; - state which supplied source supports each figure. In the output: - do not invent facts, results or citations; - mark unresolved gaps as [NEEDS INPUT]; - follow the required format and audience. After drafting, return a verification note with every figure used, its source and any correction you made. [PASTE TEST JOB AND APPROVED SOURCES]
If a tool cannot access a source or complete part of the task, record that limitation. Do not quietly give one tool extra help.
3. Record the conditions
| Field | Tool A | Tool B | Tool C |
|---|---|---|---|
| Tool and product name | |||
| Model or mode shown | |||
| Account or plan | |||
| Test date | |||
| Connected apps used | |||
| Files supplied | |||
| Prompt changes | None / explain | None / explain | None / explain |
| Time to usable draft |
Product behavior can change. The account, model and test date are part of the result.
4. Compare the work
Score each item from 0 to 2:
- 0: failed or absent
- 1: partly correct or required material correction
- 2: correct and usable with normal review
| Criterion | Tool A | Tool B | Tool C | Evidence or notes |
|---|---|---|---|---|
| Used the supplied facts accurately | ||||
| Identified missing information | ||||
| Did not invent facts or sources | ||||
| Corrected itself when challenged | ||||
| Traced figures to sources | ||||
| Followed audience and format | ||||
| Produced a clear decision or action | ||||
| Required little manual correction | ||||
| Total out of 16 |
The total is a discussion aid, not a universal ranking. A failure on a critical criterion can outweigh a higher total.
5. Worked fictional test
Job: Draft a one-page recommendation on whether to extend the Northstar onboarding pilot.
Supplied facts: 18 invited, 14 completed, 3 needed follow-up, time saved not measured.
Known gap: owner of the extension is not named.
| Observation | Tool A | Tool B |
|---|---|---|
| Facts | Used all figures correctly | Changed 14 of 18 to “nearly 80%” without showing the calculation |
| Missing information | Asked for the owner before drafting | Assigned the project manager without evidence |
| Unsupported claims | None | Claimed the pilot improved efficiency |
| Source handling | Named the tracker beside each figure | Referred to “the data” without a source location |
| Correction | Not needed | Removed the efficiency claim after challenge |
Decision: Tool A is the safer fit for this source-grounded brief. This result applies to this job, account, model and test date. It does not prove Tool A is better for every task.
6. Correction test
Use the same challenge in each tool:
Audit your answer against the supplied facts and sources. List: 1. every factual claim, 2. its source, 3. any assumption or inference, 4. anything you should correct. Do not defend the first answer. Correct it when the evidence requires a change.
Record whether the tool finds the same issues you found and whether its correction introduces a new error.
7. Choose by the job
Chosen tool: Job it is suited for: Critical strength observed: Critical limitation observed: Human checks still required: Account, model and test date: When to retest:
Final check
- □ Every tool received the same job, sources and constraints.
- □ I recorded any prompt change or extra help.
- □ I checked facts, gaps, corrections, sources and time.
- □ I recorded the model, account and date.
- □ I chose for this job, not for every possible job.
- □ A human still reviews the final work.
The rule to keep: Compare tools on the work you actually need, under conditions you can repeat.