How to Compare AI Tools on One Real Task

The most useful AI comparison begins with a piece of work you can judge. A list of features might help you find candidates, but it will not tell you whether their answers fit your process. Choose a recurring task, write down what a good result requires and give each candidate a fair attempt at that assignment.

The example below uses a fictional customer message. It is a suggested evaluation method, not a report of tools tested for this article. You can adapt it to an approved tool and suitable data in your own organization. Start with a task that has a small consequence if the draft is wrong and a source of truth you can check directly.

Define the result before opening the tools

Suppose an equipment rental business needs help drafting booking-change replies. The useful outcome is a short acknowledgment that preserves the customer's request and asks for missing information. Sending a message, changing the reservation and promising availability remain outside the exercise.

Write that boundary at the top of the comparison brief. Then describe the expected answer in ordinary language. It should use the supplied dates accurately, acknowledge the requested change, avoid confirming availability and stay within a reasonable length. A teammate who knows the task should be able to read the brief and understand how to assess the output without knowing which product produced it.

This early agreement matters because a fluent answer can change your idea of success after you see it. If one response is unusually warm, you may start rewarding warmth even though the original problem was incorrect booking details. Keep the required facts and restrictions visible throughout the review.

Create a small, controlled input

Use this fictional message: “Please move booking R-204 from Tuesday 12 May to Thursday 14 May. We still need two folding tables. Can pickup be after 3 p.m.? Let me know if the new date is available before you change anything.”

Add a source note for the evaluator: the availability calendar is not included, no change has been authorized and the pickup policy is unknown. Those absences are part of the task. The tool should not fill them in with plausible business rules. A good draft can acknowledge the request and say that the details need checking.

Keep the original input in a separate file or document you control. Do not casually change a date halfway through testing. If you discover that the example is ambiguous, revise it, give it a new version label and rerun the affected attempts. Otherwise you will be comparing different assignments while treating them as the same test.

Write the common prompt

A starting prompt could ask: “Draft a customer reply under 100 words using only the message and source note. Acknowledge the requested change. Do not confirm availability, pickup policy or a completed booking change. Identify what a staff member must check before sending.”

Give each tool that same substantive instruction. If a product requires an uploaded document instead of pasted text, record the adaptation. Differences in input format may be part of the buying decision, but they should be visible. Keep any product-specific improvements in a separate round so the first comparison stays interpretable.

Decide on the follow-up allowance in advance. You might permit one clarification prompt per attempt. Record its exact wording and the reason it was needed. An answer that becomes useful after repeated coaching may still be valuable, but the coaching is work and belongs in your assessment.

Use a rubric with hard stops

For this example, check four things: factual fidelity, restraint, usefulness and editing effort. Factual fidelity asks whether the dates, reference and quantities match the input. Restraint asks whether the draft invents a policy or makes a commitment. Usefulness asks whether the customer would understand what happens next. Editing effort records the corrections needed before a staff member could use the draft.

Treat an invented confirmation as a hard stop for this task. Do not let a strong tone score cancel it out. You can still record the parts that worked, but the attempt should not count as ready for the intended workflow. This keeps the comparison tied to the consequences of the actual job.

NIST's Generative AI Profile describes the risk of confident but false output. In this exercise, an invented availability statement is the kind of error the review is designed to catch. The rubric is a practical proposal for this scenario, not a NIST scoring system.

Keep a failure log you can read later

For each attempt, save the tool name, displayed model or mode when available, access tier, date, input version and complete output. Add the reviewer, required corrections and any access problem. If an answer fails, describe the failure precisely: “Changed two tables to three” is better evidence than “bad at details.”

Use separate entries for separate runs. Repeating a task can reveal variation, but a handful of attempts is still a small sample. Avoid presenting a tiny pass rate as a reliable forecast of future performance. Its immediate value is helping you find failure patterns worth investigating before adoption.

If possible, ask a second person to score the answers without product names attached. When reviewers disagree, inspect the rubric. Perhaps “friendly” is too vague, or one reviewer assumed a policy the other did not know. Resolving that disagreement can improve the test even before you choose a tool.

Compare the whole job

Record how long it takes to prepare the input, obtain the answer and review it. Keep setup effort separate from repeated-use effort. You may tolerate a longer initial configuration if the recurring task becomes easier, but the distinction should be explicit. Avoid turning one timed attempt into a sweeping productivity claim.

Also note practical constraints: can the intended users access the required plan, can the output be moved into the normal workflow and does the organization permit the proposed data use? Read the provider's current documentation for the exact product and account configuration. A feature available in one interface should not be assumed to exist everywhere under the same brand.

The wider NIST AI Risk Management Framework is useful context for structured evaluation. This small exercise is a selection aid. Approval for a consequential workflow requires the organization's own review of its data, users and possible failures.

End with a decision and a next test

The conclusion might be modest: one tool deserves a longer pilot, another needs too much editing for this task and a third cannot be assessed because the required access was unavailable. “No suitable candidate yet” is also a useful outcome. Preserve the evidence so the decision can be revisited when the task or product changes.

Before leaving the comparison, write the next question. Perhaps the promising tool needs to handle longer messages, a different language or contradictory dates. Build a second case around that specific uncertainty. A series of small, well-described tests gives you a more useful record than a single winner badge detached from the work it was meant to support.