Briefing a team on an AI feature

The situation

A Bengaluru SaaS company sells helpdesk software to mid sized retailers. Its customers keep asking for one thing: when a support ticket arrives, suggest a reply the agent can edit and send. The product manager writes a one line brief, "Add AI suggested replies to tickets", and asks an outside team for an estimate.

Every team that reads that line will picture a different feature. One will build a single prompt that answers from the ticket text. Another will search the customer's help articles first and quote them. A third will want to fine tune a model. The estimates will differ by hundreds of hours, and nobody will know at the end whether the result is good, because nobody said what good means.

Write the brief before work starts. The same approach suits any feature where a language model produces text, labels or decisions: suggested replies, summaries, document extraction, search over your own content, or an agent that takes steps on a user's behalf.

Why an AI brief needs more than a normal one

A normal feature gives the same output for the same input. You can write "when the user clicks Save, the record appears in the list" and test it once. A language model can give two different answers to the same ticket on two different days, and both can be fine. It can also give a confident answer that is wrong.

So the brief has to describe a range of acceptable behaviour, and the acceptance test has to measure that range over many examples. Most of the work is collecting real examples and agreeing who judges them.

What to put in the brief

The task, in one paragraph

Describe the job the feature does as you would to a new support agent. For the helpdesk: "Read the customer's ticket and the last three messages in the thread. Draft a reply in the customer's language that answers the question using our published help articles. If the articles do not cover it, draft a short holding reply and flag the ticket for a person."

Name what the feature must never do. For example: promise refunds, quote prices, or mention another customer.

The inputs

List exactly what the model will see: the ticket text, the thread, the help articles, the customer's plan, the product name. Anything outside this list is unavailable to it. This matters for quality and for privacy, and it shapes the cost.

Examples of good and bad outputs

Collect twenty to fifty real tickets, personal details removed, and write the reply you would want for each. Then add a handful of bad replies and say why each one fails: too long, wrong tone, made up a feature, answered a different question. These examples teach your standard faster than any adjective.

An evaluation set

Set aside a separate group of real cases, ideally a hundred or more, and keep it apart from all tuning work. This is the evaluation set. At acceptance, the feature runs over all of it and the results are scored. Include the awkward cases: angry customers, tickets in Hindi or Tamil, tickets with three questions in one, and tickets the feature should refuse to answer.

Agree how each output is scored. A simple scale works well: usable as is, usable with small edits, or unusable. Write down what share of usable replies you need before the feature goes live, and treat that figure as a line in the specification.

Who checks the results

Name the people who will score the evaluation set. Support agents who handle these tickets every day are usually the best judges. Agree how many people score each output and what happens when they disagree. If the team builds an automatic check, such as a second model that grades answers, keep a person reviewing a sample of its scores.

Data handling and personal data

Support tickets contain names, phone numbers, order details and sometimes worse. Write down where that data goes: which model provider, which region, whether the provider keeps prompts, how long your own logs keep them, and who can read them.

If your users are in India, the Digital Personal Data Protection Act, 2023 and its Rules apply to how you collect and process their personal data, and sending it to a model provider is part of that processing. In general terms, you need a lawful basis, a clear notice, reasonable security safeguards, and contracts with the processors you use. Confirm the detail for your product with your lawyer. Where it helps, TrustOS is the group's platform for DPDP consent and compliance.

A practical rule for the brief: send the model only the data the task needs, and mask the rest.

Model choice and running costs, agreed in writing

Every call to a hosted model costs money, and a busy helpdesk makes thousands of calls a month. Before work starts, agree in writing which model or models the feature will use, roughly what each call costs, an expected monthly figure at your current volume, and who pays the provider. Agree what happens if a provider changes its prices or retires a model.

On 2good terms, these product running costs are agreed in writing before work starts. They are separate from the USD 50 AI tool budget included in each block on our tiers, which covers the delivery team's own AI assisted tools, such as coding assistants and AI testing.

Fallbacks

Decide what the user sees when things go wrong: the provider is down, the answer takes too long, or the model declines. For suggested replies, the simple fallback is to show nothing and let the agent write as usual. For an agent that takes actions, the fallback is usually to stop and hand the case to a person. Write the time limit down, for example "if no draft arrives within a set number of seconds, hide the panel".

Acceptance tests for output that varies

With the evaluation set and scoring agreed, acceptance becomes a measured check:

1. The team runs the feature over the full evaluation set using the agreed model and settings.
2. Your named reviewers score every output on the agreed scale.
3. The share of usable outputs is compared with the figure in the specification.
4. Hard rules are checked separately. A single reply that promises a refund or leaks another customer's details is a failure, whatever the overall score.
5. Fallbacks are tested by switching the provider off or forcing a timeout.
6. Running cost per call is measured and compared with the written estimate.

Run the set twice. If the scores move a lot between runs, the feature is unstable and needs work before you accept it. Keep the evaluation set after launch. When a model is updated or a prompt is changed, run it again.

A short checklist for your brief

• The task in plain words, including what the feature must never do.
• The exact inputs the model will see.
• Twenty or more good examples and a few bad ones, with reasons.
• A separate evaluation set with awkward cases.
• A scoring scale, a pass figure and named reviewers.
• Where personal data goes and how long it stays.
• Model choice and running costs, agreed in writing.
• Fallbacks and time limits.

Where to go next

If the feature is a single step, such as drafting, summarising or extracting, see AI integration. If it plans and carries out several steps, such as looking up an order and then issuing a return label, see AI agents. Either way, the brief becomes part of the signed specification, and you pay after the feature passes the tests you wrote.

Sources

• Ministry of Electronics and Information Technology: the Digital Personal Data Protection Act, 2023 and the rules made under it.