AI agents for internal operations: practical, unglamorous use cases

The interesting demos involve an agent booking travel or researching a market. The deployments that survive contract review and are still running a year later involve matching invoices to purchase orders. This post is about the second kind, because that is where the money is and almost nobody writes about it.

What an agent is, without the marketing

A model call takes input and returns text. An agent wraps that in a loop: the model can call tools, see what came back, and decide what to do next, repeating until it finishes or hits a limit.

That loop is the entire difference, and it is what makes agents useful for operations. Most back-office work is not a question needing an answer. It is a sequence: open the document, find the reference, look it up in another system, compare, decide, record the result, and raise a flag if something does not match. A model alone cannot do that. A model with tools can.

It also explains the cost profile. One user action becomes five or ten billable model calls, each carrying the conversation so far, which is why agent features surprise people on the invoice. The arithmetic is in what it costs to run an LLM feature in production.

Which jobs qualify

Six jobs that are boring enough to work

Every one of these shares the same shape. High volume, a clear definition of correct, a human already checking the output today, and a mistake that is embarrassing rather than expensive. That combination is why they work, and it is the test to apply to any candidate.

The three rules that are not optional

Cap the loop. A fixed maximum number of steps, after which it stops and escalates rather than continuing. This is simultaneously your cost ceiling and your blast radius, and it is a single number in a config file.

Make every action reversible or approved. Reading is free. Writing needs a category: either it can be undone by a person in a minute, or it goes through an approval. Issuing payments, emailing customers and changing prices belong in the second group regardless of how good the agent gets.

Log the whole trace. Every tool call, every result, every decision, kept and readable. When someone asks why the agent did something in March, the trace is the only answer that exists. It is also what your evaluation set is built from, which connects to testing software that answers differently every time.

Measure the escalation rate, not the accuracy

Accuracy is the wrong headline metric because it hides the distinction that matters. Two agents can both be 85% correct: one escalates the 15% it was unsure about, the other confidently gets 15% wrong and tells nobody. The first is a useful colleague. The second is a slow-motion audit finding.

So track three things instead:

• Handled rate. What proportion completed without a person. This is the business case.
• Escalation rate. What proportion it correctly declined to finish. This should be non-zero, and a rate of zero is a warning rather than an achievement.
• Escaped error rate. What proportion it completed wrongly without flagging. This is the only number that should have a hard threshold attached.

An agent starting at 60% handled with a 2% escaped error rate is a good launch, and the handled rate climbs as you feed the escalations back into the prompt and the evaluation set. Launching at 95% handled with a 12% escaped error rate is worse in every way that matters, and it is what you get by tuning for the first number alone.

Build or buy

The same test as everywhere else: is this process how you compete, or is it overhead you happen to have? Invoice matching is not a competitive advantage and there are products that do it. The particular way you triage claims, price a job or route work to your own people may well be, and no product encodes it.

The wrinkle specific to agents is that the buy option often needs the same integration work anyway, because the agent has to reach your systems either way. That reduces the saving, so check it before assuming buying is cheaper. The general form of the argument is in build or buy.

Before you start

1. Pick the task by volume, not by interest. The dull, frequent one with an existing human checker beats the ambitious one every time.
2. Write down what correct means, as a rule someone could apply by hand. If you cannot, the agent has no target and neither does your evaluation.
3. Find out who does it today and involve them. They know the exceptions, and the exceptions are the project.
4. Classify every action as read, reversible write, or approval-gated. Before any code.
5. Set the step cap and the escaped error threshold as launch criteria, in writing.
6. Plan for the escalation queue to be real work at first. It shrinks, but somebody owns it from day one.

Related: from pilot to production covers why these stall, where AI actually pays off covers choosing between candidates, and data readiness covers what has to be true first.

Our AI and machine learning team builds these with the step cap, the approval gates and the trace store in the first version, because retrofitting them means explaining an incident first. If you have a high-volume process someone is doing by hand between two systems, describe it to us.