An agentic system is one that takes actions rather than merely producing text. It reads a request, decides which tools to use, executes them and repeats until the task is finished. That autonomy is precisely what makes the category attractive, and precisely what makes it difficult to operate responsibly.
The problem is not that agents make mistakes. Ordinary software makes mistakes. The problem is that when an agent makes a mistake, the organisation often cannot reconstruct what happened, which inputs it used, or why it chose one action over another. Without that record, incidents cannot be investigated and improvements cannot be measured.
What a usable audit trail contains
A useful record has four layers. The first is the request as received. The second is the sequence of decisions, including which tools were selected and what each returned. The third is the final action taken, with the exact parameters. The fourth is the identity of the human or system that authorised the run.
Most deployments capture the first and third and lose the middle. That gap makes debugging guesswork, because the interesting failures happen in the reasoning between input and output. Retaining intermediate steps also raises storage and privacy questions, which is often why teams omit them.
Tool calls are where the risk concentrates
An agent becomes dangerous when it can write, send, pay or delete. A model that drafts an email is harmless. A model that sends it to a customer list is not. The severity of an incident scales with the permissions granted to the tools, not with the eloquence of the text produced.
This argues for treating tool permissions as the primary control surface. Read-only access for exploration, approval gates for anything irreversible, and rate limits on external effects all reduce the damage that a confused agent can do. Security guidance from bodies such as CISA on identity and least privilege applies directly.
Human oversight has to be designed, not assumed
Many projects claim that a human reviews the agent’s work. In practice the review is often a click on a summary that nobody reads carefully. Oversight that works requires the reviewer to see the specific action, with enough context to judge it, at a point where rejection is still possible.
Designing this is unglamorous work. It means deciding which actions need approval, who is qualified to give it, and what happens when the approver is unavailable. It also means accepting that approval queues slow the system down, which is a real cost that must be weighed against the risk being managed.
Evaluation for multi-step behaviour
Standard model benchmarks measure a single response. Agents operate over sequences, so the relevant question is whether the whole trajectory achieved the goal without unacceptable side effects. That requires scenario based testing with simulated tools, where failure modes can be provoked safely.
Building these scenarios takes effort, and the scenarios age quickly as tools change. Teams that maintain a library of failure cases tend to catch regressions that unit tests miss. The NIST work on AI risk management provides a structure that maps reasonably well onto agent evaluation.
Incident response for autonomous systems
When an agent does something wrong, the first requirement is to stop it. That means a kill switch that actually halts execution, not a dashboard that reports the problem. The second is to identify every action the agent took, which depends on the audit trail described above.
The third is to notify affected parties, which may be a legal obligation as well as an ethical one. Teams that have rehearsed this process handle it far better than teams that invent it during an incident.
Cost of logging and what to keep
Full traces are large, and storing every intermediate step for every run becomes expensive quickly. A workable policy keeps complete records for a short window, retains summaries and decisions for longer, and stores anything tied to an incident indefinitely.
The retention choice should follow from the purpose. If the goal is debugging, days may suffice. If the goal is demonstrating that a decision was made fairly, the record may need to survive for years, and that requirement should be decided before the system goes live, because retrofitting retention is far harder than designing it in.
What good practice looks like today
Start with narrow scope. Give the agent one job, a small toolset and clear boundaries. Log everything, including the intermediate steps, and set a retention period that balances usefulness against privacy. Require approval for irreversible actions, and rehearse the shutdown procedure.
Then measure. Track how often the agent completes a task, how often it needs help, and how often a human rejects an action. Those three numbers reveal more about whether a deployment is safe than any demonstration, and they are the numbers a regulator or an auditor is likely to ask for.
Image: Esquilo · CC BY-SA 3.0 · via Wikimedia Commons.