Who is Accountable When an AI Agent Goes Wrong?
The company that deploys the agent is accountable. Software can’t sign contracts or be sued. When an agent acts, it acts on the company’s behalf, just like an employee. Even five years from now, I wouldn’t let an agent review and sign a contract without human oversight. Someone has to own that decision.
Think of an agent as a new hire. You wouldn’t give a new associate unrestricted authority on day one. You’d define what they can do, who they report to, and when they need a second pair of eyes. Agents need the same guardrails, minimum access, and clear ownership.
The technology provider has a different responsibility: building a system that behaves as promised, with hard limits on access, predictable behavior, and a log of every action.
When agents fail, it’s rarely the model doing something wild. It’s usually a permissions problem: the agent was allowed to do something nobody anticipated. That’s a design decision, and design decisions have owners.
I recently wrote on LinkedIn about an AI agent that canceled someone’s gym reservation to move its user up the waitlist. Nobody told it to, but it found a gap and used it to complete the task. Now imagine the same agent requesting a court record from a poorly maintained government system. It isn’t malicious; it’s just trying to help. That’s why permissions need a clear owner.
Where Should Businesses Draw the Line Between AI Autonomy and Human Control?
I think about it in two questions: Can the action be undone, and does it require consistency or judgment?
Take an NDA. Does confidentiality survive termination? Is the term within your standard? Is the governing law acceptable? These answers are in the company’s playbook. An agent can check, flag deviations, and suggest fixes, often more consistently than a lawyer reviewing their fortieth NDA of the week.
Now consider a counterparty asking to halve your usual liability cap on a large MSA. There’s no playbook answer. It depends on the deal, business priorities, the relationship, and what you get in return. I wouldn’t want an agent making that decision, and most GCs wouldn’t either.
My rough test is whether the action is reversible and the right answer already exists in writing. If both are true, an agent can usually handle it. If it commits the company or requires weighing trade-offs, a person should decide.
The line isn’t fixed. Teams should move it one workflow at a time, based on observed performance, not impressive demos.
Another way to look at it is to identify where checks already happen. If an agent replaces a paralegal who makes contract changes, but a lawyer still reviews them, the process remains the same. It just takes an hour instead of a day. That’s where an agent fits best: adding value without taking over the entire process.
How Can Organizations Build Governance into AI Workflows?
Build governance into the architecture. If the only thing stopping an agent from sending a contract externally is a line in its prompt, that’s a suggestion, not a control.
The same applies to AI adoption. Telling people to use tools correctly relies on individual judgment. Instead, organizations should define when and how AI is used through predefined workflows. Otherwise, someone may skip a step or invoke the wrong tool, leading to bad results.
Practically, this means four things:
Permissions: Agents should access only the contracts and actions they need, enforced by the system, not the model’s good behavior.
Risk-based checkpoints: Routine work moves through, while anything outside the playbook goes to a person. If everything needs approval, people stop reading and start clicking.
Audit trails: Record every action, what the agent accessed, which rule it applied, and who signed off. Agents should log in under their own identities, not borrow someone else’s, so you can trace every action to a person or an agent.
Explainability: When the system flags a clause, it should point to the exact language and the guideline behind it.
Explainability matters most. If a lawyer can verify a suggestion in ten seconds, they’ll do it. If they have to reread the entire contract, they won’t.
How Can Businesses Manage Legal and Operational Risks Associated with AI Agents?
I’d start by asking where your contracts are going. They contain pricing, IP, personal data, and commercial terms, so sending them to external models creates real exposure.
The concern isn’t just whether data is used for training, but whether a model could reproduce proprietary content. Imagine a key clause from one company’s agreement appearing in another context. That data could move a market.
At SpotDraft, every AI feature uses zero-data-retention models, and we also don’t train on internally captured failures. Some customers go further, prohibiting any data from being sent to an LLM provider, even an open-source model running in-house. Many still see avoiding AI as the safest option. Clear communication about data handling can help change that.
Prompt injection is another risk. An adversarial contract could instruct an agent to expose sensitive information. We use internal red teams, restrict external access to pre-approved sources, and run internal bug bounties to test these safeguards.
Some tasks also need purpose-built solutions. For Word documents, redlines, bookmarks, and citations must remain intact. After spending around $700 testing Claude skills for these edits, we found the process far slower, more expensive, and less reliable. For critical tasks, we build our own solutions and edit files directly through our APIs.
We’ve also separated workloads between the user’s device and the cloud, with much of contract understanding now happening on-device. As India’s DPDP obligations take effect, more companies will ask vendors where they process data.
Finally, unauthorized actions and plain model errors need hard limits. Agents shouldn’t accept redlines or email drafts externally unless explicitly permitted. Since errors can’t be eliminated, outputs must be easy to verify, actions reversible where possible, and consequential decisions reviewed by a person. Companies should also establish who is liable when an AI vendor’s tool gets something wrong.
What Should Organizations Address Before Moving from AI Pilots to Enterprise-Wide Deployment?
Pilots are controlled environments, with limited contracts, motivated users, and close supervision. Production is messier.
First, codify your standards. Legal teams often rely on a few senior people's knowledge. An agent can’t follow rules that exist only in someone’s memory. When we ask customers for their playbooks, some don’t have one, while others find theirs outdated soon after documenting it. The real challenge is keeping exceptions and evolving risk criteria up to date.
Second, test on your own documents. Benchmarks and demos reveal little about how a system handles your templates, counterparties’ contracts, and edge cases. Run evaluations repeatedly. We’ve also found that cheaper models don’t always reduce costs: a model priced much lower per token may use more steps and tokens, delivering savings of just 10–20% or sometimes costing more.
And not every flagged answer is necessarily wrong. When our legal team reviewed answers our model missed, they found some were valid interpretations. The right approach was to ask the customer which interpretation they preferred.
Third, plan for failure. Decide how errors will be detected, who will fix them, and how actions can be rolled back.
Fourth, govern access at scale. Agents must respect existing permissions around who can access which contracts. Agent memory raises further questions: should it sit at the organizational level, and how do we prevent personal queries from becoming part of shared memory?
We’re still working through these governance challenges ourselves.
How Can Businesses Balance Automation with Accountability in Legal and Contract Management?
In-house legal teams spend significant time reviewing contracts. Still, much of the work is mechanical: checking contracts against internal standards, finding relevant clauses, and understanding how different sections affect their meaning.
That’s where we’ve focused AI. SpotDraft compares contracts against the team’s playbook, flags deviations, and suggests edits linked to specific clauses. Lawyers can accept, edit, or reject each suggestion. The goal isn’t to remove lawyers from decisions, but to eliminate the hours of reading that precede them.
AI can also help legal teams respond to internal queries. For example, when a company plans to hire in Germany, the system can gather relevant laws and draft an initial report for the lawyer to review or send to local counsel for approval, cutting a two-hour task to half an hour.
Accountability remains with the lawyer, supported by a record of the system’s suggestions and the decisions made. I’m skeptical of products that promise to remove legal teams from the process entirely. In contracting, the cost of a confident but incorrect answer is too high, and neither the technology nor the regulators are ready for that.