I’m Lorna. I lead Legal Alignment at Flank, and most of my week is spent supervising work an agent did first.

Most conversations about legal AI ask whether the output is right.

This piece is about a harder question: whether you can put your name behind it without redoing the work.

The redline I couldn't sign off

A vendor agreement comes back from the agent looking clean. A few clauses redlined to our positions, a couple of fallbacks accepted, a liability cap left exactly where the counterparty put it. By every standard we set, the review is correct. And I still won’t send it.

I won’t send it because I can’t see why the cap was left alone. Maybe the playbook says that figure is fine for a contract this size. Maybe the agent missed it. From the document in front of me, those two possibilities look identical. So I do the thing that defeats the point of having had the agent do it at all, at least the way most of them are built today: I open the full contract and read it the way I would have read it if nothing had touched it. The few minutes the automation was supposed to save me become most of the morning.

I’ve seen this happen often enough, in my own work with agents and across the enterprises I’ve watched try to adopt them, that I’ve stopped treating it as a quirk. I think it’s the central problem in making legal AI genuinely useful, and that most of the industry is looking slightly to the left of it. We keep asking whether the output is right. The harder question is whether the person receiving it can stand behind it without redoing the work, and those turn out to be very different questions. I’ll say upfront that this is the question we spend most of our time on, so read the rest as someone who is invested in getting it right rather than neutral about it.

🧠 The thing I need is the reasoning, not the answer

A redline on its own is a verdict. It tells me what the agent decided and nothing about why it decided that, which means the only way I can verify it is to reconstruct the reasoning myself. That is the same labour as doing the review. No wonder it doesn’t feel like a saving.

The few minutes the automation was supposed to save me become most of the morning.

What changes the maths is seeing the rule the agent applied. When the work arrives with its justification attached, “our indemnity fallback is mutual, capped at fees paid in the prior twelve months, so I accepted the counterparty’s clause because it already sits inside that limit,” the judgment I have to make shrinks. I’m no longer re-reviewing the contract. I’m checking whether I agree with the rule, and whether the rule was applied where it should have been. Both are quick. Reconstructing the whole thing from a silent markup is not.

The practical test is small. Before you approve anything, you should be able to answer two questions without reopening the source document: which rule produced this change, and was it applied in the right place. If the surface can’t tell you that, you don’t have a review. You have a guess.

There’s a version of this becoming a professional obligation, not just a preference. A proposal from the California bar, out for public comment earlier this year, would require a lawyer to independently review, verify and exercise professional judgment over any output produced with these tools. I find that framing clarifying. You cannot exercise judgment over a black box. You can rubber-stamp it or you can redo it, and a careful lawyer, faced with only those two options, will redo it every time. Visible reasoning is what creates a third option, the one where supervision is actually supervision.

Correct by our standards is not the same as done

Here I’d argue against my own profession’s reflex, which is to read every change a lawyer makes to an agent’s output as proof the agent fell short. Sometimes it is. Often it isn’t.

The agent executes the playbook. The lawyer layers discretion on top of it: the read on this particular counterparty, the commercial temperature of the deal, the thing someone from the business said on a call that never made it into a written rule. A technically perfect review still gets changed because of all the judgment that lives outside the playbook, and that change is not a defect report. It’s the job. The human in the loop is where the lawyer’s discretion legitimately sits, rather than a fallback for a weak agent, and it would sit there even if the agent were flawless.

You cannot exercise judgment over a black box.

Once you accept that, you have to give up the idea that accuracy closes the trust gap. It can’t. There is no accuracy score high enough to remove the reviewer, because the reviewer was never only there to catch mistakes.

🔍 What the review costs is what decides everything

If accuracy doesn’t close the gap, what does? I’ve come to think it’s the cost of the review itself, measured not in minutes but in attention.

If marking up the agent’s work, or telling it where I disagreed, takes real effort, the kind that demands headspace I didn’t have much of to start with, I will quietly drift back into re-checking everything. And the moment I’m re-checking everything, the automation has bought the department nothing. The expensive resource is still doing the inexpensive work. We’ve just relabelled it supervision and told ourselves something changed. I suspect friction in the loop kills far more deployments than error rate ever does, and it does it silently, because nobody files a complaint about an agent they’ve simply stopped relying on.

There is no accuracy score high enough to remove the reviewer, because the reviewer was never only there to catch mistakes.

This is why I think the review experience deserves the same design seriousness as the work being reviewed, maybe more, and it’s the conviction we’ve ended up building around. The test is whether a reviewer can respond at the granularity they actually think in, instead of being pushed back into drafting. In practice that means at least three ways to say no, each a different shape of “I still need to tweak this”:

The reasoning belongs on that same surface, sitting next to the change it explains, so the decision and its justification arrive together rather than sending the reviewer off to hunt for one of them. And there’s a blunt way to tell whether the loop is working: a reviewer should get through the agent’s output far faster than they’d get through the same work by hand. The moment those two times start to converge, the agent has stopped earning its place.

Why an assistant can’t give you this

It’s worth being precise about why a general assistant, Claude Cowork, ChatGPT, a copilot in your Word ribbon, can’t hand you the same thing. I use those tools and rate them, so this isn’t a knock. The gap is structural.

With an assistant, you hold the pen. You paste the contract in, ask for the review, read the reply, then do the redlining and send it yourself. Ask it to explain its reasoning and it will, at length, because you’re sitting in the chat doing the asking. But nothing is ever waiting on your desk, finished, for a decision. The work was yours the whole time. There is no queue to clear, nothing to approve or dismiss, no exceptions to triage, because you never handed the task over. You went faster, which means you, the expensive resource, are still the one doing the routine work.

I suspect friction in the loop kills far more deployments than error rate ever does.

The supervision experience I’ve been describing only exists once an agent completes the work and hands it back for judgment. That is when reasoning on the surface, one-line rejections and a queue of exceptions start to matter, and none of it retrofits onto a chat window that assumes you’re still holding the pen. Getting that right is a different design problem, and it’s the one that decides whether the work leaves your desk or just moves through it faster.

🌱 Trust is a sequence, and the loop is how you walk it

None of this arrives at once, and I don’t think you can skip steps to get to the end faster. The pattern is fairly consistent, both in my own work with agents and in the enterprises I’ve watched move further down this road. At first you read everything the agent produces, in full, the way you’d check a new joiner’s work. Later you accept and dismiss without reopening the source each time. Eventually, on the narrow and well-behaved slices, the renewal reminders, the straightforward NDAs, you let it run and look only at the exceptions it flags.

You went faster, which means you, the expensive resource, are still the one doing the routine work.

The loop is what carries people from the first stage to the last. I’ve not seen anyone arrive at the far end because an agent crossed some accuracy threshold. They get there by reviewing, then reviewing a little less, until the reviewing itself tells them it’s safe to ease off. That’s worth saying plainly to anyone selling autonomy as a setting you switch on.

And there’s a dividend in this that’s easy to miss, which is the part I find most interesting to design for. When a reviewer dismisses the same kind of suggestion three times in a week, that’s a signal, and it usually means the playbook position is wrong or was never properly settled, rather than the agent misbehaving. Fed back in, those corrections tighten the rules, so the next month’s work is better because of the reviewing done this month. A well-designed loop compounds. A painful one just bleeds. The agents worth building are the ones that treat that signal as the point rather than exhaust.

Measure maturity on flow, not on the accuracy number

All of which points to a measure of maturity I’d trust more than a benchmark score. Picture two agents. One is right ninety-five percent of the time and miserable to supervise, every correction a small fight with the interface. The other is a little less accurate and a genuine pleasure to review, reasoning visible, decisions easy to express. I’d call the second one more mature, and I’d deploy it, because what determines whether the work leaves my desk is how well the person in the loop is served, not how high the accuracy figure climbs.

So when you’re judging an agent, the questions that predict adoption sit alongside accuracy rather than beneath it:

None of those show up on a benchmark, and every one of them decides whether the work actually leaves the building.

The benchmark that actually decides adoption isn’t an abstract bar anyway. It’s whatever the team already does today. Where a contract is already reviewed in-house, the agent and its loop have to be lighter and faster than that existing review, not lighter than doing nothing. If supervising the agent costs more attention than the old way of just doing the work, the agent loses to the status quo, and it deserves to, however good its output looks in isolation.

I don’t think anyone has this fully worked out, and I’d be wary of anyone who claims they do. Most of the attention in legal AI, and almost all of the benchmarking, still sits on the quality of the output. I find myself more interested in the unglamorous part, the review, the rejection, the small act of putting your name behind a decision that something else drafted. That’s the part that decides whether the work genuinely leaves the desk and stays gone, or quietly finds its way back onto it a week later. The longer I spend on this, the more convinced I am that it’s where the whole thing turns, and it’s where we’ve chosen to put the work.

✳️