The Hallucination in the Room

There is a particular kind of meeting—you will have sat through one—where a beautifully formatted presentation arrives. Sixty slides, colour-coded, possibly animated. Somewhere around slide eleven you realise, with quiet unease, that no one in the room has actually thought about any of this.

The slides exist. The meeting is happening. Nothing is being said.

India’s Supreme Court encountered something not entirely different this month. As reported in The Indian Express by S. Tripathi, the Chief Justice expressed alarm that lawyers were filing petitions with case citations that do not exist—invented by AI tools and submitted without anyone checking. Courts in Karnataka, the Bombay High Court, and a US federal court case, Mata v. Avianca, have seen similar material: citations that look real, precedents that sound authoritative, none of them genuine.

The Chief Justice called it “absolutely uncalled for”. Justice Bagchi, on the same bench, noticed something more telling: drafting quality had been falling for years, long before AI appeared. Petitions, he said, increasingly pile up quotations where arguments should be. AI did not create this habit. It has simply made the habit faster—and harder to spot.

AI did not create this habit. It has simply made the habit faster—and harder to spot.

When output replaces thinking

This is not only a legal problem.

The same thing happens in organisations every day.

Proposals move up the chain. Decks circulate. Reports are written, filed, forwarded and occasionally read. The volume of material produced has never been higher. Whether any of it contains genuine thinking is a question that rarely gets asked.

A Gartner survey in 2023 found that 77% of employees said they were experiencing change fatigue—not because change itself was overwhelming, but because more communications, more decks and more initiatives kept arriving with no reduction in what was already there. The inbox fills. The thinking doesn’t follow.

A junior analyst can now produce a thirty-page market report in under an hour. It will have data, structure, a conclusions section and an executive summary. It may also contain figures that are plausible but wrong, trends that point the wrong way, and analysis of competitors who have since merged, pivoted or quietly shut down. The production looks solid. The thinking often isn’t.

This is what Nandan Nilekani was pointing to when he spoke at the Infosys Investor AI Day in Bengaluru on February 17. The real challenge, he argued, is not getting access to AI. It is having the expertise and context to use it well.

“Enterprise context is so important,” he said.

That context cannot come from a tool. It lives in people who know the subject well enough to notice when something is wrong. What organisations are facing, in his view, is not an opportunity gap but an implementation gap—the widening distance between what AI can produce and what people with real expertise can actually do with it.

The real constraint, in other words, is not the technology. It is the quality of judgement inside the organisation.

The real constraint is not the technology. It is the quality of judgement inside the organisation.

The lawyers who filed fabricated citations were not short of AI access. They were short of the legal knowledge—or the professional care—to read the output critically. The two are not the same thing.

Volume is not the same as value

A petition exists to make a coherent argument before a court. It requires accuracy, reasoning and the careful use of real precedent. A petition that cites twenty cases but invents twelve of them has not saved time. It has wasted the court’s, broken professional rules and undermined trust. More pages do not make a weaker case stronger.

The same is true in business.

A strategy document is supposed to carry hard-won understanding of a market, a customer and a set of risks. A proposal should reflect judgement, not just research. When these documents are produced quickly, comprehensively and without depth, they become something else—the appearance of expertise rather than the thing itself.

AI makes this kind of appearance much easier to sustain. A language model can produce confident, well-structured text on almost any subject. It is, at its core, predicting what a convincing document looks like.

That is genuinely useful.

It also means organisations tend to get more of whatever they already had. A team that thinks carefully will use AI to think better. A team that was already producing hollow work will produce more hollow work—faster, and with better formatting.

The problem is what we are measuring

Both Justice Bagchi and Nilekani, from their very different vantage points, are describing the same thing: the slow replacement of thinking with the evidence of thinking. AI has accelerated this, but it did not start it.

The question worth asking—in a law firm, a consultancy or a strategy team—is what any piece of work is actually for. A submission, a proposal, a report: each exists to do something specific. When production becomes the goal, the original purpose tends to quietly disappear.

Nilekani, at the Infosys Investor AI Day, was direct about what this means in practice.

“It’s not about using AI tools,” he said. “It’s about productivity out of those tools. Otherwise you’ll get false productivity, which leads to more complications.”

False productivity is a good name for what the Supreme Court bench was looking at. It is also a good name for what happens in most organisations when AI arrives and output doubles but nothing actually improves.

In law, the advocate filing a petition needs to know it well enough to stand behind every line. In business, the leader presenting a strategy should be able to defend any page of it without the deck in hand. Production becomes evidence of effectiveness only when it is created by people who know enough to make it matter.

The Supreme Court’s discomfort is a useful prompt.

Most organisations, if they looked closely, would discover a version of the same problem already sitting in their meeting rooms—formatted, circulated and rarely questioned.