A short story on the dangers of misapplying generative AI technology, born of a prompt for a writing competition and more than a few at-bats with folks wanting to treat LLMs as a panacea.
The Client’s New LLM
A riff on The Emperor’s New Clothes
1130 Words
Not so long ago at a company not far from here, a CEO demanded the construction of a generative AI platform on a grand scale. “If we do not have our own LLM to write our emails and perform our research, surely we will fall behind our competitors,” they proclaimed, and all nodded along sagely, for to question the CEO was to invite joblessness.
And so, the CEO’s greatest lieutenants, the vice presidents of engineering and product, the officers of operations, and other extravagantly titled men went and scoured the land for consultants that might build the CEO’s grand vision and deliver value unto the shareholders, and bonuses unto their end of quarters.
It came to pass that the men asked a local consultancy if they might build it. During the meeting, a lone engineer asked, “To what end should we build this platform? What would it do? Who would use it, and how would it help them?”
The men of the company were wroth, as questioning the CEO was taboo, and so they went to the big three consultancies, and found wise men who could do as they were told without asking questions, as only truly wise consultants can.
Thus, it came to be that the CEOs platform was built. It was one of the largest generative platforms in all of the lands, cleverly named, expensively produced, and ready to serve the CEO and his company. Unfortunately, no applications had been built upon it, and the budget had been blown building the platform itself. And thus cheaper local talent had to be found to build tools that could leverage the wise investment of the CEO.
And so it was that the young questioning engineer was once again engaged with the platform. Once more they asked, “Who will use this software, and to what end?” Once again the great men of the company were angered, because the wisdom of the CEO’s investment was self evident. “Our advisors will use the LLM to make their reports, so that the CEO might have more expert advice at higher volumes.”
The engineer was concerned. “You do understand that these systems hallucinate, right? You’ll be generating reports filled with fabrications. You’ll be making serious business decisions on the back of fantasy if you do this,” she said. They would not hear her. “We have the grandest generative platform in all the lands, and it will not hallucinate. Even if it did, we will have our experts review the reports before sending them on and making decisions.”
The engineer objected, saying “Right, but there’s this well understood and well studied phenomenon called automation fatigue,” but the extravagantly titled men cut her off.
“Do you want the contract or not? Just build the damn app!”
And so, the engineer and her team went to build a RAG application against the platform, and they did what they could. They sourced the truest information they could find. They toiled to present it in the cleanest way. They invented new techniques and tools for tracing the sources of information and evaluating accuracy of documents. But as hard as the engineers worked, at the end of the day, you cannot build a lasting structure from worm-eaten wood. The domain experts said “These reports aren’t accurate,” and the engineer and her team nodded sadly, because they knew the domain experts were right.
But the men in finance and product lauded the application, saying it was the grandest use of the platform so far! A model for all that would come after it! And that was true, but only because total orderings behave strangely on singleton sets. And the CEO applauded the vice presidents and their captains for making use of the grand platform the company had built, and the shareholders rewarded the lot of them with more investments.
And the subject matter experts? Since the CEO had given them such a lovely platform, and the product owners and VPs had given them such a great tool, they were rewarded with more work. With increased work came decreased scrutiny on the report outputs. The subject matter experts, for their part, did what they could to turn the inaccurate reports into good advice. However, with increased workloads came less time to consider every task, and with less time came less vigilance.
And the questioning engineer and her team? With the app fielded, their services were no longer required. They went their separate way, and were but a distant memory in the minds of the client.
That was, until one day a year later. The generals of the CEO, the vice presidents and the operating officers and the captains of industry came to speak to the engineer. They had a grand vision to build more software at the suggestion of one of their latest reports. They wanted a grand system that could support serpentic AI.
The engineers were curious, what, pray tell, was serpentic AI? The company men scoffed. Everyone knew serpentic AI was harnessing huge herds of snakes to perform computations on their own. The serpents could search the web, build 3D models, reserve seats for you at a restaurant, the list went on and on.
“How would snakes do that? I’ve never heard of a snake calling a restaurant,” said the engineer.
However, the businessmen dismissed her. “Everyone is aware of the Garter Snake hype cycle, and we are determined to be early adopters this time. We’ll not miss out on the value, the way we did with LLMs. Thank you for your time, but we’ll be finding a true expert in serpentic AI”
The engineer wished them well, and offered “Remember, any good system benefits from diversity. Be sure to use lots of different snakes,” as they made their way from the conference room. The vice presidents and senior vice presidents and the vice chair of serpentic adoption nodded their acknowledgement of the advice and waved their goodbyes.
The engineer thought it a remarkably queer conversation, but sat it aside in favor of more pressing and more rewarding work that did not call for antivenom. She had nearly forgotten about the exchange, when, some months later, she was invited to a grand launch party for her erstwhile client’s serpentic AI platform.
All of the vendors in the land, and all of the hands of the company, both great and small, were in attendance. The CEO entered the room in all of their pomp and finery to conduct a grand public demonstration of the capabilities of their new serpentic AI. The crowd waited with baited breath as the CEO dictated a response to an email. The engineer smiled, with no small amount of satisfaction, as the CEO was promptly bit in the ass by a coral snake for his efforts.