Last updated on 28/06/2025
The AI gold rush has created a weird kind of pressure. If you’re not chasing the trend, you’re behind. If you ask hard questions, you’re a stick in the mud. If you jump in too fast? You might end up with a flashy prototype and no plan to support it.
Let me be clear, being contrarian and dismissing AI out of hand isn’t clever. There’s real value here, it’s just not as much as anyone’s claiming, and certainly no where as easy to realize as they’re promising. You don’t need to roll your eyes at every new LLM feature and foundational model, but you don’t need to immediately go chase after it either.
This post isn’t about saying no to AI. It’s about saying yes thoughtfully. It’s about learning to ask better questions and make better bets. Skepticism isn’t passivity; it’s your superpower. It’s what turns a shiny tech idea into actionable road maps.
Place Small Bets That Make You Smarter
You’re not going to pick the perfect AI use case on your first try. That’s not a failure. That’s normal! The key is to make a bet small enough that it teaches you something useful, even if it doesn’t fully pay off.
Ask yourself:
- What did we learn?
- Was that lesson worth what we spent?
- How will it inform the next decision?
Set expectations early. If leadership believes this first AI initiative is going to tredecuple productivity (that’s 13x, and yes, I had to look it up) or ship a game-changing feature in a quarter, they’re going to be disappointed. Worse, they’re going to shut it down and tell everyone it was a waste of time. That disappointment doesn’t come from the tech. It comes from expectations that were never realistic in the first place.
Make room to miss. A test isn’t useful if everyone fails or aces it. You need questions with enough granularity to show whether you’re directionally right or getting better. Design your first bets around questions, not promises. Build them to surface unknowns. Use them to test assumptions about data quality, integration paths, user behavior, and organizational readiness. If your pilot “fails” but shows where you’re strong, and where you’re not, that’s a win.
Keep the risk survivable. Don’t bet your credibility, your team’s capacity, or your org’s trust on a system no one fully understands yet. Absolutely try AI, but treat AI initiatives like training, not summit bids. Everest is littered with ambitious corpses. The stair treadmill at the gym? Not so much. It’s less glamorous, but it builds strength you’ll use on every climb, even the one up Everest.
Demos Are Lies of Omission
Prototypes almost always look good. It’s easy to look good when you don’t have to scale or integrate. No demo survives first contact with the enemy real users. Worse, demos are evaluated with rose-colored glasses: “If we just invest a few more hours,” “Sure, that part’s rough, but look at this other use case.” Those are signals, not of promise, but of premature optimism.
Most of the problems that matter don’t show up until the stakes are real:
- Edge cases and unexpected input
- Error handling and recoverability
- Subtle inaccuracies that erode trust
- Integration friction
- Jailbreaks and safety gaps
- Long-term maintenance questions
Most AI systems today are about turning generic inputs into arbitrary, fit-for-purpose outputs. That’s powerful — and also dangerous. “Arbitrary string” is a red flag in every secure software system for a reason. AI doesn’t make that go away. It makes it easier to miss.
That’s why early efforts should balance showing promise with surfacing failure. Your demo should highlight potential but it also needs to reveal what breaks, where it struggles, and how it handles edge cases. A good pilot proves two things: that there’s real utility, and that your team knows where the system needs work. Don’t just aim to impress. Aim to learn.
Check Your Own Tools: AI Features That Work vs. Waste
Before you roll out your own AI feature, take a hard look at the ones you already use. How many LLM-backed additions in your favorite tools are actually good? Not just flashy, but actually useful? How many new AI features have you instantly googled “how to disable…” for? Which pile is bigger?
Spotify’s generative DJ is the newest AI-enabled feature in one of my daily drivers. It’s… fine. It can follow a genre for a while and tell me interesting tidbits about a song that’s playing. Is it meaningfully better than Pandora’s recommendation engine from nearly two decades ago? No, not really. But it does have a generated voice. Which isn’t the music I wanted to hear.
Unfortunately, there’s a pattern here. Grammarly used to catch actual errors. It still does, but it also confidently suggests things that are flat-out wrong. Google has generative answers up top now, ahead of the articles I’m actually searching for. That they get in the way is worse than their semi-frequent inaccuracies.
I’m not here to dunk on tools from tech giants. Okay, maybe a little, as a treat. My point is this: even smart teams at big companies, people with good intentions, real product discipline, and all of the resources in the world shipped AI features that made things worse. Or they shipped features that didn’t improve anything and still damaged user trust.
How does that happen? Someone went chasing after something flashy. The thing is, flash doesn’t equal value. The most dangerous thing about hype is that it has a way of rewriting what counts as “working” and what counts as “good”.
If you’re replacing an existing feature with something AI-powered, make sure it works at least as well. If you don’t have a test suite yet, build one before you start. And if you’re building something new? Ask yourself if your users will notice, care, or even want it.
Would You Rather Talk to a Human or Navigate a Bot?
You already know this one. You’ve lived it.
That customer support chat that looked promising until it turned out to be a glorified FAQ scraper? The one that looped you back to the same three options no matter what you typed? The one that made you type “agent” five times before you got a human? Yeah, that’s the bar.
These systems are often deployed to reduce cost, not improve experience. That up front cost-savings may come back as churn, frustration, or brand erosion. AI isn’t free, and attention is expensive.
If your users are reaching out for help, that moment matters. Putting an AI in front of them says something, not just about your product, but about what you think the interaction is worth.
This applies not just to help desk calls and interactive chats, but any user interaction. Ask not “Can we automate this?”, ask “What benefit will we realize by automating this, and what value might we lose?”
Know What to Build vs. What to Wait For
Not every AI use case needs to be built in-house. Some of the most promising applications are already commoditizing. They’re being bundled into platforms, libraries, or APIs that are more robust than anything you can spin up quickly on your own. When a problem is well-understood and widely shared, it’s usually just a matter of time before it gets productized. Sometimes the smartest move is to wait.
But waiting doesn’t mean doing nothing.
Building small, context-specific tools, even temporary ones, teaches you how the pieces fit together. You learn what your data really looks like in practice. You learn what your teams are ready for, and where they’re blocked. You learn how your users react to even subtle changes in how a feature works. Those are lessons that make you a better builder — and a much smarter buyer.
Early experience is useful. Just don’t mistake it for a permanent solution. No technology lasts forever. Except for the LAMP stack, which is somehow still out there quietly serving up 40% of the internet.
Don’t Need to Be an AI Expert to Be an Expert Buyer
You don’t have to understand transformers to make smart decisions about AI. You don’t need to fine-tune your own models or keep up with every new paper. But you do need to ask the right questions.
- What happens when the system fails?
- What does success look like?
- How will we measure it?
- Who’s responsible for watching those metrics over time?
- Can we try it ourselves, with real inputs, and see if it helps or hurts?
You’re not buying magic. You’re making tradeoffs, and making good tradeoffs requires clarity, about your goals, about your users, and about your tolerance for risk.
Skepticism is your superpower; credulity, your kryptonite. Ask hard questions, but don’t let “doing the research” become your excuse to do nothing.
If you’re stuck on that first AI bet, or you just want another set of eyes on the problem, don’t hesitate to reach out.