AI-Native Products Will Get Things Wrong

Teams improve prompts, provide better context, evaluate outputs and switch to stronger models, all to increase reliability. This work matters and will continue, but there will likely always be moments when the system gets something wrong.

That creates a design problem traditional software doesn’t face. A button click has a defined outcome, whereas a probabilistic system makes the desired outcome increasingly likely but can’t guarantee it every time.

The hardest mistakes to deal with are the ones that look right

Some failures are easy to spot, whether that’s a nonsensical answer, obvious missing information or a system that admits it can’t complete the task.

The harder problem is an answer that’s plausible: clearly written, confidently presented and consistent with everything around it, while containing a wrong number, drawing the wrong conclusion or making an unnoticed assumption.

Better models make more convincing answers, so this problem doesn’t disappear as accuracy improves. You need ways for people to identify it, whether that’s showing where information came from, making source material easy to inspect or flagging when an answer relies on an assumption rather than fact. The aim is to give people enough context to judge what they’re seeing, especially when getting it wrong has consequences.

What happens after a mistake is part of the product too

Most design effort goes into the successful journey, but AI-native products also need to account for failure because it will happen. If an AI drafts something incorrectly, it should be easy to change. If it takes the wrong action, there needs to be a way to reverse it, while someone who spots a bad assumption should be able to correct it without abandoning the experience.

This matters more as AI moves beyond generating information and starts acting on someone’s behalf. A wrong answer in chat might be inconvenient, while a wrong action in a business process could be expensive.

Better accuracy doesn’t change this

As models improve, mistakes should become less frequent, but frequency is only part of the equation. The other is what happens when a mistake occurs.

A system that’s wrong once in every hundred interactions might be fine if the error is immediately visible and easily corrected, while the same rate could be unacceptable if the system can make expensive decisions, change important information or take irreversible actions without anyone noticing.

An AI-native product’s quality therefore depends not just on accuracy but on how visible its mistakes are, how much damage they can do before someone intervenes and how easily users can recover.

This shapes autonomy

The cost of being wrong should determine how much freedom a product gives the AI. If a mistake has small consequences, the product can let the AI do more and make correction lightweight. As consequences grow, the product may need confirmation before actions, clearer evidence behind recommendations or a straightforward way to reverse what happened.

Those decisions depend on what the product is doing and what happens if it fails. As models improve, the balance will shift, with products able to do more with less intervention as the probability of error falls, though it’s unlikely to reach zero.

The best AI-native products will make it possible for people to understand when something went wrong, correct it easily and continue using the product without one mistake undermining their trust in it.