Is AI really the problem in your product development?

[Bryan]

After diving deeper into our own AI projects, I’ve seen some patterns that are also confirmed by my customer outreach.

So I wrote a post on it.

AI is not what breaks your product development. It just exposes the cracks...

Going a bit deeper into research, seems there are quite a few reasons why they fail (at a 95% clip).

So what are the characteristics of the companies that figure it out? Here’s what I can pull:

• Pick one well-scoped problem (often operational rather than purely customer-facing).

• Focus on deep workflow fit (how the AI tool fits into actual user behaviour, tools, and processes).

• Build in metrics from day one. Track business outcomes, not just model performance.

• Engage partners with specialists who understand the domain and implementation challenge, rather than treating it as a generic tech project.

• Plan for scale and adaptation, with feedback loops, data-governance and getting the operation ready.

It seems to me that these problems can be addressed with a design and internal service oriented approach.

[MoData]

@Kevin_Schumacher and I were just discussing what makes an AI product stick or fail!

And @ben this is a good reminder that we need to tackle “Build in metrics from day one” on our own tool, which we realized last week too.

[nathaliesmith]

@Bryan

Build in metrics from day one. Track business outcomes, not just model performance.

Thissssss one

[ben]

We should definitely map this out more!

[Bryan]

I think we can align our business outcomes with the specific user needs. Our traditional UX metric stack still serves as a leading indicator- but intelligence metrics are needed for more understanding.

[ben]

Ah- we should do a better job at connecting the pieces together. Knowing how to measure AI tools is cool, but knowing how to use the right measurements that align to business goals? That’s what we want.

[EricZ]

@ben let’s put together a way to think about this for a post!

[EricZ]

@ben and I had a great convo about Intelligence metrics yesterday! I’ve articulated that thinking in our post here:

Intelligence Metrics: The User vs. Product Framework

Last week, @Bryan had posted a framework of UX Metrics, revealing the need for Intelligence metrics to address user interactions in AI product space. [image] So we took a stab at how we’re thinking about Intelligence metrics broadly. When developing intelligence metrics, we must address the multifaceted ways users engage with LLMs. By mapping the lifecycle of a prompt (Input and Output) against two distinct perspectives (User and Product), we identify four necessary quadrants for measurement…

[Doug_Curtis]

love this thread! I 1000% agree that AI is an amplifier to a person’s abilities. So therefore it will accelerate and expose problems way faster than in the olden days. Before AI it could take a team an entire quarter or more to ship a feature.

Now you start hitting “problems” even faster which is great, it may feel like AI broke the process but you’re totally right Bryan that it is just exposing existing problems that would go unnoticed before. There was so much slack in the “product building system” that you can no longer establish causal relationships, you’re only left with trying to correlate things to understand what went wrong. Overall AI has a real opportunity to make teams work way more efficiently.

Ben, Eric, you guys are on to something about how business metrics and user outcomes are now more important than ever. I think that’s definitely a byproduct of the acceleration that AI provides to learn fast, all stakeholders now should have business goals in mind as their North Star.

[Bryan]

@Doug_Curtis

There was so much slack in the “product building system” that you can no longer establish causal relationships, you’re only left with trying to correlate things to understand what went wrong.

I think that’s the biggest problem, because without user feedback and data, you’re left spinning in circles trying to figure out what is working. Even the larger LLMs are facing this problem with each new point release.

[Bryan]

Jumped back into this topic…

When AI product development can ship faster than trust can be built, frustration can build. 2026 is going to be more challenging for builders than any previous year, as @schuboxaz can attest from his research. Trust is built on truth.

People decide whether to trust AI results almost instantly. Trust starts...

[Bryan]

What AI has done is make production cheap and fast.

Flows, screens, copy, research summaries, and even strategy artifacts now appear instantly. In product and design, most teams optimize what’s easiest to see.

I don’t see that inherently as a problem in product development. The issue is whether design decides to take an active role in shaping the evolution of a product.

The top design skill now is calling out BS. AI has made production cheap and...

Glare should help teams take ownership of these challenges.