Designing with AI’s Uncertainty (Glaringly Obvious)

Using AI
1

I had a good chat with Sean Savage to talk about how AI’s unpredictability is reshaping product design frameworks. He’s full of energy…

Unlike traditional sass software, where the same input yields the same result, AI works probabilistically. That fuzziness challenges how teams define reliability, create feedback loops, and adapt their design layers.

Sean surfaced several key challenges in working with AI that teams must confront head-on if they want to keep creating meaningful value for users.

Key Takeaways:

  • AI isn’t deterministic → With AI, the same input doesn’t always give the same output. That means reliability can’t just be assumed…you have to think differently about how to measure it.
  • A new “services” layer → AI brings shifting behaviors and patterns that keep evolving. It’s not a fixed system anymore; it changes how products grow and adapt.
  • When AI improvises → AI doesn’t always play by the rules. Teams have to design for variability, catching issues early and adjusting as they go.
  • The feedback loop problem → Nobody has nailed how to handle AI feedback yet. Even the best companies are still figuring out new ways to measure, learn, and adapt.

ai-product-design2
ai-product-design21920×2400 141 KB

Sean’s take is that designing with AI isn’t about polishing pixels faster, rather it’s about building resilience into systems that will always keep shifting. His ideas pair well with Glareby measuring usability, usefulness, and enjoyment, teams can catch improvisations, spot mismatches, and see where AI-driven products move the needle for both users and the business.

:speech_balloon: Discussion:
When you think about designing with AI, how is your team handling the uncertainty it creates in feedback and product quality?

2 Likes
2

Absolutely love this insight here. Could we measure reliability?

From an engineering perspective, I see these as possible benchmark tests (where you’d run a 100 tests at a time). The hard part would be dissecting the outputs to determine whether or not some went off course.

That also would be difficult cost-wise. 100 queries from the higher language models aren’t cheap, especially when consuming images.

1 Like
3

In an AI world, that seems like 50% system, and 50% what I feel the results are?

1 Like
4

Added a followup post on LI to expand on the ideas with some more links!

5

I think so- but how do you track the system side of things?

6

Totally agree. It’s also changing how teams operate… services are no longer about delivery, but about maintaining adaptability. Every product decision now needs a feedback loop that teaches the system what to do next.

1 Like
7

I think the biggest question people struggle with (yes, even me), is how to provide that feedback loop.