We’re jumping into some Q&A on @byhuber’s article, The Science of Design. He argues that design breaks down when designers rely solely on confidence rather than evidence. The best designers test their thinking rather than trying to defend it.
The bigger question is whether design should stop leaning on gut feel and start acting more like science. Too much design work still rests on assumptions, habits, and familiar patterns rather than evidence of what actually works.
Dave points out that designers already operate this way in practice. They form hypotheses, test ideas, observe outcomes, and learn. The gap is that many teams do not frame their work this way or apply the rigor of structured experimentation.
When designers guess, they often solve the wrong problem or optimize the wrong thing. That search for design impact is something many of you have shared with me in our email exchanges, and it’s exactly the gap Glare is trying to help close.
Let’s jump into the discussion:
Most of us can agree that design works best when it’s grounded in learning.
If designers already test and learn, what gets in the way of treating design as a true learning practice? Where does confidence start replacing evidence in design work?
Let’s dive into some questions @byhuber- for those working in larger organizations like IBM, what are the biggest things getting in the way of design doing experimentation?
Curious how design departments can engage in broader exploration in design when so much focus is placed on meeting initiative deadlines and bringing alignment to stakeholder hunches?
Love this! My question is, (from a non-designer) Is it easier to ship something than to admit the data is unclear? And what do teams risk by doing that?
Infrastructure
Experimentation—especially regarding product features rather than just copy—requires substantial infrastructure, such as “feature flagging.” This becomes exponentially more challenging in regulated industries. Much of this testing occurs on the back-end or during sensitive phases like user signup. Consequently, decisions about data transmission and user permissions (e.g., registered vs. unregistered) require consensus across departments that rarely collaborate, such as legal, security, growth, and core engineering.
Training
Implementing experiments that span multiple touchpoints requires cross-functional coordination, yet training these diverse teams simultaneously is difficult. While organizations often invest heavily in learning and development, these efforts rarely engage cross-functional groups together. Without shared training, teams lack a common language and methodology for experimentation.
Culture
Building an experimental organization requires a fundamental shift in values. In traditional organizations with deep-seated hierarchies, asking “Is that true?” or “How do we know?” can be culturally abrasive. Transitioning from a top-down decision-making style to one that embraces questioning and data-driven challenge is often underestimated. You are asking for a culture where evidence outweighs rank, which is a significant departure for many established companies.
Incentives
Incentives and policies are the mechanisms that enforce culture change. To encourage experimentation, organizations must ask: Is “experimentation proficiency” part of the annual review? Are there enterprise-wide competitions? Do we offer awards for the best experiment? Research suggests that to foster this behavior, you must reward the act of experimenting, not just the outcome. Since many experiments will not result in immediate value, penalizing “failed” experiments discourages risk-taking. Incentivizing the rigorous pursuit of answers—regardless of the result—is key to finding the wins that actually matter.
Good question @EricZ - this is often a leadership issue.
Experimentation goes against our human nature (confirmation bias, group think, traditional concepts of leaders having all the “answers”) and so establishing it as part of the culture is important.
This can be done by sharing external success stories (social proof is immensely persuasive and lots of organizations have stories they brag about), building allies with leadership and advocating for experimentation privately to get them to endorse it publicly, and finding a small win internally to stand on to show that is can happen *here.
*
I recommend this book full of great stories of experimentation told from the perspective of business (more than design) that can meet your colleagues where they’re at.
This question hints at an important nuance when it comes to building a culture of experimentation: *what’s worth experimenting?
*
Not everything needs an experiment. Sometimes shipping is the experiment. Sometimes what we really need is just honest data that validates or invalidates our hypotheses.
I like thinking of maturity curves for a lot of organizations practices. It’s like the innovation adoption curve: a little matter of fact. Organizations have to move their way up the “maturity curve” of experimentation. I believe that starts with understanding why we experiment, what’s worth running experiments on, and how did we quickly test our hypotheses.
Years ago, teams risked a lot by shipping without testing their hypotheses, or having unclear data. But today, many teams are vibe coding prototypes rapidly and putting real working products in front of customers the same day to test their ideas.
Agreed here. Let’s dive deeper into this idea. Even with buy-in, there’s some level of uncertainty that tightens teams up when they ultimately have to ship code. Is this culture or process…
How do you decide when shipping is the experiment versus when unclear data is a warning sign? What signals tell you it’s safe to move forward versus slow down and learn more?
Oh, love this graph. So, in the context of your ladder diagram that we featured on Helio, would this experimentation mostly happen during testing for user outcomes?
Love this answer so much and it totally makes sense.
But today, many teams are vibe coding prototypes rapidly and putting real working products in front of customers the same day to test their ideas.
This is a total stand out to me and I see this internally so much, its soooo much easier to experiment quickly without “wasting” time / resources. It seems to be a serious plus side to the AI layers in the work. @ben is a wizard with this type of work.
“Dave points out that designers already operate this way in practice. They form hypotheses, test ideas, observe outcomes, and learn. The gap is that many teams do not frame their work this way or apply the rigor of structured experimentation.”
@byhuber do designers really come up with hypotheses and experiment like this? It’s not something I’ve noticed, but then again we work with teams that come to us because they want us to do the experimenting for them
