# Design has shifted from control to stewardship (Q&A) **URL:** \ **Category:** Using AI\ **Created:** [January 18, 2026, 4:03pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599 "2026-01-18T16:03:39Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Bryan](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/bryan/32/448_2.png) [@Bryan](https://forum.zurb.com/u/Bryan)\ **Post date:** [January 18, 2026, 4:03pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/1 "2026-01-18T16:03:39Z")
We’re jumping into @Sean_Savage’s article today, [From playwright to stage manager](https://uxdesign.cc/ai-improv-34a48bd4b17b). He argues that AI design is no longer about defining screens and flows. Design is about guiding behavior over time. Designers need to shift from controlling outcomes to setting boundaries, monitoring systems, and stepping in when things drift. The job is more than just producing good interfaces… applying human judgment to how AI behaves is the new focus. ![image](https://canada1.discourse-cdn.com/flex034/uploads/helio/original/1X/842d6ed67dda6f820a3090e49580b33470aff804.jpeg) So I was interested in how Sean uses improv theater as the core metaphor. Traditional software design is like writing a play. AI systems are more like improv performers. They respond, adapt, and sometimes go off script. **Designers should stop being playwrights and become stage managers.** Their job is to shape the environment, define boundaries, and intervene when things drift. ### Let’s jump into the discussion: If the idea is about control without micromanagement, it means designing conditions that lead to better behavior with clearer intent. This includes guardrails, feedback loops, and moments where humans step back in. **So my first big question.** When systems feel familiar, like desktops or chat interfaces, we assume they behave like old software. But they do not. Treating them like deterministic tools is risky. What is the first thing leaders need to do get their team started in the right direction? Let’s dig into setting up metrics with [Sean J. Savage](https://www.linkedin.com/in/seansavage/). Sean is a [featured Helio author](https://www.linkedin.com/posts/helio_uxresearch-productdiscovery-marketresearch-activity-7347301867057516544-iuIt) and has a fun [Glaringly Obvious chat designing with the uncertainty of A](https://glare.helio.app/interviews/sean-savage#designing-with-ai-s-uncertainty)I. Looking forward to finding some answers! ---
**Author:** ![Bryan](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/bryan/32/448_2.png) [@Bryan](https://forum.zurb.com/u/Bryan)\ **Post date:** [January 18, 2026, 4:05pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/2 "2026-01-18T16:05:41Z")
Excited to jump into this topic Sean, thanks for joining us. I’d love people to ask questions too. Here’s my first question: **If designers are now stewards of behavior, not controllers of outcomes, what is the first responsibility leaders must redefine for their teams?** ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 4:11pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/3 "2026-01-18T16:11:29Z")
Thanks @Bryan! And thanks for inviting me. I’d start with: **ownership doesn’t end at ship.** In traditional software, you hand off specs, build, validate, ship, move on. At some point you measure and circle back, but ownership is bounded by the release cycle. With AI, shipping is the start of the job. The system behaves differently tomorrow than today—not because of bugs, but because that’s how probabilistic systems work. Someone needs to own watching how it actually performs, identifying drift, and refining guardrails. That’s not QA. That’s ongoing design. The hardest part isn’t process—it’s identity. Designers tie their value to _making new things_. Stewardship means your value also comes from _tending what exists_. Leaders need to name that and reward it. ---
**Author:** ![Bryan](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/bryan/32/448_2.png) [@Bryan](https://forum.zurb.com/u/Bryan)\ **Post date:** [January 18, 2026, 4:15pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/4 "2026-01-18T16:15:45Z")
> [@sean\_savage](#): > > Stewardship means your value also comes from _tending what exists_. Leaders need to name that and reward it. Love this. In our own product-making experiences, this takes practice and a new way to work together. **How do teams measure how well their outputs are working? How has your team done this?** ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 4:16pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/5 "2026-01-18T16:16:59Z")
A movie wraps and ships. An improv troupe returns tomorrow night. We’re going from something more like a movie production to an improv theater. ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 4:18pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/6 "2026-01-18T16:18:40Z")
In improv, troupes hold “notes sessions” after each show. They don’t just ask “did the lights work?”—they ask “did the scene land? Did we miss an offer?” That’s the measurement shift. Traditional QA asks: did it work or break? AI evals ask: _how reliably_ does it work? When it fails, does it fail gracefully? Does the user recover or abandon? .. ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 4:20pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/7 "2026-01-18T16:20:15Z")
Thats where getting designers and domain experts involved in AI evals comes in. Not just looking at dashboards, but reviewing patterns in actual interactions. You start seeing the patterns. Then you ask the stage manager question: is this a props problem, a stage problem, or a rules problem? In improv theater the coach watches the show and then helps the performers and stage design improve. It starts with watching the show. ---
**Author:** ![Bryan](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/bryan/32/448_2.png) [@Bryan](https://forum.zurb.com/u/Bryan)\ **Post date:** [January 18, 2026, 4:21pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/8 "2026-01-18T16:21:47Z")
> [@sean\_savage](#): > > That’s the measurement shift. Traditional QA asks: did it work or break? AI evals ask: _how reliably_ does it work? When it fails, does it fail gracefully? Does the user recover or abandon? .. Right, so it seems there are some core differences in how we measure success. How strong is the audience’s laugh, and how soon do they laugh again? > [@sean\_savage](#): > > Not just looking at dashboards, but reviewing patterns in actual interactions. You start seeing the patterns. Then you ask the stage manager question: is this a props problem, a stage problem, or a rules problem? So for now, it’s mainly heuristics based on evaluations. **What key action, heuristic or metric would you recommend teams focus on when getting started?** ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 4:33pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/9 "2026-01-18T16:33:11Z")
RE: “it’s mainly heuristics based on evaluations.” – I think we’re on the same page but to be sure: We definitely don’t want to throw out our core UX research toolbox. The classic qual and quant research tools are still important. Evals are additions to the toolbox, not replacements. re: “ **What key action, heuristic or metric would you recommend teams focus on when getting started?”** To be clear: This is new ground and I’m still very much learning this myself. But it seems the most useful practice is: review session traces. Not just analytics dashboards—actual logged interactions. Hamel Husain and Shreya Shankar call this “error analysis”—you sample real user sessions, write critiques, then group failures into patterns. It’s the post-show notes session, systematized. ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 4:34pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/10 "2026-01-18T16:34:47Z")
Hamel Husain and Shreya Shankar are the best experts I know of around how product teams can build and use evals. I recommend checking out their posts and course. > **[Building eval systems that improve your AI product](https://www.lennysnewsletter.com/p/building-eval-systems-that-improve)** > > A practical guide to moving beyond generic scores and measuring what matters ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 4:36pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/11 "2026-01-18T16:36:52Z")
One insight from their work that resonates with the stage manager role: designate a single “benevolent dictator” as the arbiter of quality. That’s often a domain expert in whatever realm you’re building products for. In theater terms, that’s your director—one clear voice on what “good” means. One surprise for me was: using binary pass/fail judgments work a lot better than fuzzy 1-5 scales. The nuance lives in the critique, not the score. And the binary choices speed the process. When you discover a pothole or speedbump you ask the stage manager question: is this a props problem, a stage problem, or a rules problem? That diagnosis tells you where to focus. ---
**Author:** ![Bryan](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/bryan/32/448_2.png) [@Bryan](https://forum.zurb.com/u/Bryan)\ **Post date:** [January 18, 2026, 4:42pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/12 "2026-01-18T16:42:07Z")
> [@sean\_savage](#): > > re: “ **What key action, heuristic or metric would you recommend teams focus on when getting started?”** > To be clear: This is new ground and I’m still very much learning this myself. But it seems the most useful practice is: review session traces. Not just analytics dashboards—actual logged interactions. Hamel Husain and Shreya Shankar call this “error analysis”—you sample real user sessions, write critiques, then group failures into patterns. It’s the post-show notes session, systematized. Right, there aren’t tons of examples of this in the wild. [We’re leaning into it](https://forum.zurb.com/t/intelligence-metrics-the-user-vs-product-framework/417). So the review concept is like **Working Backwards** , a problem-solving strategy where you start with the expected final result and reverse the steps, using (like subtracting instead of adding, or dividing instead of multiplying) to find the original starting point or conditions. It brings me back to the days of using algebra and word problems. Inversion I think is the proper term. **Only this isn’t quite right, because the outcomes are highly variable.** So who on the team decides the best outcome? Clearly the user needs should be met, but there’s also a business goal that the executive team wants to see achieved. ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 4:47pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/13 "2026-01-18T16:47:26Z")
> [@sean\_savage](#): > > One insight from their work that resonates with the stage manager role: designate a single “benevolent dictator” as the arbiter of quality. That’s often a domain expert in whatever realm you’re building products for. In theater terms, that’s your director—one clear voice on what “good” means. One surprise for me was: using binary pass/fail judgments work a lot better than fuzzy 1-5 scales. The nuance lives in the critique, not the score. And the binary choices speed the process. > > When you discover a pothole or speedbump you ask the stage manager question: is this a props problem, a stage problem, or a rules problem? That diagnosis tells you where to focus. I love that. Working Backwards is akin to Backpropagation, one of the key underpinnings of how today’s LLMs are trained [Backpropagation - Wikipedia](https://en.wikipedia.org/wiki/Backpropagation) ---
**Author:** ![Bryan](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/bryan/32/448_2.png) [@Bryan](https://forum.zurb.com/u/Bryan)\ **Post date:** [January 18, 2026, 4:53pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/14 "2026-01-18T16:53:38Z")
> [@sean\_savage](#): > > When you discover a pothole or speedbump you ask the stage manager question: is this a props problem, a stage problem, or a rules problem? That diagnosis tells you where to focus. [Chatting with Geoff Gibbon’s on LinkedIn about “thinking about thinking”](https://www.linkedin.com/feed/update/urn:li:ugcPost:7418092772316041217?commentUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A7418092772316041217%2C7418685352728678400%29&replyUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A7418092772316041217%2C7418692225619877888%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287418685352728678400%2Curn%3Ali%3AugcPost%3A7418092772316041217%29&dashReplyUrn=urn%3Ali%3Afsd_comment%3A%287418692225619877888%2Curn%3Ali%3AugcPost%3A7418092772316041217%29) He’s got a framework for USERS to think about the outputs they get and figure out how to evaluate the value of prompts. ![image](https://canada1.discourse-cdn.com/flex034/uploads/helio/original/1X/cab47d48f78c3378fee53fcdcd50b7df44ef1eab.jpeg) I don’t know if there are many frameworks that help teams figure out the value of the outputs. We use a heuristic technique, which has worked ok, but there’s a lot of interpretation from a single perspective. ![image](https://canada1.discourse-cdn.com/flex034/uploads/helio/original/1X/79ba368b6199adfe21ec0e1dfb8749b30c88f56b.png) **Do you have recommendations for principles or frameworks for what is a good output?** ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 4:54pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/15 "2026-01-18T16:54:22Z")
> [@Bryan](#): > > **Only this isn’t quite right, because the outcomes are highly variable.** So who on the team decides the best outcome? Clearly the user needs should be met, but there’s also a business goal that the executive team wants to see achieved. That’s exactly where the “benevolent dictator” comes in—but who should that be? It depends on the domain. If you’re building tools for pediatric oncologists, you want a pediatric oncologist reviewing traces—someone who understands the context, challenges, and risks users actually face. Building for kindergarten teachers? Bring in a kindergarten teacher. RE: Business goal - this brings us to the classic challenges around connecting user research to business goals and getting stakeholder buy in (see: [It's Our Research: Getting Stakeholder Buy-in for User Experience Research Projects: 9780123851307: Computer Science Books @ Amazon.com](https://www.amazon.com/Its-Our-Research-Stakeholder-Buy/dp/0123851300) ) - but tldr = if you bring in a kindergarten teacher to make sure from the ground up the product makes kindergarten teachers’ lives easier, the business will have a much easier time selling to kindergarten teachers. Barring direct access to a domain expert, tap the user researcher who’s spent months studying the users, or the support rep who’s logged hundreds of hours on calls with customers. Barring that - on a small team with a single designer, it may fall on the designer to interview, research, understand the audience and their context and bring that to the table, acting as evals decider until the business grows to the point where it can bring in a representative user. The common thread: you want a human who deeply understands the domain, not just someone who understands the technology. (DEFINITELY this is not traditional software QA, owned by the engineering team.) In theater terms, you wouldn’t hire a lighting technician to judge whether the emotional beats landed—you’d want someone who understands the audience. ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 5:05pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/16 "2026-01-18T17:05:41Z")
“STOP” framework feels like a useful approach! RE your heuristic… thats strong! Reminds me of a framework i developed over the years for reviewing user feedback in my qual user interviews/usability tests/walkthroughs, to help build a bit more structure into fuzzier/more freeform user research. (NOTE this was from pre AI-product days, this is NOT an AI eval but user research summaries) Looks a bit like this: ![image](https://canada1.discourse-cdn.com/flex034/uploads/helio/original/1X/c8250f975c6e0b4e532b97f6a46c1e41edf98bb4.png) ---
**Author:** ![Bryan](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/bryan/32/448_2.png) [@Bryan](https://forum.zurb.com/u/Bryan)\ **Post date:** [January 18, 2026, 5:12pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/17 "2026-01-18T17:12:17Z")
> [@sean\_savage](#): > > The common thread: you want a human who deeply understands the domain, not just someone who understands the technology. (DEFINITELY this is not traditional software QA, owned by the engineering team.) In theater terms, you wouldn’t hire a lighting technician to judge whether the emotional beats landed—you’d want someone who understands the audience. Ok, last question, and this probably gets to a lot of the struggle that many teams face. Not everyone is good at this type of AI work, and taking people who have been domain experts in their respective fields doesn’t always translate to this new work style. Many teams are stuck, trying to bring everyone along for the ride. But it’s just not going to work in most scenarios. A great movie star doesn’t always do well in improv. How do you see this playing out? ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 5:17pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/18 "2026-01-18T17:17:52Z")
…I think where you could take your framework above, or my framework above, is: adapt that to form a structure for evals. The human domain expert helps to define this/fill out such a matrix that sets forth the most import assessment items to watch for, pros and cons to assess, etc. Then the expert manually scores many sessions using it (and in my experience: the person building the assessment framework tends to update/tweak it as they go - they discover new “gotchas” and things to look out for as they evaluate the examples), then when they reach something close to a steady state (ie the ”what success looks like” framework is pretty stable), you systematize that so an automated AI “judge” can use it, apply the same evaluation framework en masse to thousands of sessions and flag obvious breakages and obvious standout successful sessions. Then the human reevaluates the standout sessions from that point, as well as a randomized set of other sessions, as a repeated (weekly?) process. Continually you evolve the AI product - and evolve the eval framework itself too. ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 5:20pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/19 "2026-01-18T17:20:52Z")
Another important note: PRIVACY. I haven’t touched on that at all. But note: we’re talking about reviewing individual user sessions. So part of the evals process is to first apply processes that strip sensitive and personally identifying info from sessions before human review. That’s a whole other topic that Husain/Shankar go into. ---
**Author:** ![sean\_savage](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.zurb.com/sean_savage/32/922_2.png) [@sean\_savage](https://forum.zurb.com/u/sean_savage)\ **Post date:** [January 18, 2026, 5:27pm UTC](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599/20 "2026-01-18T17:27:36Z")
> [@Bryan](#): > > Ok, last question, and this probably gets to a lot of the struggle that many teams face. > > Not everyone is good at this type of AI work, and taking people who have been domain experts in their respective fields doesn’t always translate to this new work style. Many teams are stuck, trying to bring everyone along for the ride. But it’s just not going to work in most scenarios. A great movie star doesn’t always do well in improv. > > How do you see this playing out? Great questions. First thing is: Take heart because (almost) NOBODY is good at this type of AI work yet! Even at the top foundation model labs they’re still struggling to carve out these new practices and new hybrid types of work. That points to tremendous opportunity for designers, PMs and founders who want to dive in and trailblaze. Beyond that – honest answer: not everyone will make the transition, and that’s okay. Many brilliant actors realize improv isn’t their thing. They prefer a script. That’s not a flaw—it’s a different craft. The same is true here. Some designers thrive on pixel-perfect control and deterministic flows. And that’s still needed - we’re not entering a world where deterministic software goes away, we’re entering a world where humans and probabilistic AI use deterministic software. Don’t try to convert everyone. Identify who’s energized by ambiguity versus who’s drained by it. Some people light up when you say “we can’t predict exactly what this will do.” Those are your stage managers. Let people find their roles in the new production. [Next page](https://forum.zurb.com/t/design-has-shifted-from-control-to-stewardship-q-a/599.md?page=2)