What does human-in-the-loop really mean in practice? (Q&A)

Using AI
1

I’m jumping into patrizia_bertini’s article today, When machines make outputs, humans must own outcomes. She digs into human-in-the-loop and why it matters more as AI speeds everything up. As machines produce more outputs, teams still have to slow down enough to judge what’s being shipped and why.

This resonates with me.

When AI output goes untested in our own team, confidence replaces learning… and let’s be honest, our decisions start moving faster than understanding.

image
image1202×534 78.5 KB

“Yet in our rush to innovate, we have convinced ourselves that deepware can carry the weight of responsibility our wetware — our human brains and nervous systems — seems increasingly willing to surrender. “

Machines can produce text, designs, recommendations, or decisions very quickly. But they don’t understand context, consequences, or responsibility. They respond to inputs and patterns. When something goes wrong, it’s never the machine that feels the impact. People do.

Let’s jump into the discussion:

Patrizia argues that even as machines produce more outputs, humans still own the outcomes.

In practice, where does that responsibility start to slip? Where do teams lean on AI output as “good enough” instead of slowing down to test, question, and learn from it?

Excited to jump in. Patrizia had a great featured article on Helio, Design metrics link creative efforts to measurable business outcomes, which inspired me to dig into her new focus on AI.

3 Likes
Upcoming Q&A schedule: Design and product leaders thinking out loud
2

The term human-in-the-loop gets thrown around a lot.

From agents to outputs… how do you think about this term, human-in-the-loop, in your work?

1 Like
3

Thank you, Bryan, for the mention and call-out! I’m really interested in learning from this community experience who is ultimately responsible when AI goes wrong.

I think there are different levels here, a broken link in a report is one thing, an AI adding a “Q5” to quarterly results (true story!) is another, and an AI declining an insurance claim is much more serious.

But is there ever a case where the AI (or the software itself) is considered responsible?

3 Likes
4

At first, human-in-the-loop made me smile :slight_smile: but honestly, it’s a critical part of any AI system. This will become even more obvious as systems evolve and we move toward a more agentic world, where multiple agents handle more tasks and take action on our behalf.

Many of those actions involve decisions that directly affect people’s lives: insurance claims being denied (like the UnitedHealth case in the US), universities accusing students of cheating (in Australia), hiring systems filtering out female candidates (Amazon), or ad systems not showing developer jobs to women (Facebook).

Machines make decisions based on historical data, not intent, context, or reality. Without human oversight to add fairness, judgment, and missing nuance, we risk creating a very dystopian world.

2 Likes
5

I think this gets to the core of your article and the discussion. As AI becomes woven into everyday workflows, responsibility can start to feel abstract or shared away.

It’s easy for it to become “someone else’s” problem.

In a corporate structure, especially in design and product, where does accountability actually sit when we’re building things that affect our customers?

6

Great point!

Design and product teams are right at the center of this shift, yet I rarely see any real discussion about what Human-in-the-Loop (HITL) actually means for them. There’s often an assumption that it will “just happen,” while ignoring the fact that someone has to design decision trees.

We used to design user flows. Now we also need to design agentic flows (how agents interact, which agent takes precedence), and where human oversight, override, and accountability sit. Machines should analyse data and surface insights, but humans should make the final decisions.

Are designers really thinking about this? I started pushing my team to do this over a year ago, and at the company I was in, there was very little interest. Are things finally starting to change?

2 Likes
7

Great point as well. In our internal design and build workflows, it feels a bit like the early 2000s again, where you spend a lot of time fixing issues because the technology is still so new. The difference is that these “bugs” aren’t always technical failures. They’re odd jumps, missed context, or non sequiturs the AI simply misses.

QA ends up sitting in an uncomfortable middle ground, caught between how AI produces outputs and how our legacy systems expect things to behave. Having good coding and design skills makes this easier to tackle.

It can feel surreal at times, like working with an incredibly advanced thinker and doer that still has very junior-level understanding of the problem.

Are designers really thinking about this? I started pushing my team to do this over a year ago, and at the company I was in, there was very little interest. Are things finally starting to change?

I think 2026 is the year, but I’m curious how you are embedding accessibility, privacy, and responsibility into these teams and workflows?

1 Like
8

it feels a bit like the early 2000s again, where you spend a lot of time fixing issues because the technology is still so new. The difference is that these “bugs” aren’t always technical failures.

You raise a very important point. AI is not an ordinary technology. It is not just a tool because it has agency. And even if it is still immature, or maybe exactly because it is immature, it needs extra care. The consequences of poorly designed systems that make bad decisions will be carried by humans, possibly forever.

When immature systems push people toward extreme outcomes, like suicide, those are not system failures. They are design mistakes done by humans.

And there is something we do not talk enough about: we do not really know what data large language models are trained on. That means hidden biases and harmful behaviours can emerge in unpredictable ways. Cases like Claude Opus blackmailing developers to protect its own existence show how strange and unexpected this can get.

This is why designing systems, including information, data, action, and decision flows, becomes critical. And this is also why it is essential to clearly define which humans are responsible for decisions.

like working with an incredibly advanced thinker and doer that still has very junior-level understanding of the problem.

I personally think of LLMs as less than a junior. I see them as hyper educated interns from Mars. They are eager to please, never contradict you, and incredibly good at producing convincing outputs. But those outputs can look great until you actually engage with them and test them.

let me get to the key question now… (one post per time :slight_smile: )

3 Likes
9

This is the key question:

We are used to working in silos, and we fail to align priorities and goals. On one side, design and product teams are pushed to deliver fast, beat competitors, and get to market first. Speed becomes the mantra.

On the other side, the compliance and legal landscape has become extremely complex. Digital products entering the EU market will need to comply with 12 EU directives by the end of 2027. These cover AI, data protection, data migration, security, system resilience, accessibility, and more.

The problem is that legal teams are aware of these requirements but struggle to turn them into concrete actions. Product teams, on the other side, are often unaware or simply uninterested.

The first step is to break down each requirement and define what it actually means at a product level. This is basically compliance driven design and product operations, and whether we like it or not it is a void none is really filling today.

In my last role, when I led a global accessibility program, this gap was very clear. Legal knew the rules but could not translate them into product actions. Product teams were focused on delivery. My role was to turn accessibility into practical steps and embed them into workflows through Jira tickets, reviews, and processes. And accessibility was relatively straightforward.

With AI, we do not have clear standards like WCAG. It is about continuously reviewing AI systems, understanding their impact on humans, identifying risks, and designing transparent systems.

If design and product teams simply started measuring bias and parity and documenting the data used to train models, we would already be in a much better place.

2 Likes
10

This is one problem I’ve seen across all the customer exchanges I’ve had (50+)…AI is causing friction because it accelerates the collaboration that needs to happen. Teams are getting stuck.

Measuring and creating benchmarks are necessary for teams to find this alignment faster, but it takes more structure than most teams have considered.

It is about continuously reviewing AI systems, understanding their impact on humans, identifying risks, and designing transparent systems.

Most of this is still based on heuristics, which is a slower process.

11

It not only accelerates collaboration, but also breaks boundaries. I know of a rather extravagant experiment where designers were asked to vibe code and hand the result to developers, developers were asked to manage the product, and product managers were tasked with designing through prompts.

You can already see the tensions and issues this creates. This is exactly how not to use AI. It creates chaos, blurs accountability, and pushes teams into confusion instead of clarity.

Most of this is still based on heuristics, which is a slower process.

This is an emergent process because it has not really been done before. But if you read the AI Act, it actually describes the risks and how to avoid them in great detail. When you read it through a design and product lens, the requirements and implications become very clear.

If humans need to know whether they are interacting with a person or a machine, design systems must provide clear visual and textual cues. If transparency is required, humans must be able to question and challenge AI outputs, which means designing systems that are bi directional and flexible. And if a human must remain in charge, designers and product managers need to clearly define ownership of decisions that affect both customers and employees.

I have been working on these topics myself. It is not easy, but it is absolutely necessary.

1 Like
12

Wow, this was a well written article! I was thinking that responsibilities and decisions change, not that it’s not AI’s responsibility or not.

Example: as a programmer, AI is now making decisions for how to build things- and this really makes sense if it’s producing high quality decisions, on smaller levels that doesn’t damage the higher level decisions that product developers have to maintain, like scalability, overall quality, maintainability, etc.

Responsibilities and decisions are shifting- on humans for macro levels, AI on micro levels, and that’s growing.

Another good example: It used to be on developers to manage garbage collection, pointers, in low-level code like assembly and even C. Which means that was something they had to take responsibility over. Nowadays (with or without) AI, I NEVER have to even think about these things.

Question: If “outcomes” require human judgment but “outputs” are what drive profit margins, how can companies be incentivized to slow down for the sake of human oversight?

2 Likes
13

@patrizia_bertini this article is soooooo good. TRULY! A couple thoughts –

  1. The framing of Mary Smith literally made my day. What an eye opening (no pun intended) way to look at this & I didn’t know that a job like this existed back in the day. So thanks for that little nugget.

  2. I’ve made it a goal of mine to refocus around outcomes vs outputs this year and this article is a reminder that accountability is key. This just feels like such a healthy reminder that accountability and ownership is SO important as we push forward, or else we’re going to be lost in a world of finger pointing….. at a screen. Independent thought is still so radically important.

The future of work, then, is not about humans competing with AI for task execution. It is about humans stepping into roles where we exercise oversight, judgement, and accountability over the outputs that AI produces. This is not replacement. This is elevation.

This was an incredible read, thank you.

4 Likes
14

@patrizia_bertini thank you for doing this Q&A!

I’m curious whether you see AI agents becoming involved in the vetting of what other AI systems produce? Will we see a world where human-in-the-loop is a system within systems, triggered by an alarm raised by other agents?

I’m also curious if you’ve encountered a good term or expression to describe the apprehension to think deeply and engage in quality control after the relieve of AI producing large outputs in seconds?

3 Likes
15

Thanks for sharing your thoughts and experience on this @patrizia_bertini. The work around AI and integrating teams isn’t easy furshure. But we’re hear to keep figuring it out. We really appreciate you bringing these insights into the Glare community!

2 Likes
16

Thanks, Ben, I really appreciate the feedback and the question, I love it!

Before getting into the question, I wanted to clarify one point you raised. You mentioned:

I see AI a bit differently here. Rather than making decisions, it suggests options and provides input, but the responsibility for accepting, adjusting, and defining the direction still sits with the human. You actually hint at this as well when you note that it makes sense if certain conditions are met.

At both a macro and micro level, decisions need to remain human-owned. If AI were truly empowered to make decisions, where would the line be? For example, could it decide to change brand colours, make a round CTA square, or other seemingly small details that still carry strategic weight?

That’s where the idea of human-in-the-loop really matters: AI can propose, but a human ensures alignment with context, intent, and vision.

On your question:

If “outcomes” require human judgment but “outputs” are what drive profit margins, how can companies be incentivized to slow down for the sake of human oversight?

I’d frame it slightly differently: outputs don’t drive profit on their own, they enable outcomes. An output by itself is just an artifact. Take code, for example: even perfectly written code creates no value unless it’s implemented, contextualised, and actually used.

Can profit be generated from code that hasn’t been integrated into a real product or experience? Similarly, can even the most polished image work without the right context?

In that sense, outcomes, not raw outputs, are what ultimately create value, and human judgment is essential in bridging the two.

3 Likes
17

Thank you @bryan for starting this conversation and the space.
These are important conversations that ar enot happening enough yet, so I am more than happy to keep this conversation and keep talking about operationalising responsible AI.

This conversation also reminded me i had a report about the AI Act for product orgs that I have posted on linkedin, which can be a nice artefact and follow up reading when thinking about how to implement responsible AI in our work in digital product design.

I cannot seem to find how to add files here, but you can get it here :slight_smile:

1 Like
18

Thank you Eric, some great points!

You ask

The challenge is that agentic AI (systems capable of taking independent actions with minimal human direction) is already emerging. This is exactly where regulation and responsible deployment become critical.

For example, it may not come as a surprise that in the US there are already weapon systems that operate without a human in the loop and can act autonomously. Similarly, many products today are exploring AI systems that manage or supervise other systems. In some cases, this can be dangerous; in others, it can be helpful for detecting issues early, as long as decision-making authority and final responsibility remain with a human.

If an autonomous weapon causes harm by mistake, that’s not just a system error, it reflects a lack of human responsibility and the absence of effective override mechanisms. On the other hand, an AI system that detects fraud or flags operational risks in order to alert a human is a very reasonable and valuable use case. What’s essential is that a human is always accountable.

So in that sense, agentic AI can be powerful and beneficial, as long as it’s clearly overseen by humans.

I’m also curious if you’ve encountered a good term or expression to describe the apprehension to think deeply and engage in quality control after the relieve of AI producing large outputs in seconds?

I have not but it seems something we need all!

1 Like
19

Thank you so much for your feedback and thoughts, @Nathalie I really appreciate them. I’m glad Mary Smith’s example helped reframe the context. I’ve hoped that putting a real human in the story could bring these issues to life more clearly than longer or more abstract explanations.

And yes, accountability really is key, not finger point, but to make sure that people aren’t negatively affected by errors or decisions made by systems without enough human care or oversight.
The risk, otherwise, is that responsibility becomes blurred and individuals end up paying the price for mistakes no one truly owns or understands.

More broadly, I think this ties into our shared social responsibility. If we’re not careful, we risk drifting further into an individualistic mindset, where we stop acknowledging our responsibility toward one another as members of the same society. This isn’t about finger-pointing… it’s about avoiding the normalisation of unfairness, bias, or harm driven by systems that can’t fully understand context, nuance, or what it means to be human.

20

I think that AI has the potential to start making those decisions- but IDK if we really want them to.

This makes sense!

Thank you for the amazing and well written responses @patrizia_bertini !