Mike’s core point is that metrics should start with intent.
Before you measure anything, you need to be clear about what outcome you are trying to change, why it matters, and who it matters to. Without that, metrics become noise, dashboards grow, and teams still argue.
Another big idea is that metrics are tools for learning… we shouldn’t use them for scorekeeping.
→ Good metrics help teams ask better questions, spot tradeoffs, and adjust direction.
→ Bad metrics create pressure to “hit the number” even when it leads to the wrong decisions.
Let’s jump into the discussion:
He frames a good metric as:
Measurable without mixing too many things
Tied to a real outcome or step
Shows what to do next
Easy for others to understand
Worth keeping and using over time
So here’s the question I want to explore:
Your article focuses on setting up product metrics, and I know many of us are trying to create business impact. How do you work with stakeholders to agree on the right metrics in the first place?
Great to chat with you today Mike! I enjoy your no-frills approach to getting to product answers.
Here’s my first question: When you’re aligning on metrics with stakeholders, what’s harder: agreeing on the outcome, choosing the signal, or trusting the metric enough to act on it?
Hello, nice to be here! Thanks for featuring the article and kickstarting the session with a question!
When you’re aligning on metrics with stakeholders, what’s harder: agreeing on the outcome, choosing the signal, or trusting the metric enough to act on it?
Aligning with stakeholders usually means having a discussion so we have the flexibility to figure things out. Still the hardest part is aligning on the outcome. To make this work we usually have enough initial metrics (what’s currently going on? what’s the status today?) to build a story around the expected outcome; so how the story we have around the feature aligns with the expected goal.
Ok, so the idea is that we’re going through this metrics exploration before aligning with Stakeholders. Is the idea that you are going into these conversations before the outcomes have been baked?
There are cases where we’re talking about a smaller feature or improvement where we already have data for but not a solid story to start with. For those cases we usually view the metrics and peripherals internally with the team before talking to more stakeholders, so that we have an idea for the outcome. I recall such a case, where we wanted to improve usage/playability of a specific content type at HackTheBox (HTB) and we internally figured out what’s the metric in question, then extrapolated some data from other HTB content with “good usage” to propose a specific outcome with our key stakeholder. We then met, shared the reasoning and data gathered and worked together to define the final goal.
There are other cases where things are more unshaped where we need collaboration with more stakeholders into shaping the proper metrics and outcomes together. That was the case on building the north star metric we use at HTB. In that case, we shaped the initial questions and ideas with more stakeholders and figured out the steps together before reaching to the final outcome. In that case the outcome alignment was easier as we built the story together.
So this idea of using data to help tell stories…stakeholders are looking for product and design leaders to drive these metric conversations, but they can be tricky because they have their own business metrics. This might be a simple question… but how many product metrics should we talk about?
Again a true example from HTB, we have a north star story consisting of 5 metrics that form a funnel of the overarching story/question: “Are users using the platform?” Each step/metric is fully actionable by more than one teams - so not only Product or Platform are responsible for a metric; Customer Success, Sales, Marketing, and Customer Support, even the team building our HTB VMs are needed depending on the step!
Now there’s a deeper level, where you pick a specific step from the north star funnel to check more details. That’s another dashboard with a different set of metrics drilling down more details/info. Again on that level we’re trying to keep the metrics between 5-7.
Truth be told, we didn’t have the time to engage with user research and sessions on these metrics so far. Still, we’re having research sessions with users that are based on specific scripts of questions.
What we’ve done in the past to back up metrics with qualitative feedback is build a table list with each row having items for:
Metric of interest, what we interested in measuring
Question to ask, what’s a good question to cover this metric
Positive trigger words, phrases in an answer that are signaling a +1 around that metric
Negative trigger words, phrases in an answer that are signaling a -1 for the metric
Reasoning of relation between question/metric/triggers, as this list grew it was hard to recall everything so we needed to properly document so that we evaluate findings after the interview.
With that in place and a few sessions ran we then got back to build the outcomes table from all sessions. If I recall correctly it didn’t quite go as planned as it’s difficult to quantify an interview back to a metric leaving any discussion nuances on the side.
This is a problem many product and design leaders have, and a perfect use of UX metrics to get fast, quick feedback to align with their product metrics. It’s what we’re helping people accomplish with Glare.
That’s a tough one! So far in my experience we always had a feedback mostly triggered by some form of incoming user feedback, so we know that we’re working on solving a real problem. Usually a part of that feedback holds some insights on expected usage, thus and expected metric to use - like having a part of the initial question from the article directly from the user.
That’s our premise and story for building a metric for that feature.
Additionally, since we always base our roadmap and work on incoming features it means that there’s some desirability behind each request. This means we’re having incoming feedback post release from Customer Success/Support or direct feedback channels that we consider as a validation or not of any work done. I’ve got examples on this one:
Just today we got some user feedback on a feature released yesterday, validating the UX and happiness of the user. That one came through Customer Success.
For the last month we’ve got incoming feedback on a design decision we’ve made and we weren’t pretty certain on how to move next. That also came from Customer Success and is driving an improvement we’re working on.
@mike_giannakopoulos What was the process like to identify key product metrics, and does the team have a shared reference / database of these metrics or is it just shared negotiation? I’m wondering how the team develops a shared language and understanding in these negotiations.
This is a great answer, thanks for sharing it. I like how you anchor metrics incoming feedback and treat customer signals as the starting point.
Using Customer Success and Support is a great way to create an ongoing validation loop that feels both practical and grounded (especially when deeper qualitative work isn’t always possible).
I appreciate you taking the time to respond and share concrete examples @mike_giannakopoulos. Thank you! We’ll keep this thread open for others who want to build on this or share how they handle similar constraints in their teams.
I’m the one usually kickstarting the exploration for key product metrics. Depending on the level of product metric we’re talking (High, Mid, Detailed) I try to iterate with more people sooner or later. For high level metrics I do that pretty early when I have a rough idea in place, for mid a bit later so that the idea is more refined, and with detailed ones I spend some more time investigating before I engage people.
Still the most important aspect is building a story to share your metric. A story has the benefit that the metric sticks and resonates. Also, a story is easier to follow and contribute to. Finally, with the story in place, and if we’re talking about an important metric (High or Mid) we build an accompanying doc with some definitions, or embed the definitions in the created dashboard.
It seems with this approach you develop pattern and precedent initiative by initiative, which no doubt helps to eliminate ruts and stale thinking. Have you noticed certain stacks or pairings that help identify key user needs in your stories better than others? How have you seen this practice develop over time?
Of course there are such cases. When this happens I try to have a doc with all the reasoning in place for more people to see when they see fit. As the goal is to always move forward, we proceed with the metric & story; whenever there’s an occasion to share the story and metric - due to a change in behavior - we share a story around why we see this behavior and how this metric helps us understand it.
