Product Metrics That Matter: From North Star to AI Features

Key Takeaways

• A north star metric should capture the value customers get, not the revenue you collect, and it should stay stable for a year or more.

• A metrics tree breaks the north star into drivers and then into input metrics that a single team can move within weeks, each with a clear owner.

• Define activation as a specific action and check that users who reach it retain better; read retention by cohort so growth does not hide churn.

• If a number changing tomorrow would not change what you do, it is a vanity metric: fine for context, harmful as a goal.

• AI features need four extra lenses beyond adoption: output quality, cost per successful task, latency including slow outliers, and user trust.

Most product teams do not suffer from a lack of data. They suffer from too many dashboards, each with a different favourite number, and no shared agreement on which of those numbers should change a decision. The result is weekly reviews full of charts and very little learning.

Good metrics do three jobs. They describe whether customers are getting value, they point the team at the levers it can actually pull, and they warn you early when something is breaking. This guide walks through a simple structure for doing all three, and then adds what changes when your product includes AI features.

Start With a North Star Metric

A north star metric is a single number that best reflects the value customers get from your product. It is not revenue, although it should lead to revenue. Revenue tells you that value was captured; the north star tells you that value was created.

Qualities of a good north star

• It reflects customer value: if the number goes up, customers are genuinely better off.

• It is a leading indicator: it moves before revenue and retention do, so you can react in time.

• It is understandable: anyone in the company can explain it in one sentence.

• It is influenceable: teams can connect their daily work to it through a small number of inputs.

Examples help. A learning platform might choose "learners who complete a practical project each week." A team collaboration tool might choose "teams that finish a shared workflow each week." Notice that both count an outcome, not an activity like logging in.

Common north star mistakes

The two most frequent mistakes are picking a metric that is easy to count rather than meaningful, and changing the north star every quarter. A north star should be stable for a year or more; the inputs underneath it are what change.

Input Metrics and the Metrics Tree

A north star is too broad for any single team to own. The fix is a metrics tree: a simple diagram that breaks the north star into the drivers that produce it, then breaks those drivers down again until each branch ends in something a team can influence directly.

Building a metrics tree step by step

1. Write the north star at the top.

2. Ask "what has to be true for this number to grow?" and list two to four drivers, for example new active users, retained users and depth of use.

3. Repeat the question for each driver until you reach metrics a team can move within weeks, such as onboarding completion or time to first result.

4. Assign an owner to each leaf and agree on the target direction, not just a number.

5. Review the tree quarterly and prune branches that no longer explain changes in the north star.

The tree also protects you from local optimisation. If a team pushes its leaf metric up and the parent does not move, the tree makes that visible and starts the right conversation.

Activation, Retention and Engagement

Almost every metrics tree ends up containing some version of these three. They are worth defining carefully, because vague definitions produce arguments instead of insight.

Activation

Activation is the moment a new user first experiences the core value of the product. Define it as a specific action, not a feeling, and validate it: users who hit the activation event should retain noticeably better than those who do not. If you are still searching for that moment, the methods in validating ideas before you build apply here too.

Retention

Retention answers whether people come back and keep getting value. Look at it by cohort, grouping users by the week or month they started, so that growth in new users does not hide a leaky bucket. A flattening retention curve is one of the clearest signals of product-market fit.

Engagement

Engagement measures depth: how often and how meaningfully people use the product. Choose a frequency that matches the natural rhythm of the job. A tax tool used once a year is not failing because it lacks daily users.

Vanity Metrics to Avoid

A vanity metric looks impressive but does not help you decide what to do next. The test is simple: if this number changed tomorrow, would you do anything differently? If not, it is decoration.

• Cumulative totals such as total sign-ups or total downloads, which can only go up.

• Raw page views or sessions without any link to a completed outcome.

• Follower counts and impressions reported without conversion to real usage.

• Feature usage counts that ignore whether the feature solved the problem.

Vanity metrics are not always useless; they can be fine for context. The danger is using them as goals, because teams will find ways to inflate them.

To keep the distinction clear, here is how the main types of metrics compare in practice:

A metric earns its place on the dashboard only when someone knows what they would do if it moved. — TechSkill instructors

Metrics for AI Features

AI features behave differently from traditional features. The same input can produce different outputs, every call has a direct cost, and users decide quickly whether they trust the result. Adoption numbers alone will not tell you whether an AI feature is working. Add four lenses:

Quality

Define what a good output looks like for the task, then measure it with a mix of automated checks, a regularly reviewed sample graded by people, and user signals such as accepted, edited or regenerated results. A high regeneration rate is often the earliest quality warning.

Cost

Track cost per successful task, not just total spend. A feature that costs a little per call but needs several retries to get a usable answer may be more expensive than it looks, and costs can grow faster than usage if prompts get longer over time.

Latency

Measure time to first useful output and time to complete result, and look at slow outliers, not only the average. Users forgive a few seconds for a long document; they do not forgive it for autocomplete.

Trust

Trust shows up in behaviour: how often users verify or override outputs, whether they return to the feature after a bad result, and how many issues are reported. Pair these with short in-product feedback. Trust is slow to build and quick to lose, so watch it closely after every model or prompt change.

If AI features touch sensitive data, include safety signals in the same view; our AI data security playbook explains what to watch.

Putting It Into Practice

Start small. Agree on a north star, sketch a first metrics tree on one page, pick no more than five input metrics to review weekly, and add the four AI lenses for any feature built on a model. Revisit the whole structure every quarter. Fewer metrics, clearly owned and tied to decisions, will teach your team more than any dashboard wall.

Choosing and defending the right metrics, especially for AI products, is a central theme in our strategy programme for product leaders.