HomeToGo’s North Star Metric for our Data Domain
Introduction
“Look, if you had one shot or one opportunity
To seize everything you ever wanted in one moment metric
Would you capture it or just let it slip?”
You know the author had a hard time coming up with a proper introduction when the article starts with an Eminem song text. Indeed, talking about the single metric for the broad space of data is a very hard thing to do — and still, here we are! In this article, I aim to walk you through our attempt to design a metric that:
- Indicates the overall health and productivity of the whole data domain at HomeToGo
- Is easy to understand
- And ultimately, is actionable
So, let’s snap back to reality — here’s our approach to defining our North Star data metric.
Why are you even trying?
At HomeToGo, every larger business domain has its internal overarching metric that can help us move in the right direction in the long term and is independent from quarterly goals — you can call it a domain-specific North Star metric that comes with a respective driver tree and observability elements. While data itself is a core enabler for all of these businesses, we were asked to participate in this model. Sometimes, this opportunity comes once in a lifetime, so here we go — this is our shot.
So, what’s the actual problem?
Domain diversity
While it sounds simple to define a metric for the whole data function — which is kind of our everyday job — it gets complicated when you see the breadth of roles and functions you usually have in the data domain.
- Data and Machine Learning Platform would naturally fall back to platform metrics like overall system uptime, incident response times, cost, technical debt, or customer satisfaction
- Analytical and Data Engineering will focus on metrics like pipeline reliability, technical debt, and time to insight provision
- Data Analytics and Data Science will emphasise insight accuracy, analytical coverage, and ultimately business impact and actionability
If you try to put all of this together converging into one metric, the level of abstraction will be high, and you will likely end up with some vanity metric — something we need to avoid under all circumstances.
Transaction cost overhead
One solution businesses often propose (and there are endless articles about how you could try and why you would fail anyway): just map your efforts to revenue impact. While this is tempting, whenever it comes to monetary metrics, naturally there will be fights around the nitty-gritty details on how this should work (everyone who once implemented a more innovative marketing attribution model will nod in pain and silence). While we should never shy away from a proper discussion with our peers, we did not want to put this as the base for our metric — these discussion times are hard, and they’re getting even harder, so thank you, but no.
How did you solve it?
After walking through all these options, we decided to go back to the lab again and look for a metric that would rather describe the productivity and efficiency of the data function.
Piggybacking on the definition of a data product — and its holistic definition as outlined below — and evaluating our impact by usage and cost will equip us with the certainty of well-spent resources in enabling our business:
(Source: Data Product Pyramid)
We came up with the following:
Share of Used Data Products or Data Product Usage Rate
The metric is very simple and intuitive to understand: what is the percentage of data products — be it a dashboard, table, or model — that gets used? The metric is easily expandable to make it more specific and time-bound, e.g. the share of data products used within the last 7 days. This metric is still very easy to grasp and leaves little room for different interpretations. It can easily be compared by time frames (7 days vs. 30 days), types of products (dashboards vs. tables) or domains (e.g., product vs. marketing).
Our initial goal for this metric was to achieve a 95% usage rate by the end of 2024 and never drop below 90% across the year for a 30-day rolling window.
We are now able to find a home for all our roles, functions, and domains in the analytical lifecycle:
- The Data and ML Platform Team provides foundational data services to enable the domain teams to most effectively get their work done. At the moment, we have around 30 of these core services and capabilities (and we will provide a follow-up to our initial Data Tech Stack article soon [1])
- Analytical Engineers can manage their pipelines and dbt models (and as the end result, a table with data) as information products
- Analysts can apply this to a) dashboard assets and b) analytical components like testing frameworks (e.g., A/B testing, geo split methodologies) or revenue attributions.They focus on creating contextual knowledge products
- Data Scientists build predictive and prescriptive products in the form of models
This together makes the metric very actionable, as specific (SMART [2]) goals can be specified across functions and actions can be executed the moment a product negatively influences the metric (e.g., remove the dashboard or dbt model if it’s no longer in use). This not only leads to less clutter which focuses attention on products that have proven to provide value, it also lowers cost as these can be directly associated to specific data products.
Disclaimers
First and foremost, there are valid points of critique with this metric:
- While the metric enables us to specify SMART goals, especially concerning achievability, its success ultimately depends on our customers using the data products we build. However, having this metric equips us with data to initiate discussions and understand why certain products are not in use. This enables us to take action, such as improving the product or reducing data debt by decommissioning unused assets
- The metric doesn’t directly tell us whether we are focusing our efforts on the highest business value items, which is a fair concern. Nevertheless, this risk is mitigated because we operate under the assumption that:
- We have a company-wide understanding of value derived from the company’s strategy, which is reflected in our quarterly planning and a NOW, NEXT, LATER framework
- The straightforward logic: if it is used, it is valuable
Success is our only option — failure’s not — so we assume that if it’s used, it’s valuable for decision-making. In the end, data supports business functions and aids in making decisions. For many advanced use cases, measuring the impact of decisions is challenging, so the strongest indicator we have is that people use our tools because they help them make data-informed decisions — which is the essence of our value proposition.
How do you implement this?
Okay, sounds good so far, but let’s be honest: the challenge is not in the definition itself but rather in implementing the tracking to allow for easy slicing and dicing, automated reporting, and monitoring.
Get Stephan Claus’s stories in your inbox
Join Medium for free to get updates from this writer.
Some input will still be manual, e.g., the Data and ML Platform Services, similar to the analytical components or the Data Science models, but some of them can and should be derived from metadata like dbt artifacts or Superset assets in an automated way.
Luckily, our dbt governance was quite strong already (as we connect every model run with the Snowflake internal metadata tables). What we built on top in Q1 2024 was the same ability for our Superset environment. If you are curious about how we approached this, give the great article [3] from Boris, Analytical Engineer at HomeToGo, a read.
What about results?
When we launched the MVP of this in Q1 2024, we first wanted to see where we were without any optimisation effort — and we had this distribution:
This was already decent, as the main driver for low utilisation was, as expected, Superset, where we allow our users full freedom of self-service (which usually ends up in messiness). (Yes, dashboards are dead, but still pretty useful every now and then…)
After we rolled out our governance, we quickly hit our internal SLA of 95% utilisation rate in the last 30 days within one month. Since then, we keep an eye on the metric stability in our day-to-day.
On top of that, we gain useful insights and clean-up suggestions:
- Let’s talk: If we see that a recently added data product is not used by our stakeholders, we have a perfect opportunity to discuss why and what we need to change to make it more insightful
- The metric and the ability to break it down to an individual data product allow us to keep our turf nice and tidy:
- Keep costs at a minimum, not only for the creation cost but also for maintenance
- Makes assets more easily discoverable
And what’s next?
There are many ways you can extend this idea further by exploring these directions:
Getting more granular
Why only go to dashboards and not to charts, you might ask? Or why did we exclude dbt tests — aren’t they a data asset as well? Could we add more sub-elements for an asset (e.g., only give the dbt model a count if all columns are documented)? Or why did we decide on 30 days and not shorter time frames? What can you see when you compare 30 vs. 14 vs. 7 days?
Feel free to explore — this metric allows for it. We decided against increased granularity for the first version as it would most likely only inflate the data asset base and not necessarily add more actionability.
Getting more serious — COGS anyone?
While dbt, Superset, and Data Platform data products are more easily connected to running costs, usually you do not have an automated way to add in the initial creation costs from your human resource efforts. However, with some proper labeling and automated exports in your ticketing system, you can weave this information into the equation.
Extending automated governance
A tempting (but to be handled with caution) direction is to automate certain actions once we identify missing utilisation:
- Hard governance style: Nobody used the Superset dashboard for more than 60 days? Great, let’s archive it automatically using the Superset API, inform the dashboard owner on Slack, and see who raises concerns afterward.
- Soft governance style: The dbt model was not accessed in the last 30 days, either from user or service accounts? Inform the team on Slack (using Superset 😎) and let them first reduce the scheduling frequency and ultimately archive the model.
Adding weights and attributions
Naturally, some data assets are more important than others; for example, a Data Science capability usually involves more work and provides more potential downstream applications than, let’s say, a single dashboard. On top of that, some assets have a larger downstream impact than others — e.g., our session table is used in more than 50 dashboards, so it should be reflected somehow (proper lineage as a prerequisite, of course).
While this sounds tempting and “fairer,” it comes with the natural price of transaction costs in aligning the weights across the different domains, and it would negatively impact the simplicity and intuitiveness of the metric itself.
As always, we’ll walk this road together, through the storm, whatever weather, cold or warm. So a big thank you goes out to all our data team members for their efforts to keep us lean and focused on providing an environment for data-informed decision-making.
Way to go, high five everyone.
