How to Define KPIs and Metrics That Mean the Same Thing Everywhere
A metric means the same thing everywhere only when its definition is complete, agreed by the people who use it, and held in one place that every report reads from. Miss any of the three and the argument about whose number is right returns.
A KPI is defined when someone who has never seen it could calculate it and get the same answer as everyone else. By that test a name and a formula are not a definition. A customer churn rate stated as lost customers divided by total customers leaves open which customers, lost by what rule, over what period, counted on what date and taken from which system. Each open question is a place where two teams can make different, reasonable choices.
This guide is about the mechanics: what a definition has to fix, how to resolve two competing definitions of the same measure, and where definitions should be held so that reporting tools inherit them. It works one measure through from start to finish. The case for why agreed definitions matter to trust is made in what makes business intelligence trustworthy and is not repeated here.
Start from what the measure is for
A definition written from the data upwards describes what can be counted. One written from the purpose downwards describes what needs to be known. The second is the useful order.
The UK government’s guidance on performance metrics for digital services sets the standard in its first line: “Well designed, meaningful metrics tell you whether your service is doing what it should”1. It then works from purpose to measure in steps: state why the service exists, derive a small number of intended benefits, form a hypothesis about what will produce each, and only then decide what to measure. At that last step it asks for exactly the thing a loose definition lacks: “you should have a clear understanding of what each measurement means”1.
The same guidance is firm about quantity. It tells teams to “Choose 3 or 4 metrics that would answer the question: ‘is this service working?'”1. A team can hold many working measures. The ones designated as KPIs are few, because each one has to be defined, owned and defended.
Purpose also settles arguments later. If the churn measure exists to judge whether retention spending works, then customers lost to a product being withdrawn should probably be excluded. Without a stated purpose there is no basis for that decision. The route from business outcome to measure is described in how to align a data strategy to business outcomes.
Working one definition through: customer churn
Take the churn measure from the opening and ask what would have to be written down for two people, working separately, to compute the same number. The answer is the set of things in the table. The list is DII’s, and its test is practical: leave any one of them unstated and two careful people can still disagree.
What a metric definition has to fix, worked for one measure
| Element | What it fixes | Example: customer churn rate |
|---|---|---|
| Purpose | The decision or question the measure serves | To judge whether retention activity is working |
| Formula | The calculation, including the denominator | Customers lost in the period, divided by customers active at its start |
| Grain | What one counted thing is, and the period | One contracting customer; calendar month |
| Filters | What is included and excluded, and why | Excludes trial accounts and customers on withdrawn products |
| Owner | Who decides what the measure means | Head of Customer Operations |
| Source | The system and dataset it is calculated from | The billing system’s contract table, not the CRM |
| Refresh | When it is recalculated and when it is final | Daily provisional; final on the third working day |
The example is illustrative and describes no organisation.
Three of these are where definitions most easily come apart in the churn example, and they repay extra care.
Grain. A count of customers depends on what a customer is. A household, an account, a contract and a billing entity are different numbers. Stating the grain forces the choice into the open.
Filters. Exclusions are where definitions quietly diverge. Each team removes the cases that look wrong to it: test records, internal accounts, a legacy product. Written down, the exclusions can be compared. Left in a query, they cannot.
Refresh and finality. A figure read on the second of the month and again on the tenth may differ because late transactions arrived. Both are right for their moment. The UK Government Data Quality Framework frames the test: “Data is timely if the time lag between collection and availability is appropriate for the intended use”2. A definition should say when a figure is provisional and when it is final, so that two readers know whether they are comparing like with like.
Resolving competing definitions
When two teams already calculate the same KPI differently, the task is to find out why before deciding which survives. On DII’s analysis, the difference has one of three sources, and each has its own resolution. The three-way split is analysis, not a finding from research.
They serve different purposes. Finance counts revenue when it is recognised. Sales counts it when the order is signed. Neither is wrong, and forcing one definition on both would make one team’s number useless to it. The resolution is two measures with two names. “Recognised revenue” and “booked revenue” can coexist. Two things both called “revenue” cannot.
They make different choices on the same purpose. Both teams want the same thing and have filtered differently. This is a real conflict and needs a decision. The owner of the measure decides, with the affected teams in the room and the purpose as the test. The decision and its reason are recorded with the definition.
One of them is a mistake. A join that duplicates rows, a filter that was meant to be temporary. This is found by reconciling the two numbers down to individual records until the difference is explained. It is laborious, and it is the only way to find out.
A reconciliation table is the working tool: start with one team’s number, apply each difference in turn, and arrive at the other’s. When every step is explained, the dispute has become a list of choices, and choices can be decided.
Where one version has to be withdrawn, the history matters. A KPI that changes definition without notice produces a step in every trend line. Restate the history on the new basis where that is possible, mark the break clearly where it is not, and tell the readers before they find it themselves.
Where definitions should live
A definition agreed in a meeting and written in a document will be re-implemented by hand in every report that uses it, and the copies will drift. The remedy is to hold the calculation in one place that tools read from.
That place is a modelling or semantic layer between the stored data and the reporting tools. Two vendors’ documentation states the idea in similar terms. dbt’s describes the aim of its semantic layer this way: “By centralizing metric definitions, data teams can ensure consistent self-service access to these metrics in downstream data tools and applications”3. It also states the practical benefit of a single home: “If a metric definition changes in dbt, it’s refreshed everywhere it’s invoked and creates consistency across all applications”3. Google’s documentation for LookML makes the same point from the analyst’s side: “you write SQL expressions once, in one place, and Looker uses the code repeatedly to generate ad hoc SQL queries”4.
Both are product descriptions, cited for the mechanism and not as a recommendation. The capabilities of these tools change between releases and should be checked against current documentation before a choice is made. The principle does not depend on either: a metric should be computed in one governed place and consumed everywhere else.
Three levels are available, in increasing strength.
- A register. The seven elements for each KPI, in a shared and controlled document. It makes definitions visible and comparable, and it does not stop anyone re-implementing them.
- A shared dataset. The measure is calculated once in a certified dataset that reports connect to. Reports built on it agree. Reports built elsewhere do not.
- A semantic layer. The definition is code, version-controlled, and served to several tools. A change is made once, reviewed, and takes effect everywhere.
A register is the right start for an organisation with no agreed definitions, because the agreement is the hard part and the technology is not. The KPI Register / Definition Worksheet holds the seven elements for each measure, with the decision record for any dispute. Business terms that the definitions rely on, such as what counts as a customer, belong in the Data Glossary; how a glossary, a dictionary and a catalogue differ is explained in business glossary, data dictionary or data catalogue.
Keeping definitions true
A definition is a commitment that has to be maintained. The business changes, systems are replaced and a rule that was right when it was written stops describing what the organisation does. Four habits keep the register honest, and none of them needs special tooling.
- Change through the owner. A request to alter a KPI goes to its owner, who weighs it against the purpose. Changes made directly in a report bypass the definition.
- Version it. Record what changed, when and why. A reader looking at last year’s figure should be able to find the definition that applied.
- Test it. Where the definition is code, a known input with a known answer catches an accidental change.
- Review the set. Measures accumulate. A KPI nobody has asked about in a year is a candidate for retirement.
How much of this to enforce centrally, and how much to leave to report authors, depends on how self-service is arranged, which is the subject of self-service BI: how much freedom, how much governance. A new dashboard is the natural moment to check that the measures it shows are defined, as dashboard requirements: what to settle before anyone builds describes.
Sources
- Government Digital Service (UK Government). How to set performance metrics for your service (Service Manual). 5 December 2017.
- Government Data Quality Hub (UK Government). The Government Data Quality Framework. 3 December 2020.
- dbt Labs. dbt Semantic Layer; Creating metrics (dbt Developer Hub).
- Google Cloud. Introduction to LookML (Looker documentation).
This article is general professional information. It is not legal or regulatory advice.