- Setting up Auditing & Logging of Files/Objects Using Native Windows File Server Tools - 16th October 2020
- Designing Key Performance Indicators (KPI) - 15th July 2020
- DDOS Attacks and Website Hacking - 6th July 2020
Key Performance Indicators (KPI) are metrics that are used to measure the success of a strategy, programme or product. As strategies are implemented by programmes/initiatives, Programme Managers in particular, as well as Product Owners and Departmental Heads in general, are the ones that usually create the KPI for the initiative, build a consensus around them and subsequently use them to measure success. In this article, I explain how I design KPIs.
Tie the KPI to the Business Goal of the Programme
Clarity on the business goals of the initiative and how it will meet that goal is critical in selecting and designing a good KPI. The initiative has to benefit the company somehow, and the KPI needs to establish this relationship. A good method of determining this is to ask why the company should invest in this initiative, and state the goals the programme needs to achieve to realise this benefit. For e.g. if the programme implements a strategy that directly reduces operating cost, such as data centre footprint, through virtualisation by reducing the number of physical servers by moving them onto virtual machines (VM), then measuring data centre power draw would be a good KPI for this strategy.
Make the KPI Measurable
AKPI that is easy to measure, and without too much cost or resource requirement to measure it is the most effective. A measurable KPI is one that enables the programme to analyse the resulting data and take the right actions.
Use Ratios and Ranges to Make Achieving the KPI Realistic
Quantifying the KPI with an achievable range or a ratio is key to this design aim. Sometimes when I have been parachuted in to heal a programme that has been classified as failing, my efforts have resulted in having to fix a badly designed KPI that is failing to either provide accurate results, results not tied in to business goals or goals that are simply not achievable.
To continue the reducing of data centre footprint example above, instead of stating that the programme should save X amount of costs, we can say that the programme should reduce data centre power consumption between 5% to 10% by the end of 12 months. This focuses the programme on its goal, which is the means to achieve the business goal. The benefit of this is that if the programme is achieving it’s goal, but the business goal is not being achieved, then the discussion will turn to how to improve the strategy that is meant to achieve the business goal, instead of how to improve a failing programme.
This helps the programme manager conduct another core part of their responsibilities – expectation and stakeholder management. By selecting the correct KPI the entire language that is used to frame the programme and discuss it's progress changes.
An Aside – Difference Between a Strategy and the Programme that Implements It
It is important to highlight the distinction between a strategy and the programme created to implement it. The strategy’s goal is to save X amount of money for the business and the way this will be achieved will be to reduce data centre footprint. A programme has been created to reach that goal by reducing power consumption.
If the X amount of cost savings is not achieved, then it doesn’t necessarily imply the failure of the programme. The programme, after all, could have successfully reduced power consumption within the targeted range. However, the strategy could still have failed for other reasons. Such as another programme also implementing another part of the strategy not achieving it’s target. Or the programme that created the strategy in the first place not correctly designing the cost savings approach, for e.g. the power consumption saves not being enough to cause the required reduction in data centre footprint.
Avoid Hollow Metrics
A “hollow” metric is one that does not add value, particularly in achieving the programme or strategy aim, but does make the programme or strategy look good. For e.g. counting the number of servers virtualised might give the impression of the programme being successful, and might indeed be a great metric if you were wanting to reduce data centre floor space occupancy, but it does not give us information about server usage and power draws as those servers may not have been heavily utilised or, relatively speaking, had much power draw in the first place.
Avoid Too Many Measures
With the advent of big data, data warehouses, presentation layers and data visualisation software (and data science), it is easy to get carried away with the type of KPI being used and their quantity. It’s best to not measure everything that can be measured and to not blindly trust an analytics tool to collect the right data. Instead, I prefer to keep things as simple as possible and use the business goals to choose a small amount of metrics that help understand performance. This helps in
- Your own understanding of the success of the programme
- Other’s understanding of the status of the programme, particularly presenting to business sponsors and stakeholders.
- Reducing the wastage of time and effort in analysing data that creates little or no insight.
Use both Quantitative and Qualitative KPIs
A quantitative KPI is that which measure a quantity, such as the amount of power draw. A qualitative indicator studies the quality, such as user feedback, and will help understand why something has happened. Combining the two will give a balanced outlook and truer picture of success/failure as well as the causes.
For e.g. there is little point in moving physical servers to VMs if the processing power degrades causing end user dissatisfaction. For one thing, developers may be so dissatisfied with the results that they may opt to order new physical servers to replace their underperforming VMs or shift the processing load onto existing physical servers as they perform better than the VMs. This, obviously, would reduce the impact our efforts are having in achieving our goal of reducing power consumption.
This is where the understanding of the why behind a result is aided by a qualitative KPI. Measuring user satisfaction in this way would give us a better picture, or at least a starting point for further investigation, as to why power consumption is not decreasing in line with the increasing number of physical servers being virtualised through our programme.
Additionally, a qualitative KPI such as customer satisfaction surveys may also provide further insight into strategy performance. For instance, we may have designed and tested the solution to use VMs that we think provides adequate performance and achieved our goal of power consumption saves. However, customers may feel otherwise and although we may be saving on operating costs, we may also be losing revenue through customers moving onto other products, not because of better features, but because of lagging performance caused by our programme.
Use Lagging and Leading KPIs
Lagging indicators tell you about the outcome of past actions, such as revenue, profit, and cost. Leading indicators tell you how likely it is that your product will meet a goal in the future.
For e.g. if it’s getting more and more difficult to get the performance of candidate applications in scope for moving onto VMs, during testing, to match their performance benchmarks when they are running on physical servers, then we know that migrating apps in future onto VMs will be more difficult, take more time and will generally make meeting our goal of reducing power consumption tougher. We would have already moved most of the likely applications that can run well on VMs onto VMs and moving additional applications will result in us using more powerful servers, that would have more power consumption, to host our VMs.
Use Trends and Patterns
I prefer comparing KPI data over time periods to help spot patterns and trends that can inform how the programme is doing, the success of programme decisions and potential future performance.
For e.g. I would see if the trend in power consumption is changing, by measuring rate of change over time, to see if the decrease is tapering off and if, as a result, we would need to look at different technologies to continue making progress.
Regularly Review, Present and Syndicate the KPI
I, as part of the regular reporting cycles, collect KPI data, analyse it, disseminate it to stakeholders and include it in part of regular status presentations to sponsors. I tend to showcase the KPI data not just to the sponsors, but also to the programme team as well as I feel it’s important to let them know how the programme is doing overall and the effect their work is having. Having a programme status report, that includes a section for KPI data, helps provide a good vehicle for this.