The instrument panel of a car can be seen as something similar to the subject of this article. They give the driver of the car indications of the performance, effectiveness, and similar things that contribute to for example safety.

In this article, I will explain the terms KPI, KRI, and KCI and also provide a holistic and visual model of their interrelationships. And I think the analogy, to the instrument panel in a car, has many similarities.

Before we jump into the visual models and dive deeper into the subject, let us take a look at a simple description of each acronym. What do they mean?

KPI – Key Performance Indicators

It is a set of measures used to monitor and track performance. KPIs are used by organizations to measure and oversee the progress and direction toward the set KPI. The KPI should be aligned toward desired and prioritized objectives that are key to the organization’s success. For example strategic goals, operational excellence, and financial achievements.

KRI – Key Risk Indicators

It is a set of measures used to monitor and track the organization’s risk level. How the risk levels change and to track if they are within the set tolerance and appetite. KRIs are supportive of KPIs.

KCI – Key Control Indicators

It is a set of measures used to monitor and track the organization’s control effectiveness in meeting its intended objectives. To achieve a relevant measurement of KCIs they need to be related to both the control and the risk for which the control is implemented. The KPI and the KRI.

COMMON ATTRIBUTES

KPI, KRI, and KCI have as common attributes which are:

• Measurable – through quantitative or qualitative measures.
• Indicators – they indicate that something is to happen or has happened.
• High-level measurements – more about holistic measures and less about details.
• Leading – means they are about indicating a future event.
• Lagging – means they are about reporting historical events.
• Objectivity – means they provide objective perspectives.

I will, at places in this article use K[x]Is as a summarized acronym for KPIs, KRIs, and KCIs for convenience.

WHAT ABOUT SECURITY?

So what do all these measurement things and indicators have to do with security you may ask? It more or less comes down to:

Security within an organization is a supporting function.

One way, not only applicable to security though, to showcase how security (or other supporting functions) is contributing to the success of an organization can, IF done adequately, be accomplished through measurements, for example with the help of KPIs.

KPIs, KRIs, and KCIs are well-known methods used to understand how an entity (individual, group, team, function, organization, company etcetera) performs. Understanding the application of how KPIs, KRIs, and KCIs work and how they can be applied in reality can be helpful methods for security practitioners/leaders/professionals to better understand how security contributes to the success of an organization.

Rule number one for KPIs, KRIs, and KCIs
Develop and align your security indicators towards your organization’s mission, vision, and objectives.

If you at any time during your KPIs, KRIs, and KCIs development feel lost, default back to this rule, and you will be in a good place once again.

CHALLENGE

Most things, and I would say almost anything, can be measured. And in today’s society and digital ecosystem, data sets and metrics are more or less found everywhere. And this, as I see it personally, is one of the most common challenges.

As anything can be measured and due to that we as humans and organizations love metrics we tend to over-complicate or measure things without a clear purpose.

K[x]Is should be, according to my beliefs, designed, developed, selected carefully, and mindfully. Why is this the case? Why not just start to measure and see where it goes? This is absolutely one way to do it but I would say it is a sub-optimal approach.

Instead of starting out by “guessing” and based upon the color of the sky when choosing K[x]Is, start by asking:

“Why are we measuring?”

If there is no clear understanding and purpose for why things are measured, why are you doing it? What is the purpose of it?

Measuring things takes time and resources. The data shall be produced (most often automatically though) and then processed (for example manually reviewed and analyzed). And when the K[x]Is are produced and processed they shall also lead to something value-adding. In most cases some form of improvement or action that is needed to be taken.

K[x]Is needs resources to be allocated. If things are just measured, for the fun without a clear purpose, I think the time and energy are better spent on other things. From a security perspective, where in most organizations there already exists a resource gap or limitation, I do not think it is the smartest thing to add up with more irrelevant tasks that do not add value to the organization.

I have been around in many places who have been measuring things without having a clear understanding of why they do things. This.

“[…Conversation…]”
“We need to report these metrics and K[x]Is to the management?”
“Ok, why do they need them?”
“I don’t know!”
“What do they do with them?”
“I don’t know!”
“Are they used to <insert>?”
“I don’t know!”

I have had this dialogue more than once. And I am fine with that organizations and individuals can not provide me with an answer. Here and there individuals and organizations inherit a reporting and measurement structure from someone that was totally applicable at one point in time when it was constructed.

I am by no means saying there is necessarily something wrong if you can not justify the questions for why certain things are measured in a certain way. But usually, this is a strong observation and indicator that things are not calibrated as they should be and that some tweaks could improve the situation.

In addition to my empiric observations and experiences, I also think that K[x]Is should be kept to a low number of items. I have seen organizations that have around 10-20 security KPIs for example. And then my question is, are these actually KPIs? Remember, a KPI is a measurement on a high level that is related to the organization’s Key activities to measure Performance. The measurement of these key activities provides future or historical Indications of events.

Are all those 10-20 things equally important one may ask? The answer is probably: most likely NOT. One way to find it out is to try to link all these things with your organization’s mission, vision, and objectives = organizational KPIs. I would not be surprised if the majority of the security indicators will not be linkable and there is nothing wrong if this is the case.

But if this is the case, you can ask yourself, “Why are we measuring these things?”. If the measurement is used to drive operational improvements, that is a good thing. But maybe the things you are measuring are not key indicators? Or maybe they are and they should be communicated to your executive leaders and business stakeholders? If that is the case, make sure to communicate and speak in a language that makes sense to your audience and stakeholders.

KPI, KRI, & KCI – MODELED

KPIs, KRIs, and KCIs are high-level measurements. They are less about details. They are more about holistic representations than a collection of detailed things, also known as metrics.

KPIs, KRIs, and KCIs can be modeled according to the following illustration in Figure 1.

Here and there, in real-life situations, I have seen and used PI (performance indicator). This is a layer between the KPIs and the Metrics. I think this makes perfect sense IF there is a strong need to be able to measure according to this structure.

This structure of three levels could, for example, make (some) sense if for example metrics are generated from individuals, PIs are the aggregated view of the performance of a group of individuals such as for example a team, business unit, division etcetera. The KPIs are the company-wide metrics, that are an aggregation of the PI, which includes metrics that together are aggregated up to the KPI level.

But I am, and as you might have figured out from reading other articles on my website, a pragmatic dude so I strive to make things simple and easy to understand. In this article, we will therefore stick to the two-layered model.

And this is also what I preach to my customers. Keep things simple. Do not start to overcomplicate things from the beginning. This is not only relevant to KPIs, KRIs, and KCIs. This is my general belief when it comes to security. Be pragmatic and do those things that make sense and are relevant to your organization. Advance and do more complicated stuff as you and your organization are maturing.

THE INTERRELATIONSHIP

KPIs, KRIs, and KCIs have an interrelationship with each other, and this might not come as a surprise to you. Every acronym has the prefix “Key“.

The interrelationship between KPIs, KRIs, and KCIs is modeled and explained in Figure 2 below.

To exemplify the model let us assume that your organization has several KPIs. In this example, we look at KPI #1 and the interrelationship of this KPI to its KRIs and KCIs. I will use the model in Figure 2 but illustrate the model through an example.

KPI #1, in Figure 3 below, is based on a set of metrics, which are pulled (manually or automatically), from the business landscape and information systems. These metrics support the measurement of the progress and trend for the organization in relation to the KPI.

KPI #1 in this example is “Increase <goal> through <plan> to achieve <objective>”. For example, higher market penetration in a certain geographical region and through these initiatives gain more and a broader revenue stream that increases the organization’s bottom line with X%.

Two KRIs, KRI #1 and KRI #2, are developed to monitor the risks related to the KPI. As the KPI, as exemplified above, directly means that physical and digital presence will be increased in certain geographical regions these initiatives come with risk. For example, regulatory compliance and laws related to data transfer and management.

For example, KRI #1 is related to a regulatory law applicable to the geographical region that is connected to KPI #1. Changes to the regulatory law, i.e. how digital and information assets are managed (stored and transferred), will implicate the risk level.

KRI #2 is related to the corruption ratio in the region, i.e. the geographical region is known for having a high number of insider threats. There have been several data breaches taking place that have been traced back to insider threats.

As KPI #1 potentially will be highly negatively impacted by KRI #1 and KRI #2 changes to the risk levels need to be monitored. To ensure that the risks are mitigated, controls have been implemented to reduce the potential likelihood and impact for each risk related to each KRI, i.e. KRI #1 and KRI #2. For example, in relation to KRI #1 changes regulatory laws and compliance requirements are monitored with the help of external partners and experts.

KRI #1 is an aggregation of two risks, Risk #1 and Risk #2 which are both mitigated with the help of the same control, i.e. Control #1. The control, i.e. control #1, is the equivalent to the processes and methods implemented to monitor changes related to regulatory laws and compliance. Control #2 implemented for Risk #2, which is related to KRI #2, is based on mainly technological security controls to increase the protection of data loss or leakage conducted by malicious insiders, i.e. insider theft.

The control effectiveness is monitored through KCIs. In this case, and how KCIs work, two controls are connected to the same KCI.

KCIs can be and are usually connected to several controls in the same way as risk and KRIs. One KRI will, most often, be connected to several risks. Several KRIs will often be connected to one KPI, as there might be several risks that need to be monitored to understand changes in the risk level in relation to the KPI.

EPILOGUE

I personally love measuring things. But measuring stuff just for measuring I think is only a waste of resources and energy. And when it comes to security, these forms of things that do not contribute to an increased security posture or improved cyber resilience for an organization I think is worth contemplating if they should be conducted. And everything related to security does not need to be measured. Most of the things can be measured but that does not mean they need to be measured.

Metrics and K[x]Is are still also just indicators. Use them as such. And I strongly recommend using them to create and enable intelligent discussions. Use the indications to better understand what has happened or is about to happen. They can be supportive, likewise as a compass if you are navigating in the forest. But if the compass is not used in the right way or if it is broken the effects and positive outcomes will not be gained.

And keep in mind, K[x]Is they will not solve all security issues or make bad things go away. But they can be helpful for your organization to increase monitoring, tracking, and positive progress toward creating a stronger security posture and cyber resilience. But, I repeat, everything related to security does not need to be measured just because it can be measured. There is no need to create K[x]Is for more things than necessary or needed within your organization.

I think that many organizations would make better use of their time and resources by starting to prioritize security hygiene and by making sure they are doing these things correctly. Still today (and the date when this article was first created) and for years to come, I am very confident in saying that security hygiene plays an incredibly strong role in an organization’s security posture and cyber resilience.

But at the same time, measuring an organization’s security maturity (with the help of for example a couple of security KPIs linked to KRIs and KCIs) might be what is needed to get the ball rolling. You might need to showcase to your executive leaders and business stakeholders that things are not done adequately and that there is a need to invest more resources and funding in those security hygiene things.

In this case, use those relevant K[x]Is to accomplish what is needed for your organization to become more secure and resilient. Once again, ask yourself, your team, and your organization:

“Why are we measuring?”

If you together figure out “Why”, I can promise you that the “What” and “How” will become fairly simple to accomplish related to things that are of interest to be measured from a security point of view in your organization.

UNPOPULAR OPINION

For some security experts and pros, things in this section go into the list of unpopular opinions but I think it needs to be put out there.

I think that you now have an idea of the interrelationship between K[x]Is and how they can be and how they are linked together. They make up an ecosystem on their own. But this is not always how organizations do it in reality, i.e. use KPIs, KRIs, and KCIs. And those who use them, less often really think them through and ensure they are valid over time.

Many textbooks and theories dictate that security K[x]Is should be developed and established when for example:

• a security strategy is put in place and/or
• when that ISMS is implemented and/or
• when a security program is created and/or
• when security improvement projects are run and/or
• when a risk is treated and to be monitored and/or
• <the list goes on>

Nothing wrong here according to those writing but this is not always how organizations approach security in reality. And it doesn’t matter what I think or what the textbooks say what should be in place. In the end, it is the organization itself that needs to make the decision and come to the conclusion of what is needed. Realizing what is needed and when in time the organization is mature enough to start measuring is key. Let us call this timing.

And to get a better feeling for the timing of security and when these things shall be created becomes easier if you listen and communicate with your stakeholders. Do so in a language that makes sense for them. Security is less about you. You are there to support your organization and your stakeholders. That textbook or theory will not make the measurement happening. It is you, as a security leader/professional/practitioner, who needs to show the way. It is you who need to guide your organization to reach this form of maturity together with your stakeholders.

The most common thing used in organizations is KPIs and I think it also makes perfect sense. But even these are in many places not that useful. Developing relevant and good security KPIs can in many cases be enough for an organization to accomplish what is needed. Unfortunately, these are rare to find and see being kept relevant over time.

When you have managed to put those KPIs in place, you can take a look at those KRIs and KCIs. This is a very natural second progression when good KPIs are put in place. And the better those KPIs are, guess what? The better those KRIs and KCIs will be. But, as I have said before, measuring stuff requires resources. Just measuring stuff and putting it on a dashboard is not how you work with K[x]Is. Not at least how I see it. Do those things that are relevant to your organization.

And if I were to give you only one takeaway from this article, it would be:

Focus on developing, agreeing, and establishing well-thought-through security KPIs. These will not substitute for KRIs or KCIs. But if these are developed and made sure they align with your organization’s mission, vision, and objectives you have accomplished something many organizations haven’t when it comes to security metrics and measurement.

It is not those cool and fancy-looking dashboards that will make your organization secure. It is how the things, presented and communicated on them, are practically applied in reality that will improve the security posture and cyber resilience in your organization. These things are most often carried out through different forms of security initiatives, activities, operational tasks, tactical & strategic investments, and so forth.

Use the information, from those indicators and measurements, to accomplish those things that make a difference in reality. Do not get trapped in the measurement rabbit hole. Use the measurement to accomplish an improvement in reality, that is where the bad things are happening and that will contribute to a negative business impact if they are actualized. And from a security point of view, this is how those K[x]Is should be developed according to how I see it.

PRO TIP! Well-thought-through security KPIs are not created in isolation by the security team sitting in a dark basement and deciding on what to measure on their own. They are created by spending time together with your executive leaders and business stakeholders. You need their help, if you do not already understand, how those security KPIs shall be aligned with your organization’s direction —> the mission, vision, and objectives that are set out to make your organization successful.

RECOMMENDATIONS

Before starting out with KPI to its KRIs and KCIs make sure to answer the questions:

• Why do we measure things, i.e. use KPIs , KRIs, and KCIs?
• What shall we measure?
• How can we measure these things?

When developing KPIs, KRIs, and KCIs I recommend to:

• Develop and align your security KPIs, KRIs, and KCIs to your organization’s mission, vision, and objectives.
• Keep things simple, do not overcomplicate the measurements.
• Keep the number of KPIs, KRIs, and KCIs fairly low.
• Keep them relevant and repeatable.
• Revisit them periodically to ensure they are valid.
• Remove them if they only create noise, i.e. are irrelevant.
• Assign an owner to each KPI, KRI, KCI.
• Monitor the KPIs, KRIs, KCIs.
• Tweak them and improve them over time.

And last but not least:

• Be pragmatic.

Henrik Parkkinen