Security Metrics


In a prior article, I discussed the cycle of inaction. With respect to cybersecurity, the cycle results in managers putting too much faith in protection, which can lead to complacency.  To help improve the situation, management needs to get a better understanding of the effectiveness of the cyber defenses. Basically, managers need effective security metrics.

There have been attempts at coming up with security metrics, but they are usually not very useful. Consider, for example, this less than ideal security metric.

Percentage of systems with current anti-virus software.

Why is this a poor security metric?  Because it is actually an operations metric. Consider, if the number is 100% of systems have anti-virus, does that mean that all systems are then immune from malware? The answer is NO, because a zero day virus could get through. That 100% of systems are covered means that, operationally, we are doing a good job.

What if the answer is 0%, does that mean we are in big trouble? Again, the answer is NO, because the systems might be lab systems, in a closed network, immune from any new software. Or, the systems might be running an operating system like plan9, for which there are currently no known malware.

The problem with the above metric is that it does not directly tell us anything about the state of cybersecurity. So, how can we improve that?

The percentage of systems with antivirus should be tracked as an operational issue, or course. However, as for cybersecurity, the real measure will be how effective are the controls that currently exist.

The role of incident response in security metrics

Consider that a single desktop in an organization gets malware. The key metric here will be to determine how that happened, and it will tell you which part of your cyber defense failed. For example, consider the case where an employee receives an email message with an attachment. The attachment contains a virus, and the employee clicks on the attachment, because the attachment looks like a resume and the employee’s job is to look at resumes from strangers, as they work in the Human Resources department.

If the “resume” that the employee opened actually contained a virus, and that virus infected systems in the company, that would be a problem. Once the virus has been detected, an incident response would then tell us which security control failed. Perhaps it was the content filter in this case, along with the desktop antivirus. This is security metric data, how effective are our existing controls. Track this data.

While tracking this data and performing an incident response, it is very important to build an environment where users will report these events. Chastising users for opening an email attachment focuses too much on prevention and could hamper our ability to collect information to perform incident response. Therefore, I encourage security teams to encourage reporting and discourage the statements of “you should not have opened that attachment.”

Over time, with incident response and user reporting, you will have information on the effectiveness of security controls in use in the organization. You will see whether the current network architecture can be secured with existing tools, or whether it needs to be tuned.

Security assessments as a measure of cybersecurity

Many organizations run periodic security assessments, such as a pentest or vulnerability assessment. These are effective tools to help understand the current state of cybersecurity, if they are used correctly.

Consider the vulnerability scan, for example. Vulnerability scans from tools, whether OpenVAS, Nessus, Nexpose or others, may contain false positives or might contain too little data. The scanners offer great options, but if these options are not set correctly, the scanners might look at too little or too much. For the purpose of this discussion, let’s assume that your scanner is properly configured, and it not sitting behind a firewall that blocks all of its requests.

After a vulnerability scan has been completed, the output of a vulnerability scan should be cleansed of false positives. (A reminder, a false positive is a finding reported by the vulnerability scanner that is not a real finding.) Once the false positives have been removed, you have an accurate report on the number and severity of known vulnerabilities existing in a system.

Let’s assume we have scanned a webserver and we find that it is susceptible to the old heartbleed vulnerability. The web server administrator then either knew about this issue, or they were surprised to learn about it. That is a metric worth tracking. So, consider this metric.

How many security issues that were identified during a vulnerability scan were already known about by the application team?

This number should be zero, in an organization that has a good understanding of how to deploy secure applications. And this is much more realistic metric. Rarely are applications deployed with zero defects, zero vulnerabilities, because it is too hard to get there. Instead, applications are generally deployed with known vulnerabilities that we can monitor and control, and where the application owner can accept the risk for the issues.

Security awareness

Organizations generally have security controls in place, such as a firewall and virus scanners. Others have gone a little further and put a proxy in place to screen http traffic (but not https, sadly). As organizations put these tools in place, they are making it more difficult for attacker.

Attackers then respond by changing their tactics. Instead of going for a direct attack, they try to trick a user. This process is typically known as social engineering. And, one popular type of social engineering is a phishing email. A typical phishing email tries to trick a user to give out their username and password to an attacker.

In response, organizations perform a phishing email test. Controlled, safe phishing emails are sent to users, and their responses are tracked. What are good metrics to track with these phishing tests? I suggest that the metrics that we care most about are:

  • How many users have given away valid user credentials?
  • How many users have done this multiple times?
  • How has security awareness training reduced the number that give away credentials?


Cybersecurity metrics should be able tracking the organization’s current efforts  to protect their information, detect cybersecurity related issues, and  respond to cyber security threats. With the right metrics, the cybersecurity program can then focus on driving the metrics in the correct direction. For example, it is not about the number of systems infected by malware that is our primary concern anymore. It is, what existing control failed that allowed the malware in? It isn’t about how many users fall for phishing. It is about, does our awareness program actually show a meaningful change in the number susceptible to phishing attacks.

Take the time to build the right cybersecurity metrics. Once in place, the organization will be positioned to naturally response to changes in the cybersecurity landscape.

wannacry ransomware

A new ransomware outbreak has captured a lot of press over the past couple of days. So much so that the US Department of Homeland Security has put out a statement on this ransomware. While trying to prevent exposure to this type of malware would be great, it will be difficult to do. Antivirus, content filtering, firewalls, proxies and other cybersecurity devices are good, but the only work against threats that they know about. Now that the wannacry malware is known, cyber defenses can be tuned to block it. However, a new version of the malware, with just a small change, might not be recognized as a threat until it is too late.

This again points out that protection against this type of threat can not be done by cyber defense tools alone. Ransomware is an effective attack if you do not have any other way to recover your files. This might sound obvious, but this is the best time to ensure that your backups are up to date, and stored off line. If your cyber defenses fail against the next ransomware and you get hit, at least you can recover at your pace, and without paying.

Here is what I have shared with many that I have been working with on this malware. I hope it helps you.

Recently, there has been a new cyber attack that has been spreading across the Internet. The attack has been named “wannacry” or “wannacrypt”. It encrypts files and demands a ransom in order for users to recover the files. Typically, the ransom is approximately more than $300 per system.

This attack uses a flaw in the Windows operating system to spread from one system to another. Microsoft has recently release an update to fix this problem, and we are working hard to ensure that systems are protected against this attack. However, patching alone won’t be enough to protect us from this attack, we will need your help.

Please make sure that you do the following to maximize your cyber safety.

  • Ensure that your most important files are backed up. Further, make sure that backup is removed from your computer. For example, copy your important files to a USB disk drive and remove the disk once the copy has been completed.
  • Do not open email attachments unless you are absolutely sure you know who sent you the email and what the attachment is. If you have any doubts, call the person that sent you the email to confirm the message.
  • Do not download any freeware or “too good to be true” utilities from the Internet. These tools may be infected with malicious code.
  • Do not visit any suspicious websites, because the ads that play on suspicious websites may be infected with malicious code.

And, if you see anything suspicious, or notice that files have been encrypted on your system, please contact your local IT support team at once.

Stay safe…

Don’t Let the Press be your Intrusion Detection System

All of the highly-publicized breaches last year continue to highlight that organizations are still wrestling with how to get a handle on their cybersecurity[1].  Breaches put the confidentiality and the integrity of your information at risk, as we recently saw with the hack into the Democratic National Committee’s email[2]. A denial of service attack impacts your availability, as we have recently seen with the attacks against the DNS provider Dyn[3].  In cases like these or similar, organizations were not aware of the extent of the issue until they read it in the press.

So, why are organizations usually the last to know?

  1. Protecting confidentiality requires surgically reducing access to information. The information needs to be available and modifiable, just not to everyone. To do this takes an understanding of the workflow. Just opening the data up to all is a fast way to get a system deployed.
  2. Management lacks clear metrics on the state of cyber security in their organization. Few know any real information on how effective their current protection is. For example are all the virus scanners up to date in an organization? Can people bypass the proxies? Currently, management is given useless data like number of attacks blocked at the firewall, number of spam messages stopped, or number of viruses caught by virus scanner. (Why do I call these useless? I’ll be following that up in my next post – and tell you what you should be looking for. But suffice it to say – you have the data – you just aren’t looking at it correctly.)
  3. The perimeter defense just isn’t working. Many organizations have firewalls, web proxies and virus scanners that protect laptops at work. However, those same laptops are then used at home, where they are not behind the web proxy or firewall.
  4. There are very few really good cybersecurity professionals out there, which probably contributes to #2
  5. The bad guys are relentless.

As management is not seeing the right picture, most then are unaware that their cybersecurity defenses are inadequate. They don’t yet see a need to invest in monitoring the technologies they’ve invested in. And, this leads to no monitoring, which re-enforces the strategy of not investing in cybersecurity.

The Cycle of Inaction


Good management means that you invest efficiently, and investing in something that is not needed is inefficient. Lacking effective information, the perception becomes that there isn’t a problem. This feeds what I call the Cycle of Inaction. This cycle is caused by believing the investment in protection is enough, and lacking additional information, must be working. This leads to complacency, when metrics are actually needed. A complacency that sometimes is broken by a press article.

This cycle of inaction can lead to spectacular failures. Of note over the past couple of years, we have the hack of the NSA toolkit, the recent release of the CIA cyber toolkit, the hack of Yahoo!’s passwords, the hack of Target, the hack of …

We know of these events because the press is the Intrusion Detection System (IDS) of default for many organizations. That IDS, however, is not easy to control, and definitely reports what we call “trailing metrics,” or a metric about a problem AFTER it has happened.

What Is Your Cybersecurity Maturity

I’ve found that the cybersecurity issue that the industry is confronting is very similar to the quality issues that the industry tackled in the 1970s and 1980s. To address and improve quality, the ultimate solution was to install a mature process within an organization. A mature process is defined as a process that is repeatable, with quality-based decisions made using meaningful metrics.

I offer that many organizations are at a maturity level of 1, if the Capability Maturity Model (CMM) metrics are used. Getting to a CMM maturity level of 2 (of which there are 5) appears to be a little bit away for cybersecurity. If the struggle is to get to CMM 2, perhaps it makes sense to sub-divide the maturity level 1 into sub levels, as in the list below.

Level Action You are first to tell the story You can investigate privately You can prevent a large incident
1.1 Organization learns about cybersecurity failures via the press, where the message is uncontrolled and incident needs to be addressed.
1.2 Organization learns about cybersecurity failures via a third party, privately (e.g. law enforcement or a business partner) The message can be controlled, as can the response to the incident.
1.3 Organization learns about cybersecurity failures internally. This allows the organization to control the message of the incident as well as the response.
1.4 Organization notes indicators that an incident is about to happen. Here, the organization can take steps to mitigate an incident before it happens.

To increase your cybersecurity maturity, you need to improve your ability to monitor the cybersecurity of your digital assets, by analyzing the outputs of the technologies you have invested in.

Consider, your organization currently has firewalls to protect against bad things coming from the outside. You have web proxies and even content filters to protect against bad things coming from the outside. And, you have anti-virus scanners on your desktop.

With all of those layers of defense, it seems reasonable to conclude that no virus should ever reach the desktop. Measure that. Any time that any computer’s virus scanner detects a virus, a root cause investigation should be performed to determine which security control failed. For example, if a desktop has recently been infected with ransomware, a forensic analysis should be performed to determine how the virus got on the system. At the highest level, the cause will be one of these two things:

  1. The user violated a security practice, such as plugging in a USB.
  2. An existing cybersecurity technology failed. Did it not work? Was it improperly deployed?

Collect these metrics on the root causes, and soon you will have a clearer picture of the effectiveness of the controls.

Next topic, suggestions for effect metrics, to help you increase your “sense” of cybersecurity within your organization.


[1] Let’s define cybersecurity as the protection of the confidentiality and integrity of information, along with ensuring that the information is available when needed to whomever needs it

[2] Krebs, B. (2017,January). The Download on the DNC Hack. Retrieved from

[3] Newman, L. H. (2016, December). The Botnet That Broke the Internet Isn’t Going Away, retrieved from