Security Metrics

Intro

In a prior article, I discussed the cycle of inaction. With respect to cybersecurity, the cycle results in managers putting too much faith in protection, which can lead to complacency.  To help improve the situation, management needs to get a better understanding of the effectiveness of the cyber defenses. Basically, managers need effective security metrics.

There have been attempts at coming up with security metrics, but they are usually not very useful. Consider, for example, this less than ideal security metric.

Percentage of systems with current anti-virus software.

Why is this a poor security metric?  Because it is actually an operations metric. Consider, if the number is 100% of systems have anti-virus, does that mean that all systems are then immune from malware? The answer is NO, because a zero day virus could get through. That 100% of systems are covered means that, operationally, we are doing a good job.

What if the answer is 0%, does that mean we are in big trouble? Again, the answer is NO, because the systems might be lab systems, in a closed network, immune from any new software. Or, the systems might be running an operating system like plan9, for which there are currently no known malware.

The problem with the above metric is that it does not directly tell us anything about the state of cybersecurity. So, how can we improve that?

The percentage of systems with antivirus should be tracked as an operational issue, or course. However, as for cybersecurity, the real measure will be how effective are the controls that currently exist.

The role of incident response in security metrics

Consider that a single desktop in an organization gets malware. The key metric here will be to determine how that happened, and it will tell you which part of your cyber defense failed. For example, consider the case where an employee receives an email message with an attachment. The attachment contains a virus, and the employee clicks on the attachment, because the attachment looks like a resume and the employee’s job is to look at resumes from strangers, as they work in the Human Resources department.

If the “resume” that the employee opened actually contained a virus, and that virus infected systems in the company, that would be a problem. Once the virus has been detected, an incident response would then tell us which security control failed. Perhaps it was the content filter in this case, along with the desktop antivirus. This is security metric data, how effective are our existing controls. Track this data.

While tracking this data and performing an incident response, it is very important to build an environment where users will report these events. Chastising users for opening an email attachment focuses too much on prevention and could hamper our ability to collect information to perform incident response. Therefore, I encourage security teams to encourage reporting and discourage the statements of “you should not have opened that attachment.”

Over time, with incident response and user reporting, you will have information on the effectiveness of security controls in use in the organization. You will see whether the current network architecture can be secured with existing tools, or whether it needs to be tuned.

Security assessments as a measure of cybersecurity

Many organizations run periodic security assessments, such as a pentest or vulnerability assessment. These are effective tools to help understand the current state of cybersecurity, if they are used correctly.

Consider the vulnerability scan, for example. Vulnerability scans from tools, whether OpenVAS, Nessus, Nexpose or others, may contain false positives or might contain too little data. The scanners offer great options, but if these options are not set correctly, the scanners might look at too little or too much. For the purpose of this discussion, let’s assume that your scanner is properly configured, and it not sitting behind a firewall that blocks all of its requests.

After a vulnerability scan has been completed, the output of a vulnerability scan should be cleansed of false positives. (A reminder, a false positive is a finding reported by the vulnerability scanner that is not a real finding.) Once the false positives have been removed, you have an accurate report on the number and severity of known vulnerabilities existing in a system.

Let’s assume we have scanned a webserver and we find that it is susceptible to the old heartbleed vulnerability. The web server administrator then either knew about this issue, or they were surprised to learn about it. That is a metric worth tracking. So, consider this metric.

How many security issues that were identified during a vulnerability scan were already known about by the application team?

This number should be zero, in an organization that has a good understanding of how to deploy secure applications. And this is much more realistic metric. Rarely are applications deployed with zero defects, zero vulnerabilities, because it is too hard to get there. Instead, applications are generally deployed with known vulnerabilities that we can monitor and control, and where the application owner can accept the risk for the issues.

Security awareness

Organizations generally have security controls in place, such as a firewall and virus scanners. Others have gone a little further and put a proxy in place to screen http traffic (but not https, sadly). As organizations put these tools in place, they are making it more difficult for attacker.

Attackers then respond by changing their tactics. Instead of going for a direct attack, they try to trick a user. This process is typically known as social engineering. And, one popular type of social engineering is a phishing email. A typical phishing email tries to trick a user to give out their username and password to an attacker.

In response, organizations perform a phishing email test. Controlled, safe phishing emails are sent to users, and their responses are tracked. What are good metrics to track with these phishing tests? I suggest that the metrics that we care most about are:

  • How many users have given away valid user credentials?
  • How many users have done this multiple times?
  • How has security awareness training reduced the number that give away credentials?

Conclusion

Cybersecurity metrics should be able tracking the organization’s current efforts  to protect their information, detect cybersecurity related issues, and  respond to cyber security threats. With the right metrics, the cybersecurity program can then focus on driving the metrics in the correct direction. For example, it is not about the number of systems infected by malware that is our primary concern anymore. It is, what existing control failed that allowed the malware in? It isn’t about how many users fall for phishing. It is about, does our awareness program actually show a meaningful change in the number susceptible to phishing attacks.

Take the time to build the right cybersecurity metrics. Once in place, the organization will be positioned to naturally response to changes in the cybersecurity landscape.