In my prior post I wrote about how security metrics can be leveraged to protect large, complex enterprise networks. Here I want to explore what makes a good metric, and contrast that to many not-so-great metrics that are already in use today.
Security metrics is a topic that has been under study for quite some time. The folks over at securitymetrics.org, a 7-year-old community website for security practitioners, point out that no consensus exists on what security metrics should be used for measuring security effectiveness. They have, however, published the following list of security metrics known to be in wide use (from a Robert Frances Group survey):
One thing that jumps out about these metrics is they all refer to past events: viruses detected, intrusion attempts, invalid logins, and so forth. What if you want proactive security metrics? What if your goal was metrics that revealed the effectiveness of your current controls, with actionable remediation steps identified where controls are not effective? What kind of questions would you want answered?
Chances are, you’d like to know how easily attackers can get in—and where their potential access points are. You’d probably also like to know how big your attack surface is, and if there are any undocumented potions of your network. For any given question, you’d like the ability to drill down to access underlying details. And for all of those questions, you’d want to know whether things are getting better or worse:
How do my metrics compare to last month? Last year? And where can I direct my investments in order to improve my security posture?
Finally, you’d like a dashboard that provides “at a glance” answers to these questions and how things are trending, as shown in the RedSeal 5 executive dashboard below:
So let’s return to our original question: How do we define a good security metric? Andrew Jaquith, in his seminal book “Security Metrics”, provides the following 5-point definition of a good metric:
- Consistently measured, without subjective criteria: Metrics confer credibility when they can be measured in a consistent way… “Metrics” that depend on the subjective judgments of humans are not metrics at all.
- Cheap to gather, preferably in an automated way: Metrics ought to be computed at a frequency commensurate with the process’s rate of change… “often” is better than “sometimes.”
- Expressed as a cardinal number or percentage, not with qualitative labels like “high,” “medium,” and “low”: For example, “vulnerabilities directly exposed” evaluates to a cardinal number that can be counted.
- Expressed using at least one unit of measure, such a “defects,” “hours,” or “dollars”: Good metrics should contain at least one unit of measure that characterizes things being counted; for example, “vulnerabilities directly exposed” expresses one unit of measure, namely vulnerabilities that are directly exposed to attack from the Internet.
- Contextually specific—relevant enough to decision-makers so that they can take action: Good metrics mean something to the persons looking at them. They shed light on an underperforming part of the infrastructure under their control, chronicle continuous improvement, or demonstrate the value their people and processes bring to the organization.
Finally, Jaquith summarizes good metrics by pointing out that getting the right metrics depends on knowing the right things to measure. In security, business leaders typically ask the following:
- How effective are my security processes?
- Am I better off than I was this time last year?
- How do I compare with my peers?
- Am I spending the right amount of money?
- Where are my risk transfer options?
If you can answer these questions, then you have good metrics; if not, well, you ned to rethink your approach… and of course one of the recommendations I’d make, in addition to adhering to the above criteria, to contact RedSeal.
We feel we’ve got the best set of metrics yet produced by a single product to meet these parameters, specifically within the context of networks security and risk. Take a closer look and we think you’ll agree.
