Rethinking How You Work With Detection and Response Metrics – Source: www.darkreading.com

Source: Dzmitry Skazau via Alamy Stock Photo

Sorting the false positives from the true positives: Ask any security operations center (SOC) professional, and they’ll tell you it’s one of the most challenging aspects of developing a detection and response program.

As the volume of threats continues to rise, having an effective approach to measuring and analyzing this kind of performance data has become more critical to an organization’s detection and response program. On Friday at the Black Hat Asia conference in Singapore, Allyn Stott, senior staff engineer with Airbnb, encouraged security professional  to reconsider how they use such metrics in their detection and response programs.

Stott broached the topic at last year’s Black Hat Europe, where he explained how to create a detection and response framework. “At the end of that talk, a lot of the feedback I received was, ‘This is great, but we really want to know how we can get better at metrics,'” Stott says. “That’s an area where I’ve seen a lot of struggles.”

The Importance of Metrics

Stott says that metrics are critical in assessing the effectiveness of a detection and response program because they drive improvement. Providing quality metrics is an essential step in the detection and response process, he adds, because it will reduce the impact of threats and validate investments in detection and response programs.

Stott says metrics also enable security managers to demonstrate how detection and response lowers risk to the business. “Metrics help us communicate what we do and why people should care. That’s especially important in detection and response because it’s very difficult to understand from a business perspective.”

The most critical area for delivering effective metrics is alert volume. “Every security operations center I’ve ever worked in or ever walked foot in, it’s their primary metric,” Stott says.

He emphasizes that knowing how many alerts are coming in is important, but that in itself is still not enough. “The question is always, ‘How many alerts are we seeing?'” Stott says. “And that doesn’t tell you anything. I mean, it tells you how many alerts the organization receives. But it doesn’t actually tell you if your detection and response program is catching more things.”

He warns that effectively utilizing metrics can be complex and labor-intensive, adding to the challenge of effectively measuring threat data. Stott acknowledges he has made his share of mistakes when it comes to engineering metrics to assess the effectiveness of security operations.

As an engineer, Stott routinely evaluates the effectiveness of the searches he conducts and the tools he uses, seeking to get an accurate true- and false-positive rates for detected threats. The challenge for him and most security professionals is connecting that information to the business.

Implementing Frameworks Properly Is Critical 

One of his biggest mistakes was his approach to focusing too much on the MITRE ATT&CK framework. While Stott believes it provides critical details on threat actors’ different threat techniques and activities and says organizations should use it, that doesn’t mean they should apply it to everything.

“Every technique can have 10, 15, 20, or 100 different variations,” he says. “And so having 100% coverage is kind of a crazy endeavor.”

Besides using MITRE ATT&CK, Stott recommends using the SANS Institute’s Hunting Maturity Model (HMM), which helps describe an organization’s existing threat-hunting capability and a blueprint for improving it.

“It gives you the ability to, as a metric, say where you’re at as far as your maturity today and how the investments you’re planning to make or the projects you’re planning to do will increase your maturity,” Stott says.

He also recommends using the Security Institute’s SABRE framework, which provides risk management and security performance metrics validated with third-party certification. “Rather than test across all of the MITRE ATT&CK framework, you’re actually working on a prioritized list of techniques, which includes using MITRE ATT&CK as a tool,” he says. “That way, you’re not just looking at your threat intel but also at security incidents and threats that would be critical risks for the organization.”

Using these guidelines to provide useful metrics requires buy-in from CISOs, since it means gaining organizational adherence to these different maturity models. Nevertheless, it tends to be driven by a bottom-up approach, where threat intelligence engineers are the early drivers.