table of content

Part 1 Building the foundation

1 Introducing cybersecurity metrics

1.1 Understanding metrics

1.2 Significance of metrics in cybersecurity

1.3 Traditional vs. innovative metrics

1.4 The changing landscape of cybersecurity

1.5 The importance of frameworks in cybersecurity

1.5.1 HITRUST framework

1.5.2 Center for Internet Security CIS Top 18 framework

1.5.3 NIST Cybersecurity Framework v2.0

1.6 AI and predictive cybersecurity metrics

1.7 Defending against AI

2 Cybersecurity analytics toolkit

2.1 Tool selection

2.1.1 Key tool selection factors

2.2 Dashboard development

2.2.1 Choosing the right metrics

2.2.2 Knowledge points

2.2.3 Dashboard example

2.3 Statistical analysis

2.3.1 Trend analysis

2.3.2 Correlation analysis

2.3.3 Probability distribution

2.4 Integrated security analytics environment

2.4.1 Integrated security analytics environment exercise

2.5 Continuous improvement and iteration

2.5.1 Iteration techniques

2.5.2 Continuous improvement cycle

3 Implementing a security metrics program

3.1 Introducing metrics program design

3.1.1 Key components of a security metrics program

3.2 METRICS methodology

3.2.1 Measure

3.2.2 Evaluate

3.2.3 Threshold

3.2.4 Report

3.2.5 Improve

3.2.6 Communicate

3.2.7 Sustain

3.3 Using the METRICS methodology example

3.3.1 Step 1: Measure

3.3.2 Step 2: Evaluate

3.3.3 Step 3: Threshold setup

3.3.4 Step 4: Report

3.3.5 Step 5: Improve

3.3.6 Step 6: Communicate

3.3.7 Step 7: Sustain

3.3.8 Exercise summary

3.4 Building a metrics dashboard

3.4.1 Different dashboards for different stakeholders

3.4.2 Security dashboard example

3.5 Tools and technologies for metrics collection

3.5.1 Open source tools for metrics collection

3.5.2 Commercial tools for metrics collection

3.5.3 Automation and integration

3.5.4 Scripting for metrics collection

3.6 Common pitfalls in metrics programs

3.6.1 Pitfall 1: Overreliance on quantitative metrics

3.6.2 Pitfall 2: Misalignment with business goals

3.6.3 Pitfall 3: Stagnation and complacency

4 Integrating metrics into business strategy

4.1 Business alignment

4.1.1 Business-aligned security metrics example

4.1.2 Mapping metrics to business performance

4.1.3 Supporting innovation with metrics

4.2 Security metrics alignment with business strategy exercise

4.2.1 Step 1: Understanding MediHealth’s business objectives

4.2.2 Step 2: Selecting key metrics

4.2.3 Step 3: Mapping metrics to business goals

4.2.4 Step 4: Measuring metrics

4.2.5 Step 5: Communicating the Metrics

4.3 Security metric reporting

4.3.1 Presenting metrics in executive reports

4.3.2 Demonstrating return on investment

4.3.3 Communication strategies

4.3.4 Metrics communication exercise

Part 2 The metrics that matter

5 Establishing the foundation

5.1 Governance

5.2 Organizational context

5.2.1 Understanding the differences between mission, vision, and values

5.2.2 Strategic objectives

5.2.3 Organizational metrics

5.2.4 Organizational metrics exercise

5.3 Risk management strategy

5.3.1 Risk mitigation

5.3.2 Risk management metrics

5.3.3 Risk management metrics exercise

5.4 Roles, responsibilities, and authorities

5.4.1 Roles, responsibilities, and authorities metrics

5.4.2 Roles and responsibilities metrics exercise

5.5 Policy, processes, and procedures

5.5.1 Policy, processes, and procedures metrics

5.5.2 Policy, processes, and procedures metrics exercise

5.6 Oversight

5.6.1 Governance structure

5.6.2 Simulated governance structure

5.6.3 Governance exercise

5.6.4 Oversight metrics

5.6.5 Oversight metrics exercise

5.7 Cybersecurity supply chain risk management

5.7.1 Supply chain risk management practices

5.7.2 Cybersecurity supply chain risk management metrics

5.7.3 Supply chain risk management metrics exercise

5.8 Governance metrics

5.9 Answer to exercise 5.6.3

6 Foundations of cyber risk

6.1 Identify

6.2 Asset management

6.2.1 Asset management metrics

6.2.2 Asset management metrics exercise

6.3 Risk assessments

6.3.1 NIST 800-30: Guide for conducting risk assessments

6.3.2 Risk assessment metrics

6.3.3 Risk assessment metrics exercise

6.4 Continuous improvement

6.4.1 Continuous improvement metrics

6.5 Identity metrics exercise

7 Protecting your assets

7.1 Identity management, authentication, and access control

7.1.1 Authentication

7.1.2 Access control

7.1.3 Identity and credential management

7.1.4 Context-based identity proofing and credential binding

7.1.5 User, service, and hardware authentication

7.1.6 Identity assertion, protection, conveyance, and verification

7.1.7 Access permissions management

7.1.8 Physical access controls

7.1.9 Types of security controls

7.1.10 Identity management, authentication, and access control metrics

7.1.11 Identity and access management dashboard exercise

7.2 Awareness and training

7.2.1 Awareness and training metrics

7.2.2 Awareness and training metrics exercise

7.3 Data security

7.3.1 Data at rest

7.3.2 Data in transit

7.3.3 Data in use

7.3.4 Data backup and recovery

7.3.5 Data security metrics

7.3.6 Data security metrics exercise

7.4 Configuration management

7.4.1 Software maintenance

7.4.2 Hardware maintenance

7.4.3 Log maintenance

7.4.4 Unauthorized software prevention

7.4.5 Secure software development

7.4.6 Platform metrics

7.4.7 Platform security metrics exercise

7.5 SDLC testing exercise

7.6 Technology infrastructure resilience

7.6.1 Networks

7.6.2 Environmental threats

7.6.3 Resilience mechanisms

7.6.4 Resource capacity

7.7 Protection metrics

7.8 Answer to exercise from section 7.5

8 Continuous threat detection

8.1 Implementing continuous threat monitoring systems

8.1.1 Defining strategy

8.1.2 Establishing architecture, implementing data collection, and analysis

8.1.3 Responding to findings

8.2 Open source alternative to continuous threat detection

8.3 Continuous monitoring metrics

8.3.1 Continuous monitoring metrics exercise

8.4 Understanding ATLAS

8.4.1 ATLAS methodology

8.4.2 Review and update

8.4.3 ATLAS benefits

8.5 Determining valid threat detections

8.5.1 False rejection rate

8.5.2 False acceptance rate

8.5.3 Equal error rate

8.5.4 FRR, FAR, and EER metrics

8.5.5 FRR, FAR, and ERR exercise

8.6 Adverse event analysis

8.6.1 Adverse event analysis metrics

8.6.2 Adverse event analysis metrics exercise

8.7 Alternatives to ATLAS and Wazuh

9 Incident management and recovery

9.1 Incident management

9.2 Planning and preparation

9.3 Testing an IRP

9.3.1 Tabletop exercise example

9.3.2 Tabletop exercise metrics

9.3.3 Table for tabletop exercise evaluation sample

9.4 Detection and documentation

9.4.1 Three-tier severity model

9.4.2 Five-tier severity model

9.4.3 Changing severity over time

9.4.4 Practical example of incident documentation

9.4.5 Chain of custody

9.4.6 Incident management metrics

9.4.7 Incident management metrics exercise

9.5 Incident triage and analysis

9.5.1 Case study: Financial institution data breach

9.5.2 Incident metrics

9.5.3 Incident metrics exercise

9.5.4 Incident metrics table

9.6 Notification and communication

9.7 Containing validated incidents

9.8 Evidence gathering and forensic analysis

9.9 Eradication of incidents

9.9.1 Incident response metrics

9.10 Recovering to operational status

9.11 Post-incident and lessons learned

9.11.1 Incident reporting and communication metrics

9.11.2 Incident reporting and communication metrics exercise

Part 3 Beyond the basics: Advanced analytics, machine learning, and AI

10 Advanced cybersecurity metrics

10.1 Risk exposure and predictive analysis

10.2 Risk exposure and predictive metrics

10.2.1 Risk exposure index

10.2.2 Predictive threat index

10.2.3 Risk exposure and predictive metrics exercise

10.2.4 Using open source tools to calculate REI and PTI

10.2.5 How cybersecurity teams work together with open source solutions

10.3 Advanced threat detection

10.3.1 Anomaly detection

10.3.2 Time to predict

10.3.3 Dynamic risk scoring

10.3.4 Advanced threat detection metric exercise

10.4 Effectiveness of AI in cybersecurity

10.4.1 False-positive suppression rate

10.4.2 AI-based decision accuracy rate

10.4.3 Effectiveness of AI in cybersecurity exercise

10.5 Cloud and network threat management

10.5.1 Cloud threat detection

10.5.2 Cloud threat detection accuracy

10.5.3 Attack surface index

10.5.4 Cloud and network threat management metric exercise

10.6 Incident management

10.6.1 Threat hunting

10.6.2 Incident impact estimation

10.6.3 Incident management metric exercise

10.7 Continuous improvement

10.7.1 Risk mitigation

10.7.2 Time to predict

10.7.3 Continuous improvement metric exercise

11 Advanced statistical analysis

11.1 Continuous improvement with statistical metrics

11.1.1 Key concepts for continuous improvement

11.1.2 Free resources for statistical analysis

11.2 Using statistical metrics for continuous improvement

11.3 Implementing a feedback loop

11.4 Finding hidden relationships in data

11.5 Using moving averages to track trends

11.5.1 Moving averages example

11.5.2 Detailed description of the plots

11.6 Grouping similar events for better insights

11.7 Forecasting cybersecurity trends

11.7.1 ARIMA in action

11.8 Bayesian inference

11.8.1 Interpreting the results

11.9 Statistical models for vulnerability management

11.9.1 Interpreting the results

12 Advanced machine learning analysis

12.1 Code requirements

12.1.1 What is AI

12.2 DBSCAN for threat detection

12.2.1 DBSCAN example

12.3 Random forest and SVM for threat detection

12.3.1 Random forest and SVM exercise

12.3.2 Isolation forest and autoencoders exercise

12.4 Natural language processing

12.4.1 NLP exercise

12.5 Deep learning

12.5.1 Deep learning exercise

12.6 Reinforcement learning

12.6.1 Reinforcement learning example in Python code

13 Generative AI in cybersecurity metrics

13.1 Understanding generative AI

13.2 Open source generative AI alternatives

13.2.1 LM Studio

13.2.2 Ollama

13.2.3 LM Studio vs. Ollama

13.2.4 Why we chose LM Studio for this chapter

13.3 A note on LLMs

13.4 Prompt engineering

13.4.1 Why prompt engineering matters

13.5 Prompt engineering in cybersecurity

13.5.1 Basics of prompt engineering

13.5.2 Open source vs. cloud API integration

13.6 Generating and analyzing cybersecurity data

13.6.1 Generating synthetic cybersecurity data

13.6.2 Analyzing the output

13.7 Enhancing reporting and visualization

13.7.1 Failed login reasons reporting

13.8 Automating incident response and trend analysis

13.8.1 Setting the context

13.9 Exploring generative AI

13.9.1 Learning resources

13.9.2 Inspiration to experiment

13.9.3 Your next steps

13.10 Limitations and ethical considerations

Overview

10 Advanced cybersecurity metrics

The chapter explains how rapid advances in AI—especially generative and agentic systems—are reshaping cybersecurity measurement from reactive, point-in-time reporting to proactive, predictive decision-making. It contrasts traditional, backward-looking metrics with advanced metrics that fuse historical incidents, real-time telemetry, and threat intelligence to describe what is happening, why it is happening, who is involved, and what is likely to happen next. Building this capability requires collaboration between security teams, data scientists, and leadership, plus a tooling stack that can collect, enrich, and model large volumes of data to guide risk-informed action.

It introduces a suite of advanced metrics across the defense lifecycle. For risk exposure and forecasting, it presents the Risk Exposure Index (REI) and Predictive Threat Index (PTI). For detection performance, it emphasizes anomaly-centric measures such as Anomalous Behavior Detection Rate (ABDR), Time to Predict (TTP), and Dynamic Risk Scoring (DRS). In cloud and network contexts, it focuses on Cloud Threat Detection Accuracy (CTDA) and the Attack Surface Index (ASI). To gauge AI effectiveness, it tracks False Positive Suppression Rate (FPSR) and AI-based Decision Accuracy (AI-DA), alongside reductions in response time and improved alert prioritization. Throughout, it shows how open-source and commercial platforms—such as vulnerability scanners, SIEM/analytics stacks, EDR/XDR, and threat intelligence platforms—can operationalize these metrics.

Practical scenarios demonstrate setting baselines, targets, and feedback loops to drive measurable improvements: lowering exposure and predicted attack likelihood, raising anomaly detection accuracy while cutting prediction time, tightening cloud detections and shrinking attack surface, and improving threat hunting outcomes while reducing incident impact. The chapter closes by framing continuous improvement around Risk Mitigation Velocity (RMV) and Mean Time to Predict (MTP), reinforcing that the “right” metrics are organization-specific. By aligning metrics to business objectives and tuning models to reduce false positives and accelerate response, teams can move from reactive firefighting to a continuously learning, proactive security posture.

The left shows risk exposure and predictive analytics before implementing improvements. On the right shows risk exposure and predictive analytics after implementing improvements.

On the left shows the advanced threat detection metric dashboard for SecureShield before implementing improvements. On the right shows the metrics after implementing dynamic risk scoring with anomaly detection and real-time threat intelligence feeds.

On the left shows the effectiveness of the AI Dashboard, showing daily false positive rates, true positive detection rates, and the average incident response time. On the right shows the effectiveness of the AI Dashboard after implementing an AI-driven solution, showing daily false positive rates, true positive detection rates, and the average incident response time.

On the left shows the cloud and network threat management dashboard showing CTDA and ASI metrics prior to improvements. On the right shows the cloud and network threat management dashboard showing CTDA and ASI metrics after implementing improvements.

On the left shows the incident management dashboard showing ATHSR and IIE metrics prior to improvements. On the right shows the Incident management dashboard showing ATHSR and IIE metrics after implementing improvements.

On the left shows the continuous improvement dashboard prior to implementing improvements. On the right shows the continuous improvement dashboard after implementing improvement.

Summary

Advanced cybersecurity metrics provide a data-driven approach to anticipating and mitigating future threats.
Predictive analytics, like the Risk Exposure Index (REI) and Predictive Threat Index (PTI), help organizations estimate potential risks and prepare proactive defenses.
Metrics such as Anomalous Behavior Detection Rate (ABDR) and Dynamic Risk Scoring (DRS) enhance the accuracy of threat detection and response, especially when integrated with AI.
False Positive Suppression Rate (FPSR) and AI-Based Decision Accuracy (AI-DA) assess the efficiency and accuracy of AI-driven cybersecurity systems.
Continuous improvement through metrics like Risk Mitigation Velocity (RMV) ensures that organizations adapt quickly to emerging threats and vulnerabilities.

FAQ

What distinguishes advanced cybersecurity metrics from traditional metrics?

Advanced metrics are forward-looking and predictive. They don’t just report what happened; they estimate what is likely to happen next and why. They leverage AI, machine learning, and real-time data to anticipate threats, guide proactive defenses, and align security actions with business impact.

How do Risk Exposure Index (REI) and Predictive Threat Index (PTI) work, and how are they calculated?

REI quantifies current organizational risk by combining vulnerability burden, asset criticality, and likely impact. Simple form: REI = (Total Vulnerability Score / Total Assets) × Business Impact Factor. PTI estimates the probability of future incidents using history and threat intel. Simple form: PTI = (Number of Incidents × Threat Likelihood) / Total Assets. Track both over time to prioritize mitigation and validate improvements.

Which data sources and tools are recommended to build predictive analytics for REI/PTI?

- Vulnerability data: OpenVAS (or Qualys/Nessus) with CVSS scores - Logs and events: Elastic Stack (Elasticsearch, Logstash, Kibana) - Threat intelligence: MITRE ATT&CK, OTX; platforms like Cortex/TheHive - Real-time monitoring/ML integration: Wazuh, Splunk - Modeling and custom ML: Python with scikit-learn/TensorFlow Combine historical incidents, current vulnerabilities, and threat intel into curated datasets for modeling.

How can anomaly detection be measured with ABDR, and how do we reduce false positives?

ABDR (Anomalous Behavior Detection Rate) measures the share of true anomalies correctly detected out of all anomalies flagged. Improve it by: - Building strong baselines of “normal” behavior - Using adaptive ML models that learn over time - Tuning thresholds and features - Feeding quality threat intel - Regularly reviewing false positives/negatives and retraining models

What is Time to Predict (TTP) and Mean Time to Predict (MTP), and why do they matter?

TTP = Prediction Time − Data Ingestion Start Time for a specific event; MTP is the average across events. Shorter TTP/MTP provide earlier warning, enabling preemptive actions (e.g., blocking ransomware pre-encryption). Reduce TTP/MTP by streamlining data pipelines, improving model efficiency/accuracy, and integrating real-time threat intelligence.

What is Dynamic Risk Scoring (DRS), what inputs does it use, and how is it applied operationally?

DRS assigns real-time risk to assets/users/events and updates continuously. Typical inputs: - Current vulnerabilities (e.g., CVSS-based) - Threat intelligence (active IOCs, campaigns) - Monitored activities (anomalous logins, data egress, rare connections) - Contextual criticality (business impact) Example form: DRS = w1×Vulnerabilities + w2×ThreatIntel + w3×Activities + w4×Context. Apply DRS to drive automated containment, risk-based alerting, and prioritized investigations.

How do we measure the effectiveness of AI in security with FPSR and AI-DA?

- FPSR (False Positive Suppression Rate): measures reduction in false positives after AI adoption. FPSR = (FP_before − FP_after) / FP_before. - AI-DA (AI-based Decision Accuracy): overall decision accuracy across TP, TN, FP, FN. AI-DA = (TP + TN) / (TP + TN + FP + FN). Track alongside incident response time and alert prioritization to quantify operational benefit.

Which cloud and network threat metrics should we track, and how do we improve them?

- CTDA (Cloud Threat Detection Accuracy): correctness and efficiency of cloud-specific detections; improve via cloud-native telemetry, tuning rules, and better intel. - ASI (Attack Surface Index): count/weight of internet-exposed assets/services; reduce by decommissioning unused services, hardening configurations, least privilege, segmentation, and WAFs. Use tools like Wazuh/Elastic and cloud-native controls (e.g., AWS GuardDuty, Config; Azure Defender, Policy).

Which incident management metrics support proactive defense, and how do we estimate incident impact?

- ATHSR (Advanced Threat Hunting Success Rate) = Successful Hunts / Total Hunts; indicates proactive detection of stealthy threats. - IIE (Incident Impact Estimation) = Downtime Cost + Data Loss Impact + Recovery Costs; include regulatory, legal, restoration, and reputation effects. Use incident logs, financials, and operational inputs to model realistic impact ranges.

How do we build a continuous improvement loop with RMV and MTP, and who should be involved?

- RMV (Risk Mitigation Velocity) = Average time from detection to resolution; aim to shorten via automation, playbooks, and cross-team workflows. - MTP (Mean Time to Predict): average lead time of forecasts; improve with better data quality, retraining, and model tuning. Create a feedback loop owned by security, data science, and leadership: review metrics regularly, adjust controls/models, retrain, and realign to business risk priorities.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more