table of content

Part 1 Building the foundation

1 Introducing cybersecurity metrics

1.1 Understanding metrics

1.2 Significance of metrics in cybersecurity

1.3 Traditional vs. innovative metrics

1.4 The changing landscape of cybersecurity

1.5 The importance of frameworks in cybersecurity

1.5.1 HITRUST framework

1.5.2 Center for Internet Security CIS Top 18 framework

1.5.3 NIST Cybersecurity Framework v2.0

1.6 AI and predictive cybersecurity metrics

1.7 Defending against AI

2 Cybersecurity analytics toolkit

2.1 Tool selection

2.1.1 Key tool selection factors

2.2 Dashboard development

2.2.1 Choosing the right metrics

2.2.2 Knowledge points

2.2.3 Dashboard example

2.3 Statistical analysis

2.3.1 Trend analysis

2.3.2 Correlation analysis

2.3.3 Probability distribution

2.4 Integrated security analytics environment

2.4.1 Integrated security analytics environment exercise

2.5 Continuous improvement and iteration

2.5.1 Iteration techniques

2.5.2 Continuous improvement cycle

3 Implementing a security metrics program

3.1 Introducing metrics program design

3.1.1 Key components of a security metrics program

3.2 METRICS methodology

3.2.1 Measure

3.2.2 Evaluate

3.2.3 Threshold

3.2.4 Report

3.2.5 Improve

3.2.6 Communicate

3.2.7 Sustain

3.3 Using the METRICS methodology example

3.3.1 Step 1: Measure

3.3.2 Step 2: Evaluate

3.3.3 Step 3: Threshold setup

3.3.4 Step 4: Report

3.3.5 Step 5: Improve

3.3.6 Step 6: Communicate

3.3.7 Step 7: Sustain

3.3.8 Exercise summary

3.4 Building a metrics dashboard

3.4.1 Different dashboards for different stakeholders

3.4.2 Security dashboard example

3.5 Tools and technologies for metrics collection

3.5.1 Open source tools for metrics collection

3.5.2 Commercial tools for metrics collection

3.5.3 Automation and integration

3.5.4 Scripting for metrics collection

3.6 Common pitfalls in metrics programs

3.6.1 Pitfall 1: Overreliance on quantitative metrics

3.6.2 Pitfall 2: Misalignment with business goals

3.6.3 Pitfall 3: Stagnation and complacency

4 Integrating metrics into business strategy

4.1 Business alignment

4.1.1 Business-aligned security metrics example

4.1.2 Mapping metrics to business performance

4.1.3 Supporting innovation with metrics

4.2 Security metrics alignment with business strategy exercise

4.2.1 Step 1: Understanding MediHealth’s business objectives

4.2.2 Step 2: Selecting key metrics

4.2.3 Step 3: Mapping metrics to business goals

4.2.4 Step 4: Measuring metrics

4.2.5 Step 5: Communicating the Metrics

4.3 Security metric reporting

4.3.1 Presenting metrics in executive reports

4.3.2 Demonstrating return on investment

4.3.3 Communication strategies

4.3.4 Metrics communication exercise

Part 2 The metrics that matter

5 Establishing the foundation

5.1 Governance

5.2 Organizational context

5.2.1 Understanding the differences between mission, vision, and values

5.2.2 Strategic objectives

5.2.3 Organizational metrics

5.2.4 Organizational metrics exercise

5.3 Risk management strategy

5.3.1 Risk mitigation

5.3.2 Risk management metrics

5.3.3 Risk management metrics exercise

5.4 Roles, responsibilities, and authorities

5.4.1 Roles, responsibilities, and authorities metrics

5.4.2 Roles and responsibilities metrics exercise

5.5 Policy, processes, and procedures

5.5.1 Policy, processes, and procedures metrics

5.5.2 Policy, processes, and procedures metrics exercise

5.6 Oversight

5.6.1 Governance structure

5.6.2 Simulated governance structure

5.6.3 Governance exercise

5.6.4 Oversight metrics

5.6.5 Oversight metrics exercise

5.7 Cybersecurity supply chain risk management

5.7.1 Supply chain risk management practices

5.7.2 Cybersecurity supply chain risk management metrics

5.7.3 Supply chain risk management metrics exercise

5.8 Governance metrics

5.9 Answer to exercise 5.6.3

6 Foundations of cyber risk

6.1 Identify

6.2 Asset management

6.2.1 Asset management metrics

6.2.2 Asset management metrics exercise

6.3 Risk assessments

6.3.1 NIST 800-30: Guide for conducting risk assessments

6.3.2 Risk assessment metrics

6.3.3 Risk assessment metrics exercise

6.4 Continuous improvement

6.4.1 Continuous improvement metrics

6.5 Identity metrics exercise

7 Protecting your assets

7.1 Identity management, authentication, and access control

7.1.1 Authentication

7.1.2 Access control

7.1.3 Identity and credential management

7.1.4 Context-based identity proofing and credential binding

7.1.5 User, service, and hardware authentication

7.1.6 Identity assertion, protection, conveyance, and verification

7.1.7 Access permissions management

7.1.8 Physical access controls

7.1.9 Types of security controls

7.1.10 Identity management, authentication, and access control metrics

7.1.11 Identity and access management dashboard exercise

7.2 Awareness and training

7.2.1 Awareness and training metrics

7.2.2 Awareness and training metrics exercise

7.3 Data security

7.3.1 Data at rest

7.3.2 Data in transit

7.3.3 Data in use

7.3.4 Data backup and recovery

7.3.5 Data security metrics

7.3.6 Data security metrics exercise

7.4 Configuration management

7.4.1 Software maintenance

7.4.2 Hardware maintenance

7.4.3 Log maintenance

7.4.4 Unauthorized software prevention

7.4.5 Secure software development

7.4.6 Platform metrics

7.4.7 Platform security metrics exercise

7.5 SDLC testing exercise

7.6 Technology infrastructure resilience

7.6.1 Networks

7.6.2 Environmental threats

7.6.3 Resilience mechanisms

7.6.4 Resource capacity

7.7 Protection metrics

7.8 Answer to exercise from section 7.5

8 Continuous threat detection

8.1 Implementing continuous threat monitoring systems

8.1.1 Defining strategy

8.1.2 Establishing architecture, implementing data collection, and analysis

8.1.3 Responding to findings

8.2 Open source alternative to continuous threat detection

8.3 Continuous monitoring metrics

8.3.1 Continuous monitoring metrics exercise

8.4 Understanding ATLAS

8.4.1 ATLAS methodology

8.4.2 Review and update

8.4.3 ATLAS benefits

8.5 Determining valid threat detections

8.5.1 False rejection rate

8.5.2 False acceptance rate

8.5.3 Equal error rate

8.5.4 FRR, FAR, and EER metrics

8.5.5 FRR, FAR, and ERR exercise

8.6 Adverse event analysis

8.6.1 Adverse event analysis metrics

8.6.2 Adverse event analysis metrics exercise

8.7 Alternatives to ATLAS and Wazuh

9 Incident management and recovery

9.1 Incident management

9.2 Planning and preparation

9.3 Testing an IRP

9.3.1 Tabletop exercise example

9.3.2 Tabletop exercise metrics

9.3.3 Table for tabletop exercise evaluation sample

9.4 Detection and documentation

9.4.1 Three-tier severity model

9.4.2 Five-tier severity model

9.4.3 Changing severity over time

9.4.4 Practical example of incident documentation

9.4.5 Chain of custody

9.4.6 Incident management metrics

9.4.7 Incident management metrics exercise

9.5 Incident triage and analysis

9.5.1 Case study: Financial institution data breach

9.5.2 Incident metrics

9.5.3 Incident metrics exercise

9.5.4 Incident metrics table

9.6 Notification and communication

9.7 Containing validated incidents

9.8 Evidence gathering and forensic analysis

9.9 Eradication of incidents

9.9.1 Incident response metrics

9.10 Recovering to operational status

9.11 Post-incident and lessons learned

9.11.1 Incident reporting and communication metrics

9.11.2 Incident reporting and communication metrics exercise

Part 3 Beyond the basics: Advanced analytics, machine learning, and AI

10 Advanced cybersecurity metrics

10.1 Risk exposure and predictive analysis

10.2 Risk exposure and predictive metrics

10.2.1 Risk exposure index

10.2.2 Predictive threat index

10.2.3 Risk exposure and predictive metrics exercise

10.2.4 Using open source tools to calculate REI and PTI

10.2.5 How cybersecurity teams work together with open source solutions

10.3 Advanced threat detection

10.3.1 Anomaly detection

10.3.2 Time to predict

10.3.3 Dynamic risk scoring

10.3.4 Advanced threat detection metric exercise

10.4 Effectiveness of AI in cybersecurity

10.4.1 False-positive suppression rate

10.4.2 AI-based decision accuracy rate

10.4.3 Effectiveness of AI in cybersecurity exercise

10.5 Cloud and network threat management

10.5.1 Cloud threat detection

10.5.2 Cloud threat detection accuracy

10.5.3 Attack surface index

10.5.4 Cloud and network threat management metric exercise

10.6 Incident management

10.6.1 Threat hunting

10.6.2 Incident impact estimation

10.6.3 Incident management metric exercise

10.7 Continuous improvement

10.7.1 Risk mitigation

10.7.2 Time to predict

10.7.3 Continuous improvement metric exercise

11 Advanced statistical analysis

11.1 Continuous improvement with statistical metrics

11.1.1 Key concepts for continuous improvement

11.1.2 Free resources for statistical analysis

11.2 Using statistical metrics for continuous improvement

11.3 Implementing a feedback loop

11.4 Finding hidden relationships in data

11.5 Using moving averages to track trends

11.5.1 Moving averages example

11.5.2 Detailed description of the plots

11.6 Grouping similar events for better insights

11.7 Forecasting cybersecurity trends

11.7.1 ARIMA in action

11.8 Bayesian inference

11.8.1 Interpreting the results

11.9 Statistical models for vulnerability management

11.9.1 Interpreting the results

12 Advanced machine learning analysis

12.1 Code requirements

12.1.1 What is AI

12.2 DBSCAN for threat detection

12.2.1 DBSCAN example

12.3 Random forest and SVM for threat detection

12.3.1 Random forest and SVM exercise

12.3.2 Isolation forest and autoencoders exercise

12.4 Natural language processing

12.4.1 NLP exercise

12.5 Deep learning

12.5.1 Deep learning exercise

12.6 Reinforcement learning

12.6.1 Reinforcement learning example in Python code

13 Generative AI in cybersecurity metrics

13.1 Understanding generative AI

13.2 Open source generative AI alternatives

13.2.1 LM Studio

13.2.2 Ollama

13.2.3 LM Studio vs. Ollama

13.2.4 Why we chose LM Studio for this chapter

13.3 A note on LLMs

13.4 Prompt engineering

13.4.1 Why prompt engineering matters

13.5 Prompt engineering in cybersecurity

13.5.1 Basics of prompt engineering

13.5.2 Open source vs. cloud API integration

13.6 Generating and analyzing cybersecurity data

13.6.1 Generating synthetic cybersecurity data

13.6.2 Analyzing the output

13.7 Enhancing reporting and visualization

13.7.1 Failed login reasons reporting

13.8 Automating incident response and trend analysis

13.8.1 Setting the context

13.9 Exploring generative AI

13.9.1 Learning resources

13.9.2 Inspiration to experiment

13.9.3 Your next steps

13.10 Limitations and ethical considerations

Overview

2 Cybersecurity analytics toolkit

This chapter introduces a practical cybersecurity analytics toolkit built around four core capabilities—data collection, processing, visualization, and automation—working within a defined scope to accelerate threat detection and response. It explains how diverse telemetry from logs, network traffic, and endpoints is aggregated with tools like SIEMs and IDSs, then parsed, enriched, and correlated to create analyzable data. Effective visualization through dashboards distills large data volumes into trends, anomalies, and events that analysts can interpret quickly, while automation and seamless integration across tools shorten response times and improve the organization’s overall security posture.

Guidance on tool selection emphasizes aligning choices to specific objectives (threat detection, compliance, or incident response), as well as scalability, ease of integration, and usability for the team’s expertise. The chapter compares open-source and commercial options, highlighting tradeoffs among cost, customization, support, and time-to-value; it also underscores the importance of tuning SIEM rules and combining SIEM outputs with vulnerability management to reduce noise and prioritize risk. For dashboards, it lays out principles of clarity, relevance, accessibility, and consistency; shows how to choose metrics that are actionable and trend-aware; and illustrates role-based designs for analysts, executives, and compliance stakeholders that prompt investigation and decisions.

Statistical techniques are presented as essential for data-driven security: trend analysis to track shifts in attack activity, correlation analysis to uncover relationships (with caution that correlation is not causation), and probability distributions to forecast likely incident volumes and allocate resources. The chapter then outlines how to build an integrated analytics environment using SIEM, threat intelligence, and analytical platforms via APIs and automated workflows, optionally orchestrated through SOAR for faster, consistent response and reporting. Finally, it promotes a continuous improvement cycle—review, plan, implement, evaluate—to refine detections, evolve metrics, and incorporate new technologies such as AI, ensuring the toolkit adapts to emerging threats and changing business needs.

Cybersecurity toolkit consisting of the proper scope, tool selection, data collection, processing, visualization, and automation.

Example TechGuard executive dashboard showing an approximation of what the dashboard could look like.

Example TechGuard analyst dashboard showing an approximation of what the dashboard could look like.

Monthly trend analysis of phishing attempts for the trend analysis exercise python code.

Correlation analysis Python code output.

Probability distribution exercise Python code output.

Example of an integrated security analytics environment.

Continuous improvement cycle diagram as an example of how an organization can implement

Summary

A cybersecurity analytics toolkit includes tools for data collection, analysis, and visualization, enabling organizations to monitor, detect, and respond more effectively to security threats.
When selecting tools, organizations should consider functionality, integration ease, and scalability, ensuring that each tool aligns with security objectives and can adapt to future needs.
Effective dashboards enhance decision-making by clearly presenting key metrics, while strong communication practices ensure that security insights reach relevant stakeholders in an understandable format.
Statistical and AI techniques, such as anomaly detection and predictive modeling, equip analysts to uncover patterns and detect potential threats, strengthening overall security posture.
An integrated security analytics environment centralizes data and automates analysis, providing a cohesive approach to threat management across the organization.
Continuous improvement is essential for maintaining a relevant toolkit, with regular updates and adjustments ensuring the tools remain effective as security needs evolve.

FAQ

What are the core components of a cybersecurity analytics toolkit and what do they do?

A complete toolkit spans data collection (logs, network, endpoints via SIEM/IDS), processing and normalization (parse, enrich, correlate), visualization (dashboards and graphs), and automation/integration (alerts, reporting, incident workflows). Together they deliver timely, actionable insights within the defined scope.

How should we choose the right security analytics tools for our organization?

Start by clarifying objectives (threat detection, compliance, incident response) and needed capabilities (collection, analysis, visualization). Evaluate integration with existing systems, scalability, usability, team skills, support model, and budget to ensure long-term fit.

What are the trade-offs between open-source and commercial security tools?

Open-source tools are cost-effective and customizable with community support but require more setup and expertise. Commercial tools offer out-of-the-box features and vendor support, at higher licensing costs and with potential vendor lock-in or reduced flexibility.

Which tool categories are essential, and what are common examples?

Data collection/monitoring: Wireshark, Sysmon, OSQuery. Threat intelligence: MISP, Anomali. SIEM: ELK Stack, Splunk, QRadar. Vulnerability management: OpenVAS, Tenable Nessus, Qualys. Together they provide visibility, context, correlation, and risk prioritization.

How can we tune a SIEM to reduce false positives without missing real threats?

Continuously refine correlation rules and detection thresholds, update rules regularly, and apply machine learning–driven analytics. Aim for a balance between sensitivity and specificity to limit alert fatigue while detecting emerging threats.

What are best practices for designing effective cybersecurity dashboards?

Follow clarity, relevance, accessibility, and consistency. Keep visuals simple with clear labels and logical layout; choose appropriate chart types (bars for comparisons, lines for trends, pies for proportions); and make views actionable with drill-downs.

Which metrics should appear on analyst, executive, and compliance dashboards?

Analysts: real-time alerts, event details, vulnerability status, threat intel/IoCs. Executives: risk exposure, compliance status, incident trends, MTTR, financial/ROI impact. Compliance: framework status (SOC2, HIPAA, GDPR), control coverage, audit findings/remediation.

How can statistical analysis improve security operations?

Trend analysis highlights changes over time (e.g., rising phishing attempts). Correlation analysis surfaces relationships (e.g., late-night logins and failed attempts). Always apply context—correlation does not imply causation—to avoid misinterpretation.

How do probability distributions help forecast security incidents?

Use distributions like Poisson to estimate the likelihood of monthly incident counts (e.g., most probable near the average). These forecasts guide staffing and capacity planning and should be revisited as threats evolve.

How do we build an integrated security analytics environment and keep it improving?

Integrate SIEM, threat intelligence, and analytics via APIs/data pipelines; automate enrichment and SOAR workflows; and publish role-based dashboards with scheduled reports. Operate a feedback loop—review, plan, implement/test, evaluate/refine—to adapt detections, integrations, and metrics to new threats.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more