セキュリティメトリクス: Measuring What Actually Matters

ホーム Manifesto ISMS

📊 セキュリティメトリクス: Measuring What Actually Matters

"What gets measured gets managed. What doesn't get measured gets breached." — Hagbard Celine

"If you can't measure it, you can't prove it works. If you measure the wrong thing, you prove nothing. If you only measure what makes you look good, you prove you're lying to yourself."

🍎 The Golden Apple: Vanity Metrics vs. Real Security (or: How to Lie with Statistics While Feeling Secure)

Security teams love metrics. Dashboards full of numbers. Colorful charts. Executive briefings with upward-trending lines. It's security theater with better production values.

Most security metrics measure the wrong things. FNORD. Are you measuring security or measuring the appearance of measuring security? (Asking for a friend. The friend is paranoia.)

Number of policies written? Irrelevant if unenforced (PDF doesn't stop hackers). Security training completion rate? Useless if employees still click phishing links (compliance ≠ comprehension). Vulnerability scan count? Meaningless without patch deployment speed (scanning vulnerabilities you never fix is like diagnosing cancer and celebrating the diagnosis). The security-industrial complex sells tools that generate metrics that prove you bought tools. Circular reasoning is circular.

Measure outcomes, not activities. Measure risk reduction, not effort expended. Or keep measuring inputs and wondering why outputs still suck. Your choice. Nothing is true.

ILLUMINATION FOR THE INITIATED: Vanity metrics make executives feel good (dopamine hit from green dashboards). Real metrics reveal uncomfortable truths (like that your security posture is held together by hope and duct tape). Choose truth over comfort—security depends on it. But truth is painful. Which is why most organizations choose comfort. And get breached. The cycle continues. FNORD.

🛡️ The Five Categories of セキュリティメトリクス That Matter

1. Detection & Response

How fast do you detect and stop attacks?

MTTD: Mean Time To Detect (hours/days). MTTR: Mean Time To Respond (hours). MTTR: Mean Time To Recover (hours).

2. Vulnerability Management

How fast do you patch critical risks?

Time to patch critical CVEs: Days from disclosure to deployment. Open high/critical vulnerabilities: Absolute count, trending down.

3. Incident Trends

Are you getting better or worse?

Incidents per month: Trending down? Severity distribution: More critical or more informational? Repeat incidents: Learning from failures?

4. Access Control

Who has access to what?

Accounts with excessive privileges: Count, review frequency. Unused accounts: Dormant credentials are risk. MFA coverage: Percentage of critical systems.

5. Security Awareness

Do users fall for attacks?

Phishing simulation click rate: Percentage clicking malicious links. Reported suspicious emails: User vigilance indicator. Policy violations: Incidents from user mistakes.

CHAOS ILLUMINATION: Metrics drive behavior. Measure the wrong thing, get the wrong outcome. Measure vulnerabilities found? Teams stop looking. Measure vulnerabilities fixed? Teams hunt obsessively. This is Campbell's Law in action—the observed statistical regularity tends to collapse once pressure is placed on it for control purposes. Question every metric. Especially yours.

⏰ Leading vs. Lagging Indicators: The Paradox of Time

Lagging indicators tell you what happened. They're forensic evidence of your security posture—the crime scene photos of yesterday's decisions. Useful for autopsies. Not so useful for preventing the murder.

Leading indicators predict what will happen. They're the smoke before the fire, the tremor before the earthquake, the FNORD you almost didn't notice. They're uncomfortable because they reveal problems before they become disasters—and who wants to admit their house is on fire while there's still time to grab the extinguisher?

🔴 Lagging Indicators (Autopsy)

  • Number of incidents: Counting corpses
  • Breach impact: Calculating damage
  • Time to remediate: How long you bled
  • Compliance audit findings: Report card from last semester
  • Security budget spent: How much you invested in yesterday

Looking backward feels safe. The disaster already happened. Nothing left to prevent.

🟢 Leading Indicators (Prevention)

  • Vulnerability age distribution: How long known risks sit unpatched
  • Patch deployment velocity: Speed from disclosure to protection
  • Phishing simulation trends: User vigilance improving or declining?
  • Security awareness engagement: Are people learning or clicking through?
  • Access review completion rate: Proactive privilege hygiene
  • Unpatched critical systems: Ticking time bombs still armed

Looking forward creates anxiety. The disaster hasn't happened yet. You could still prevent it. That means responsibility.

TEMPORAL ILLUMINATION: Organizations love lagging indicators because they measure failure that's already happened—no pressure to prevent what's done. Leading indicators reveal failure in progress—uncomfortable truths about what you're ignoring right now. Which explains why dashboards show mostly lagging metrics with reassuring historical trends. The past is safe. The future is terrifying. FNORD.

🤖 Automation: Measuring Without Theater

Manual security metrics are security theater with spreadsheets. If a human has to manually collect the metric, the metric lies. Not because humans are dishonest (though some are), but because manual collection introduces:

Automated metrics don't lie. They just reveal truths you'd rather not see.

🛠️ Automation Stack Examples

  • GitHub Advanced Security: Continuous code scanning, secret detection, dependency alerts
  • AWS Config Rules: Real-time infrastructure compliance monitoring
  • OpenSSF Scorecard: Weekly supply chain security assessment
  • SonarCloud Quality Gates: Every commit quality and security validation
  • FOSSA License Scanning: Continuous SBOM generation and license compliance
  • GuardDuty: Automated threat detection without human intervention

📊 Continuous Measurement Patterns

  • Every commit: SAST, secret scanning, dependency checks
  • Every deploy: Container vulnerability scanning, SBOM generation
  • Every hour: Infrastructure configuration compliance
  • Every day: Vulnerability age trending, patch status
  • Every week: OpenSSF Scorecard, access reviews, security posture

Think for yourself: If your security metric requires a human to manually collect it, ask why it's not automated. The answer usually reveals whether you're measuring security or measuring the appearance of measuring security. FNORD.

📊 Visualization: Dashboards That Tell Truth vs. Dashboards That Lie

Green dashboards are the opiate of security executives. "Everything's green! We're secure!" Translation: "I carefully selected metrics that make me look good."

Good security dashboards make you uncomfortable. They highlight problems. They trend risks. They show age distributions of unpatched vulnerabilities. They don't show "100% compliant" unless you're actually 100% compliant (you're not).

🎭 Dashboard Theater

  • All green tiles: "Everything's fine!" (Narrator: It wasn't.)
  • Percentage completions: "99% patched!" (1% = entire DMZ)
  • Trend lines pointing up: "Improving!" (More scans ≠ more security)
  • Compliance percentages: "97% compliant!" (3% = all authentication)
  • Activity metrics: "Blocked 10M threats!" (9.999M = spam)

"Look at all these green numbers! We must be secure. Right? Right??"

✅ Truth Dashboards

  • Red/Yellow/Green with context: "23 critical vulns, 19 >30 days old"
  • Age distributions: Histograms showing how long risks persist
  • Trend arrows (both directions): "MTTR improving, but MTTD worsening"
  • Absolute counts: "4 unpatched critical systems in production"
  • Ratio metrics: "800% net resolution rate (closed vs opened)"

Uncomfortable truths drive action. Comfortable lies drive breaches.

VISUALIZATION WISDOM: If your security dashboard doesn't occasionally make executives uncomfortable, it's not showing security—it's showing what you want them to see. The best metrics reveal problems while there's still time to fix them. The worst metrics hide problems until it's too late. Choose discomfort over delusion. Nothing is true when everything is green. FNORD.

🎯 The KPI Framework: What Actually Matters

Key Performance Indicators separate signal from noise. If you measure everything, you measure nothing. Focus on metrics that:

  1. Drive business decisions: "Should we delay release to patch this?"
  2. Reveal risk trends: "Are we getting more or less secure over time?"
  3. Enable prioritization: "What should we fix first?"
  4. Validate investments: "Is this security tool actually working?"
  5. Support compliance: "Can we prove we're meeting requirements?"

🔴 Critical KPIs (Business Impact)

Revenue protection, regulatory compliance, operational continuity

  • Critical Vulnerabilities in Production: Target: 0 (Zero tolerance)
  • Mean Time To Remediate (MTTR): Target: <7 days for Critical
  • MFA Coverage: Target: 100% for production systems
  • Backup Success Rate: Target: ≥99.5%
  • RTO Achievement Rate: Service restoration within targets
  • License Policy Violations: Target: 0 (Legal risk)

🟠 Operational KPIs (Risk Management)

Service quality, development efficiency, proactive security

  • Vulnerability Age Distribution: How long risks sit unpatched
  • Patch Deployment Velocity: Days from disclosure to deployment
  • Alert Resolution Rate: Currently: 800% net positive
  • Scanner Coverage Ratio: % of repositories with automated security
  • Secret Detection Coverage: Currently: 100% (0 bypassed)
  • Build Success Rate: Development velocity indicator

OpenSSF Scorecard Targets: Hack23 maintains ≥9.5/10 across all repositories (currently: CIA 7.2, improving). This single composite metric encompasses 18 security categories from security policy to SAST to signed releases. One number, comprehensive security posture visibility.

KPI REALITY CHECK: Your KPIs reveal your priorities. If you measure training completion but not phishing click rates, you care about compliance paperwork, not actual user vigilance. If you measure vulnerabilities found but not vulnerabilities fixed, you care about looking thorough, not being secure. Your metrics expose your values. Choose metrics that expose your actual security, not your desired appearance.

📋 Hack23's セキュリティメトリクス Dashboard

Our metrics program focuses on risk reduction: ISMS-PUBLIC Repository | セキュリティメトリクス

META-ILLUMINATION: Security metrics aren't about proving you're perfect—they're about proving you're improving. Trend matters more than absolute numbers. A system that was 60% secure last quarter and is 75% secure this quarter is safer than a system claiming 100% security with no evidence of measurement. Progress beats perfection. Honesty beats theater.

🔍 GitHub セキュリティメトリクス: Real-Time Transparency

Hack23 practices radical transparency through live public security metrics. Our GitHub Security Organization Overview exposes actual vulnerability management performance:

This is what real security metrics look like. Not all green. Not perfect. Not hiding problems. Measuring actual security posture with automated tooling that can't be gamed.

TRANSPARENCY ILLUMINATION: Publishing your real security metrics publicly is Chapel Perilous for organizations. It reveals you're not perfect. Competitors see your weaknesses. But it also proves you're honest about security—which builds more trust than performative perfection. Customers prefer "We patch critical vulnerabilities in 8 days and are working to improve" over "We're 100% secure, trust us." One is evidence. The other is marketing bullshit. FNORD.

📊 Compliance Monitoring Metrics

ISO 27001 A.5.36 demands systematic policy compliance monitoring. Compliance metrics prove you're following your own policies—or expose that you're not. Most organizations measure compliance once per year (audit season theater). Continuous compliance monitoring reveals drift in real-time.

🏛️ Framework Implementation Coverage

  • ISO 27001:2022: 63% control coverage (target: 85% by 2026)
  • NIST CSF 2.0: 85% function alignment (mature)
  • CIS Controls v8.1: 91% IG1, 81% IG2, 65% IG3
  • OpenSSF Scorecard: Average 7.8/10 (target: 9.5+)

Honest assessment beats checkbox compliance

📋 Policy Compliance Metrics

  • Policy Review Currency: 100% (all policies reviewed on schedule)
  • Vulnerability SLA Compliance: Critical <7d, High <30d, Medium <90d
  • Backup Success Rate: 99.7% (exceeds 99.5% target)
  • Change Success Rate: Tracked per Change Management policy
  • Incident Response Time: 18 minutes average (improving from 22 min)

Continuous measurement reveals policy effectiveness

Compliance isn't a destination—it's a trajectory. Measuring compliance maturity over time reveals whether you're building security culture or just checking boxes for auditors. Progressive improvement beats static certification.

🎯 Vanity Metrics vs. Real Metrics

🎭 Vanity Metrics

  • ❌ Number of security policies
  • ❌ Training completion percentage
  • ❌ Vulnerability scans performed
  • ❌ Security tools purchased
  • ❌ Compliance certifications held
  • ❌ Security team size

Why they're useless: Measure activity, not outcome.

✅ Real Metrics

  • ✅ Mean Time To Detect/Respond
  • ✅ Phishing simulation click rate
  • ✅ Critical vulnerabilities patched <48h
  • ✅ False positive rate in alerts
  • ✅ Repeat incidents (learning failure)
  • ✅ Accounts with excessive privileges

Why they matter: Measure security posture improvement.

🔍 The Five Principles of Effective セキュリティメトリクス

  1. Measure Outcomes, Not Activities - "Vulnerabilities fixed" > "vulnerability scans run"
  2. Focus on Trends, Not Snapshots - Direction matters more than absolute numbers
  3. Make Metrics Actionable - Every metric should drive specific decisions
  4. Avoid Perverse Incentives - Don't measure what can be gamed without improving security
  5. Report Honestly - Metrics revealing problems are valuable—hiding problems is deadly
ULTIMATE ILLUMINATION: Security metrics should make you uncomfortable. If your dashboard shows only green, you're measuring the wrong things—or lying to yourself.

🎯 Metrics for Different Audiences

Different stakeholders need different metrics:

Executive Metrics

  • Risk reduction over time
  • Compliance status
  • Incident trend (severity & frequency)
  • Budget efficiency (risk/$)

Technical Team Metrics

  • MTTD/MTTR by incident type
  • Vulnerability age distribution
  • Alert false positive rate
  • Coverage gaps by asset type

Audit Metrics

  • Control effectiveness evidence
  • Policy compliance rates
  • Access review completion
  • Training completion & testing

🎯 Conclusion: Measure What Matters, Question Everything

Security metrics are observability for your defensive consciousness. Like meditation revealing mental patterns, metrics reveal security patterns. Are you getting more secure over time? Where are you weakest? What should you fix next? Good metrics answer these questions. Bad metrics avoid them.

The Five Metric Categories (Law of Fives naturally emerges): Detection & Response, Vulnerability Management, Incident Trends, Access Control, Security Awareness. Each measuring different dimensions of security reality. Together forming a complete picture—if measured honestly.

MTTD and MTTR show detection capability. Patching speed shows vulnerability management. Incident trends show learning effectiveness. Phishing rates show awareness impact. But only if you're honest about the numbers.

Most security metrics are vanity theater. Dashboards showing only green lights. "Number of phishing emails blocked" (measuring your mail filter, not your security). "Training completion percentage" (measuring compliance, not comprehension). "Vulnerability scans performed" (measuring activity, not outcomes).

Vanity metrics stroke egos. Real metrics drive improvement. Choose discomfort over delusion.

🍎 ULTIMATE REVELATION: THE MEASUREMENT PARADOX

Metrics measure security. But metrics also CREATE security culture—or destroy it. Measure vulnerabilities found? Teams stop looking (bad news punished). Measure vulnerabilities fixed? Teams hunt obsessively (problems rewarded). Measure false positive rates? Analysts tune alerts carefully. Ignore false positives? Alert fatigue kills detection.

You become what you measure. Choose metrics that reward the behavior you want:

  • Measure time to remediation → Teams fix faster
  • Measure phishing simulation trends → Users learn vigilance
  • Measure repeat incidents → Organizations learn from failures
  • Measure coverage gaps → Teams close blind spots
  • Measure false positive rates → Analysts tune quality over quantity

But remember Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." Metrics optimized become gamed. Question every metric—especially when it makes you look good. The metric that feels comfortable is probably lying to you.

Are you paranoid enough about your metrics lying to you? If your dashboard shows only green, you're either: (a) perfectly secure (you're not), or (b) measuring the wrong things. FNORD.

Think for yourself, schmuck! Question your security metrics. Especially when they tell you what you want to hear. Security theater performs measurement. Real security measures reality.

🔮 The Future of Security Measurement

Security metrics are evolving from static dashboards to dynamic consciousness. Imagine security measurement as continuous observability of your defensive reality:

The psychedelic vision: Security metrics become a living nervous system for your infrastructure. Not dashboards you check—awareness you inhabit. Not numbers you report—consciousness you experience. Measurement as meditation. Metrics as enlightenment.

But only if you measure truth instead of comfort. Only if you choose reality over theater. Only if you're paranoid enough to question whether your green dashboard is hiding red reality.

All hail Eris! All hail meaningful measurement!

Ready to implement measurable security outcomes? Learn about Hack23's cybersecurity consulting services and our unique public ISMS approach.

All hail Eris! All hail Discordia!
"Think for yourself, schmuck! Question your metrics—especially when they make you look good. Nothing is true when everything is green. Are you paranoid enough about what your dashboards aren't showing you?"
🍎 23 FNORD 5
— Hagbard Celine, Captain of the Leif Erikson

P.S. If this post made you uncomfortable about your security metrics, good. Discomfort drives improvement. Comfort drives breaches. Choose paranoia over complacency. Question authority—especially the authority of green dashboards telling you everything's fine. Because it's not. And admitting that is the first step to actual security. FNORD.