Severity Scores are More Subjective Than You Think
In the Vulnerability Management processes, we treat the CVSS scores as reliable information. We build automated ticketing pipelines around it, we set SLAs based on its decimals, and we report “Criticals” to leadership with absolute confidence. But what if the math we rely on is built on a foundation of human inconsistency?
An empirical study published sheds light on a growing “reliability crisis” in CVSS v3.1 scoring. After surveying nearly 200 professional security analysts, the data suggests that for several key metrics, we might as well be flipping a coin.
The ‘Scope’ Problem
If you’ve ever debated whether an XSS vulnerability should have an “Unchanged” or “Changed” Scope, you aren’t alone. The study found that Scope (S) is the most inconsistently rated metric in the entire framework. For common vulnerabilities like SQL Injection, analysts were split almost exactly 50/50.
“If you ask 10 people for their opinion on Scope, you get 10 coin tosses.” — Survey Participant
Because a Scope change (S: C) increases the weight of impact metrics, this single subjective choice can swing a score from a manageable 7.5 to a board-level 9.0. This isn’t just a technical nuance; it’s the difference between a routine patch and a midnight fire drill.
Consistency Over Time
Perhaps the most jarring finding wasn’t the disagreement between different analysts, but the disagreement of analysts with themselves. In a follow-up study conducted 9 months later:
68% of participants assigned different severity ratings to the same vulnerabilities they had previously assessed.
30% of professional users admitted to never reading the official documentation, relying instead on the high-level tooltips in the online calculator.
Strategic Takeaways for Product Security
For those of us securing complex SDLCs and building automated security pipelines, this research demands a shift in strategy:
Automate the Context: Don’t leave metrics like “Attack Vector” or “Scope” to manual interpretation. Use DAST and asset inventory data to programmatically inject these values based on the application’s actual architecture.
Adopt Decision Trees: Shift toward frameworks such as SSVC (Stakeholder-Specific Vulnerability Categorization). While CVSS indicates technical severity, SSVC helps determine priority based on mission impact and active exploitation.
Standardize Internal Guides: Since the official docs are rarely read, create a “one-pager” tailored to your organization’s technology stack to ensure every engineer defines “Security Authority” consistently.
Conclusion
CVSS is a powerful tool, but it measures severity, not risk. As we continue to automate our security posture, we must account for the human variance that these numbers represent. Accuracy in triage isn’t just about the formula; it’s about the consistency of the input.