SBOM Toolchains Can Skew Vulnerability Results by 5,000+ CVEs
Why SBOM generator choice materially impacts vulnerability detection accuracy in DevSecOps pipelines
A 2024 study analyzing 2,313 Docker images found that changing only the SBOM generator — while keeping the container and analyzer constant — altered vulnerability results by up to 5,456 CVEs.
Same-vendor toolchains reported more findings than mixed stacks. Certain combinations produced near-zero results. Approximately 43.7% of images triggered tool processing failures.
SBOM generation is not neutral. Standardize and validate your toolchain or risk underreporting vulnerabilities.
Introduction
Security teams often treat vulnerability scanning as deterministic:
Container Image + Vulnerability Database = Vulnerability Report
If the scan completes successfully and returns low findings, we assume the artifact is safe to ship.
However, recent research suggests that assumption does not always hold in SBOM-based workflows.
A 2024 academic study titled “Impacts of Software Bill of Materials (SBOM) Generation on Vulnerability Detection” demonstrates that simply changing the SBOM generator — while keeping the container image and vulnerability analyzer constant — can produce differences of thousands of reported vulnerabilities for the same artifact.
Research Reference:
Shamim et al., Impacts of Software Bill of Materials (SBOM) Generation on Vulnerability Detection
NIOS Lab, Montana State University, 2024
https://nios.montana.edu/cyber/products/Impacts%20of%20Software%20Bill%20of%20Materials%20-%20SBOM%20-%20Generation%20on%20Vulnerability%20Detection%20Final%20Version.pdf
This research does not imply SBOMs are ineffective. It demonstrates that interoperability assumptions must be validated.
What the Study Tested
The researchers generated SBOMs from 2,313 Docker images. The container artifacts were held constant. Only the SBOM generation tool and format were varied.
SBOM Generators
Syft (Anchore)
Trivy (Aqua Security)
SBOM Formats
CycloneDX 1.5
SPDX 2.3
Vulnerability Analyzers
Trivy
Grype
CVE-bin-tool
The goal was to isolate how SBOM generation affects downstream vulnerability detection.
The 5,456 CVE Difference
When keeping the analyzer constant (Trivy) and switching only the SBOM generator (Syft → Trivy), the difference in reported vulnerabilities for a single image ranged from:
–94 to +5,456 CVEs
Same image.
Same analyzer.
Different SBOM generator.
This demonstrates that SBOM generation is not a neutral preprocessing step. It directly influences vulnerability matching outcomes.
Why Results Diverge
The paper highlights two primary causes.
1. Vendor Coupling Effect
Same-vendor generator and analyzer combinations consistently reported higher vulnerability counts than mixed-vendor combinations.
Examples from the study:
Syft + Grype (Anchore stack) → highest median detections
Trivy + Trivy (Aqua stack) → second highest
Mixed stacks (e.g., Syft + CVE-bin-tool) → significantly lower findings
This suggests that vendor ecosystems may share normalization logic, metadata handling assumptions, or matching strategies not fully preserved across tools.
While CycloneDX and SPDX aim to standardize interoperability, implementation details still matter.
2. SBOM Format Ambiguity
SBOM format also introduced variability, though less than generator choice.
The study observed inconsistencies in:
Supplier field interpretation
Package naming normalization
CPE (Common Platform Enumeration) resolution
If an analyzer cannot correctly map package metadata to vulnerability databases (e.g., NVD, GitHub Advisory), vulnerabilities may not be reported.
No match results in silent false negatives.
Tool Failures and Dropout
Approximately 43.7% of images were excluded in parts of the study because certain tool and format combinations failed to process generated SBOMs.
This indicates that SBOM pipelines may fail in two ways:
Semantic failure — incorrect or missing vulnerability matches
Mechanical failure — parsing errors or tool crashes
In CI/CD environments, fail-open behavior can introduce significant risk.
Security Implications
For organizations relying on SBOM-based scanning for:
Release gating
Compliance reporting
Executive metrics
Risk scoring
these findings introduce measurable uncertainty.
A “clean” SBOM-based scan does not necessarily indicate absence of vulnerabilities. It may indicate metadata mismatch or interoperability limitations.
Practical Recommendations
1. Standardize Generator and Analyzer Pairing
Where possible, keep generation and analysis within the same vendor ecosystem unless cross-tool compatibility has been validated.
Interoperability should be tested — not assumed.
2. Add CI/CD Sanity Checks
Implement automated controls such as:
Failing builds if dependency counts drop unexpectedly
Flagging images that report zero vulnerabilities despite known dependencies
Ensuring scanner crashes fail closed
Zero findings should trigger investigation, not celebration.
3. Periodically Cross-Validate
Do not rely solely on SBOM-based detection.
Occasionally compare:
SBOM-based results
Direct container filesystem scans
Alternative analyzers
This helps detect silent false negatives caused by metadata interpretation gaps.
Conclusion
The SBOM ecosystem continues to mature. The research demonstrates that SBOM generation materially impacts vulnerability detection outcomes.
Treating SBOM generation as a commoditized, interchangeable step in your pipeline introduces risk.
Before trusting vulnerability dashboards derived from SBOM workflows, validate the generation step itself.
Because “zero vulnerabilities” may simply mean “zero successfully matched.”