A poster on cream paper of a tall, leaning stack of blank tickets. A white robot has pulled one red ticket out of the stack and holds it together with a young researcher in glasses and a green jacket, who reads the small green tag clipped to it.

Research · product security triage

Not every finding is a way in.Trace the ones that are.

Antonio Firmiano

CEO · · 7 min read

Every product security lead I talk to has the same pile. Thousands of scanner findings, tickets nobody closes, and a team of a few people looking after dozens or hundreds of engineers. Finding issues stopped being difficult years ago. The challenge now is working out which findings deserve attention.

Most findings give an attacker little to work with. A known-vulnerable function in a dependency only creates an opportunity for exploitation when your code calls it with input an attacker controls. This post examines what the research tells us about that gap, and how security findings should reach the people responsible for fixing them.

Key findings

  • Under 1% reach the code. Across about three million Maven package versions, about 40% looked affected by a known vulnerability at the package level; under 1% had a call path to the vulnerable code [1].
  • Scanners disagree by nearly 20×. Nine software composition analysis tools reported between 17 and 332 vulnerable dependencies for the same project [3].
  • Exploitation is rare and fast. Fewer than 4% of CVEs have been publicly exploited, and 42% of those were used on the day they were disclosed [4].
  • Noise doesn't get fixed. In a week dedicated to static analysis warnings at Google, engineers fixed 16% of the warnings they reviewed [6].
  • Timing beats accuracy. At Facebook, the same analysis with the same false positive rate went from a fix rate near zero to over 70% when findings arrived with the code change under review [10].

1. Most known-vulnerable code is never reached

The largest study I know of is by Mir, Keshani and Proksch. They examined about three million Maven package versions released over a year and checked them against GitHub's advisory database [1]. At the package level, about 40% appeared affected. When they followed the call graph into the vulnerable code itself, "less than 1% of packages have a reachable call path to vulnerable code in their dependencies, which is far lower than that of a naive dependency-based analysis" (p. 1).

Bar chart. Maven package versions exposed to the ten most widespread CVEs: 786,921 depend on a vulnerable version; 4,485 have a call path to the vulnerable code, about 1 in 175.
Figure 1. For the ten most widespread CVEs, about 1 in 175 exposed package versions actually calls the vulnerable code. Source: [1], §V-C, p. 8.

Two caveats matter. The study covers libraries only, and its call graph ignores control flow (p. 10). As a result, "reachable" is a looser test than whether attacker-controlled input can actually reach the vulnerable code. Smaller studies point in the same direction. Pashchenko et al. found that "about 20% of the dependencies affected by a known vulnerability are not deployed" at all, because they're only used in tests or provided by the runtime [2].

2. The tools don't agree on what's there

Imtiaz, Thorn and Williams ran nine software composition analysis tools on the same application. "The count of reported vulnerable dependencies ranges from 17 to 332 for Maven and from 32 to 239 for npm projects across the studied tools" [3]. The one tool in the study that checked whether vulnerable code could run "found only 2.1% of the alerts were potentially executable and 1.6% of the alerts were actually executed" (§5.3.1, p. 8).

Range chart. Vulnerable dependencies reported for the same project by nine tools: Maven from 17 to 332, npm from 32 to 239.
Figure 2. Same project, nine scanners: the count of vulnerable dependencies varies by nearly 20×. Source: [3], abstract.

These differences have practical consequences. Teams can get dramatically different backlogs depending on which scanner they use. A useful security process needs to establish which findings have an exploitable path through the application.

3. Exploitation is rare, and fast when it happens

CISA's guidance on its Known Exploited Vulnerabilities catalogue says that "less than 4% of the total number of CVEs have been publicly exploited," and that of those, "42% are being used on day 0 of disclosure" [4]. Jacobs et al. measured exploitation activity seen by network sensors and observed it against 6.4% of published vulnerabilities between 2016 and 2022 [5].

Grid of 100 squares, each 1% of published CVEs. Fewer than 4 are red, meaning publicly exploited. 42% of those were used on the day they were disclosed.
Figure 3. Fewer than 4 in 100 CVEs are publicly exploited, but those move fast. Source: [4]; for a sensor-based measure, [5].

This creates two competing demands for security teams. Most findings need careful filtering, while the small subset associated with real exploitation needs immediate attention. Prioritisation should account for both the application's exposure and evidence of active exploitation.

4. Noise has a cost: people stop acting

An unfiltered backlog consumes engineering time and gradually erodes trust in the tools generating it.

Google learned this early. In a 2009 week dedicated to FindBugs warnings, engineers "reviewed a total of 3,954 such warnings (42% of 9,473 total), but only 16% (640) were actually fixed" [6]. Their conclusion was that analysis tools "measure their utility by the number of issues they identify, while integration attempts fail due to the low number of bugs actually fixed or prevented" (p. 65). They now count any finding a developer doesn't act on as an "effective false positive", and keep checks shown in code review under 10% of them (pp. 59, 62).

Bar chart. Google's 2009 FindBugs week: 9,473 warnings reported, 3,954 reviewed by engineers (42%), 640 fixed (16% of reviewed).
Figure 4. Of 9,473 warnings, 640 were fixed. Source: [6], p. 61.

Security operations teams describe a similar experience from the alarm side. One analyst in Alahmadi, Axon and Martinovic's study put it bluntly: "We know 99% of the alarms we generate are false positives, but we still have to look at them" [7]. That's one analyst's estimate, and the authors note that most of those alarms were set off by benign activity. Still, it captures the experience of a team dealing with constant noise. NIST's research on security fatigue found that people faced with too many security decisions slide into resignation and avoid deciding at all [8].

5. What a reachable finding has to show

A reachability claim needs evidence a reviewer can verify quickly. For a finding to earn an engineer's attention, it should show:

  1. Where attacker input enters. A route, a message handler, a file upload, a CLI argument: the first place the attacker's data touches your code.
  2. Every hop to the vulnerable call. File and line for each step, so an engineer can follow it in their editor and verify the path.
  3. What was traced and what was assumed. If the path depends on something the analysis couldn't see, such as whether a service is exposed to the internet, the finding should explicitly mark that assumption.
  4. Whether anyone is exploiting it. A CVE on CISA's Known Exploited Vulnerabilities catalogue has been used in real attacks. Given how quickly those vulnerabilities move, it belongs at the top of the queue.
Diagram of an illustrative attack path. Exposure: internet to load balancer, assumed because the service config is not readable. Entry point: POST /refunds/:id/receipt in routes/refunds.ts line 41, traced. Hop: buildReceipt(orderRef) in services/receipt.ts line 88, traced, no sanitiser on the path. Vulnerable call: exec with orderRef in lib/pdf.ts line 31, in red. Beside it, in green, the fix as a pull request: execFile with an argument array, plus an allowlist for orderRef.
Figure 5. An illustrative reachable finding: every hop with a file and line, the unseen part marked assumed, and the fix next to it.

Reachability analysis has inherent limits. Ponta, Plate and Sabetta, who built one of the first reachability tools for open-source dependencies, are direct about it: the analysis cannot "prove (decide) whether vulnerable code can or cannot be exploited" [9]. In the Imtiaz study, the static and dynamic checks agreed on only 13 alerts [3]. A traced path is strong evidence that the code can be reached. Whether it can be exploited is a separate question. Clear labels help engineers understand the strength of the evidence and make informed decisions. Overstated findings quickly lose their credibility.

6. A finding gets fixed when it arrives with its fix

Good triage gets the right findings to the right people. The next challenge is getting those findings fixed.

Facebook's experience offers some of the clearest evidence of where that happens. When Infer's findings were assigned to engineers in batches, "we had worked hard to get the false positive rate down to what we thought was less than 20%, and yet the fix rate … was near zero." Reporting them on the code change being reviewed produced a very different result: "the fix rate rocketed to over 70%. The same program analysis, with same false positive rate, had much greater impact when deployed at diff time" [10].

Bar chart. Fix rate of Infer findings at Facebook: near zero when assigned in batch, over 70% when reported on the diff being reviewed. Same analysis, same false positive rate.
Figure 6. The same analysis, delivered at review time, was fixed over 70% of the time. Source: [10], p. 66.

The most useful delivery format is a pull request containing the evidence and the proposed fix. Engineers can inspect the path, run their tests, review the change, and decide whether it should merge. The finding arrives where the work already happens, with a clear route to resolution.

What we're building at Arvion

That's the workflow our product-security team runs. It reads your code, open-source dependencies, infrastructure as code and the AI agents in your repositories. For each finding, it traces whether attacker input can reach the vulnerable code and marks each step of the path as traced or assumed. Known-exploited CVEs go to the top of the queue.

For findings with an exploitable path, it writes the fix, checks it again, and opens a pull request. Your engineers review the evidence and decide what gets merged.

We trace the findings that matter and help your team fix them.

References

  1. J. I. Mir, M. Keshani, S. Proksch. "On the Effect of Transitivity and Granularity on Vulnerability Propagation in the Maven Ecosystem." SANER 2023, pp. 201–211. arXiv:2301.07972
  2. I. Pashchenko, H. Plate, S. E. Ponta, A. Sabetta, F. Massacci. "Vulnerable Open Source Dependencies: Counting Those That Matter." ESEM 2018. arXiv:1808.09753
  3. N. Imtiaz, S. Thorn, L. Williams. "A comparative study of vulnerability reporting by software composition analysis tools." ESEM 2021. arXiv:2108.12078
  4. CISA. "BOD 22-01: Reducing the Significant Risk of Known Exploited Vulnerabilities," FAQ. cisa.gov
  5. J. Jacobs, S. Romanosky, O. Suciu, B. Edwards, A. Sarabi. "Enhancing Vulnerability Prioritization: Data-Driven Exploit Predictions with Community-Driven Insights." IEEE EuroS&PW 2023, pp. 194–206. arXiv:2302.14172
  6. C. Sadowski, E. Aftandilian, A. Eagle, L. Miller-Cushon, C. Jaspan. "Lessons from Building Static Analysis Tools at Google." Communications of the ACM 61(4), 2018, pp. 58–66. PDF
  7. B. A. Alahmadi, L. Axon, I. Martinovic. "99% False Positives: A Qualitative Study of SOC Analysts' Perspectives on Security Alarms." USENIX Security 2022. PDF
  8. NIST. "Security Fatigue Can Cause Computer Users to Feel Hopeless and Act Recklessly," 2016. nist.gov
  9. S. E. Ponta, H. Plate, A. Sabetta. "Beyond Metadata: Code-centric and Usage-based Analysis of Known Vulnerabilities in Open-source Software." ICSME 2018, pp. 449–460. arXiv:1806.05893
  10. D. Distefano, M. Fähndrich, F. Logozzo, P. W. O'Hearn. "Scaling Static Analyses at Facebook." Communications of the ACM 62(8), 2019, pp. 62–70. PDF

Find out which of your findings are a way in

Bring one data source. We'll trace which findings an attacker can reach and write the fix for the ones that can. Book 15 minutes with the Arvion team.

Book 15 min