AppSec & Supply Chain SecurityDecember 4, 2025

Researchers studied how well the top frameworks mitigate modern attack techniques. They found serious security gaps.

With the increase in supply chain attacks, multiple frameworks to reduce the risks of falling prey to such an attack have been widely adopted, but we have had little insight into how well they actually mitigate supply chain threats. That’s what a team of researchers set out to provide in their report, “Closing the Chain: How to Reduce Your Risk of Being SolarWinds, Log4j, or XZ Utils.”

The researchers — Sivana Hamer, Jacob Bowen, Md Nazmul Haque, and Laurie Williams, of North Carolina State University; Chris Madden, of Yahoo; and Robert Hines, of Yahoo and Short Hill Advancements — mapped the attack techniques used in three high-profile supply chain compromises — SolarWinds, Log4j, and XZ Utils — to 73 tasks in 10 supply chain frameworks — noting, however, that “Given that organizations have limited budget, time, and resources for software supply chain security, adopting all 73 tasks is impractical.”

Worse, those 73 tasks did not include three that are known as capable of mitigating risks such as those posed by those three notable supply chain attacks . “Thus, software products would still be vulnerable to software supply chain attacks even if organizations adopted all recommended tasks,” the researchers said.

The researchers also reported that of the 73 tasks, fewer than half (34) mitigated techniques in the three high-profile attacks. Taking those 34 tasks together with the three missing ones, the researchers found that all 37 mitigated techniques used in the SolarWinds attack, 34 mitigated techniques used in Log4j, and 29 mitigated techniques used in XZ Utils. Only 27 of the tasks mitigated techniques in all three attacks, the researchers said.

Here’s what you need to know about these important findings about the effectiveness of application security (AppSec) frameworks — and why the frameworks are not up to the job of managing supply chain risk.

Join webinar: AI Redefines Software Risk: Develop a New Playbook

Frameworks’ shortcomings are call to action

Rosario Mastrogiacomo, chief strategy officer at Sphere Technology Solutions, said the researchers’ findings underscore the need for continuous threat modeling and a more dynamic approach to defense.

Relying solely on predefined checklists leaves organizations vulnerable to emerging tactics. Security leaders should consider augmenting frameworks with intelligence-driven insights and tailored mitigation strategies to address operational blind spots, especially those related to identity, access, and clarity of ownership.

Rosario Mastrogiacomo

Jeff Williams, CTO and co-founder of Contrast Security, said that misalignment of the frameworks’ tasks with mitigation shows the age of some of the frameworks in the study.

Most of the 10 frameworks, such as NIST SSDF [the National Institute of Standards and Technology’s Secure Software Development Framework], SLSA [Supply-chain Levels for Software Artifacts], SCVS [OWASP’s Software Component Verification Standard], and BSIMM [Building Security in Maturity Model], were created many years before real empirical data was available from incidents like SolarWinds, Log4j, or XZ Utils. As a consequence, they are more theoretical and don’t specifically target the exact techniques used in the [SolarWinds, Log4j, and XZ Utils] attacks.

Jeff Williams

There are a lot of angles to software supply chain security, and a framework can help organizations make progress, Williams said, but he added, “I just wish the frameworks were all built off the same threat model [and were] clear about what risks they are intended to cover and how they prioritize different types of attacks. Currently, some frameworks shoot for perfection, and others are more practical.”

Mastrogiacomo said that frameworks provide a necessary structure for organizations to assess, benchmark, and improve their security postures. However, they are only as effective as the accuracy and timeliness of the risks they address.

Frameworks can help standardize best practices, but they shouldn’t be mistaken for comprehensive or static defenses — especially in the face of evolving techniques like those seen in [the major supply chain attack] incidents.

Rosario Mastrogiacomo

Many frameworks were designed to be broadly applicable across industries, necessarily sacrificing depth for general applicability, Mastrogiacomo said. “This can result in controls that fail to map directly to specific attack vectors. Additionally, gaps can occur when frameworks don’t keep pace with novel threat techniques or when implementers misunderstand the intent of certain controls,” he said.

Visi Stark, co-founder of the Vertex Project, said that in cybersecurity, frameworks are like battle plans. “They never survive first contact with the enemy. Good frameworks are a starting point to standardize terminology and drive consistent outcomes but need to be adapted to each use case and should never be treated as one size fits all,” he said.

Stark said that compliance-oriented frameworks such as the SSDF often focus on checklists and easily accomplished tasks. “Real mitigations often involve complex trade-offs between resources (read, money) and impact, making them more difficult to generalize and quantify,” he said.

Organizations need to keep in mind that any framework is a simplification and the devil is always in the details. Checking all the boxes specified by a given software supply chain security framework does not, in itself, make you secure from software supply chain attacks.

Visi Stark

Why the frameworks are failing

Contrast Security’s Williams said the key takeaway of the research is that a framework, by itself, is insufficient to mitigate the risk of your software supply chain being compromised by a well-resourced attacker.

Organizations should view the frameworks as guidance and focus on improving their resilience over time while delivering strong metrics to management.

Jeff Williams

After mapping framework tasks to mitigations, the researchers ranked the tasks and recommended the top 10 as a “starter pack” for organizations:

• Role-based access control
• System monitoring
• Boundary protection
• Monitoring changes to configuration settings
• Environmental scanning tools
• Security design review
• Dependency updates
• Information flow enforcement
• Protecting information at rest
• Risk-based vulnerability remediation

Williams said organizations should focus on those.

Why binary analysis is key to managing supply chain risk

Patrick Enderby, senior product marketing manager at ReversingLabs (RL), said that the study reveals one critical truth: Even when organizations follow every best practice outlined across leading supply chain security frameworks, significant gaps remain.

This means traditional compliance and documentation efforts like SBOM reporting or vulnerability scanning fail to prevent the most dangerous attack vectors, such as tampering, impersonation, and build compromise.

Patrick Enderby

Enderby said the gap can be closed with binary analysis, which assesses the compiled binaries of the software, not just metadata or source code. Binary analysis allows for the detection of modern supply chain threats — including the major compromises noted in the study.

It provides the missing layer of independent validation needed to close the chain between trust and verification.

Patrick Enderby

Enderby added that the attacks profiled in the paper share a common pattern: compromise occurs at the binary level within build systems or dependencies long before deployment. The SolarWinds breach was executed through a malicious build process. XZ Utils embedded a backdoor deep within compressed binary artifacts. “Conventional SCA or SBOM checks can’t detect these because they rely on self-reported component data rather than examining the delivered binaries themselves,” Enderby said.

Performing deep binary analysis exposes tampering, embedded malware, and misconfigurations invisible to traditional tools. This capability turns software assurance from a passive, post-factum activity into a proactive control that prevents supply chain compromise before deployment.

Patrick Enderby

The Vertex Project’s Stark pointed to another lesson from the study: While frameworks can be a starting point for cybersecurity, they can’t replace mitigations that are tailored to an organization’s environment and business use case. “Attacker techniques currently have a far more granular classification system than the mitigations specified by the frameworks, which implies that the mitigations need to be more detailed and less abstract,” he said.

Checkbox compliance doesn’t equal security

Sphere Technology’s Mastrogiacomo said the findings highlight the importance of going beyond checkbox compliance.

Organizations must scrutinize the real-world effectiveness of each control and continuously validate whether those controls are reducing actual risk. Visibility into account behavior, ownership, and privilege — across both human and machine identities — is essential.

Rosario Mastrogiacomo

RL’s Enderby said another key takeaway from the report is the need for stronger traceability and accountability throughout the software acquisition lifecycle. Even when security tasks are performed, organizations often can’t prove what was verified, by whom, or when, leaving them exposed during audits and post-incident investigations, he said.

A combination of automation, transparency, and immutable evidence can transform software assurance into a measurable, repeatable control, enabling enterprises to demonstrate due diligence and compliance with mandates like EO 14028, DORA, and NIS2 while materially reducing their exposure to the next SolarWinds- or XZ-style attack.

Patrick Enderby