Software products evolve constantly. New features are added, dependencies change, interfaces expand, and deployment environments become more complex. Each change can create new security assumptions – and new opportunities for those assumptions to fail.
That is why vulnerability assessment should not be treated as a one-time activity performed shortly before a product ships. Effective security testing needs to accompany a product throughout its full lifecycle: during design, throughout active development, before major releases, and during long-term maintenance.
This is particularly important for software used in high-impact environments, including industrial, aerospace, and other mission-critical systems. But the principle applies to every software product: an application that was thoroughly assessed last year is not necessarily secure today.
Why Internal Testing Is Not Enough
Development teams should test their own software continuously. Secure code review, automated scanning, fuzzing, dependency monitoring, integration tests, and threat modeling all play an essential role in reducing risk early.
However, internal teams inevitably carry context and assumptions. They understand how a feature was intended to work, which can make it harder to notice how an attacker might misuse it. They may also develop blind spots around legacy components, accepted design decisions, or areas believed to have been sufficiently tested.
External security teams bring a different perspective. They approach a product without the same assumptions, explore unexpected attack paths, chain together smaller weaknesses, and often test boundaries that internal teams may not consider relevant. Multiple independent teams provide even greater value because each researcher has different experience, methods, tooling, and intuition.
The goal is not to prove that a product is “secure.” No realistic assessment can do that. The goal is to repeatedly identify and reduce meaningful risk before an adversary finds and exploits it.
A Lesson From StarPWN CTF
A practical example came from StarPWN, a capture-the-flag competition that debuted at the DEF CON Aerospace Village. At VisionSpace, we designed its challenges around real-world space systems that are actively used in the industry. We intentionally incorporated vulnerabilities that we had previously discovered in those systems, along with a few newly introduced issues. This gave participants several intended routes to compromise the targets.
From an organizational perspective, everything initially appeared to be in good shape. The challenges had been designed, tested, and built around the injected vulnerabilities. But shortly after the CTF started, teams began submitting flags for some of the easier challenges far sooner than expected.
Our initial assumption was that the challenge logic contained a mistake or an unintended shortcut. A quick investigation revealed something much more interesting: rather than following the intended solution paths, they had identified two previously unknown vulnerabilities – two 0-days – in the underlying software. They used those flaws to fully compromise the targets and bypass the planned paths entirely.
The lesson was clear. Even software that has received substantial attention, including prior security testing, can contain vulnerabilities that remain undiscovered. A determined and capable external team may look at the exact same system and find something entirely new.
Security Changes as Software Changes
The affected software had already been tested extensively. Yet it continued to evolve: new functionality was introduced, code was changed, and the attack surface expanded. With those changes came additional vulnerabilities, several of which were later assigned CVEs and disclosed through the project’s security advisory process.
Then, two years after our initial assessment, we returned to assess a newer version of the system. Again, we found a new vulnerability: CVE-2026-55566.
This is not an unusual outcome, nor does it indicate that previous assessments were ineffective. On the contrary, it demonstrates why assessments must be repeated. A vulnerability assessment represents a point-in-time evaluation of a particular version, configuration, threat model, and deployment context. Once any of those elements change, the conclusions need to be revisited.
A report stating that no critical vulnerabilities were found should therefore be read as: “No critical vulnerabilities were found within this scope, at this time, using these methods.” It should never be interpreted as a permanent security guarantee.
When to Test
Security testing should be integrated into both development and maintenance rather than reserved for a single release milestone.
- During design, perform threat modeling and architecture reviews to identify risky trust boundaries, security-sensitive interfaces, and dangerous assumptions before they become difficult to change.
- During development, use secure code review, automated testing, static analysis, dependency scanning, fuzzing, and targeted security tests in the CI/CD pipeline.
- Before major releases, major architectural changes, or exposure to new customers and networks, commission an external vulnerability assessment or penetration test.
- After significant changes, reassess the components and interfaces affected by new features, protocol changes, authentication changes, integrations, or deployment models.
- During maintenance, conduct recurring independent assessments – typically at least annually for most products, and more frequently for internet-facing, safety-critical, high-value, or rapidly changing systems.
- After an incident, a severe vulnerability disclosure, or the discovery of a systemic weakness, expand the review beyond the immediate bug to find related issues and failures in the surrounding security controls.
The right cadence depends on the product’s risk profile. A small, stable internal tool does not need the same assessment schedule as a ground-segment platform, an internet-facing SaaS product, or software operating in a safety- or mission-critical environment. But every product benefits from security reviews that recur as the codebase and threat landscape change.
Independent Eyes Create Real Value
External assessments are most valuable when organizations treat them as a collaborative part of product security rather than an audit hurdle. Teams should provide assessors with realistic deployment information, documentation, architectural details, and sufficient time to investigate in depth. They should also prioritize remediation, validate fixes, and incorporate the findings into future engineering practices.
Using different assessment teams over time can be especially effective. One team may excel at web application testing, another at protocol analysis, embedded systems, cloud infrastructure, binary exploitation, or attack-chain development. Diverse perspectives reduce the likelihood that the same blind spots persist across successive reviews.
The StarPWN experience is a useful reminder that attackers do not follow expected paths. Even when a system has been tested, and a scenario appears carefully controlled, independent researchers can uncover routes that its designers and previous assessors did not anticipate.
Security is an ongoing race between defenders and adversaries. Continuous internal testing builds a strong baseline, while recurring assessments by independent external teams challenge assumptions and reveal what routine processes miss. The organizations that keep testing, retesting, and learning from each result are the ones best positioned to stay ahead.
Resources
Read more about the tales from StarPWN CTF 2025: here.
Join us for StarPWN CTF 2026 at DEF CON 34: here.
If you’re at DEF CON, visit us at the Aerospace Village – let’s talk space security.
