Application Security Cannot Remain a Periodic Exercise

On 7 August, OpenAI said that preliminary evaluations of Astra, an upcoming model, were strong enough that it could no longer rule out what its Preparedness Framework calls Critical cybersecurity capability.

That threshold has a specific meaning. It includes the ability to find and develop working zero-day exploits across many hardened real-world systems without human intervention, or to carry out a novel end-to-end attack against a hardened target from a high-level objective.

Astra is not an application-security product that an IT director can buy today. The result is preliminary, the model is unreleased, and the announcement is not evidence that every enterprise application has suddenly become easy to compromise. It is still a serious planning signal. Sophisticated vulnerability research is becoming faster and easier to scale.

The uncomfortable question is whether an enterprise can continue to assess its most important applications mainly through annual penetration tests, periodic audits and scans attached to release gates. Those controls remain valuable. Their cadence no longer looks sufficient as the main security feedback loop.

My answer is to make application-security verification continuous across the systems that carry real business consequence. Capable agents can do much of the persistent investigative work: maintain threat context, inspect changes, form attack hypotheses, validate findings, propose repairs and rerun the relevant tests. The surrounding system must control where they can operate and who can accept the result.

That last condition matters. Giving the strongest available cyber model broad production access and asking it to secure the estate would add a highly capable attack path to the estate. The useful operating model is continuous investigation inside narrow technical and organizational boundaries.

Astra and Codex Security answer different questions

Two OpenAI names are easy to mix up here.

Astra is the upcoming model whose preliminary cyber evaluation triggered the Critical capability response. Codex Security, previously called Aardvark, is OpenAI’s application- security agent in research preview. Codex Security shows what a defensive workflow around a capable model can look like.

It connects to a source repository, builds an editable threat model, searches for plausible attack paths, tries to reproduce findings in an isolated environment, proposes a minimal patch for human review and revalidates the issue after remediation. Reviewer feedback can change the threat model and improve later scans.

Other labs are moving in the same direction. Anthropic has introduced a security agent that reasons across a codebase, validates findings and proposes patches for human approval. Google’s Big Sleep work has found previously unknown vulnerabilities in production software, including a vulnerability that Google said threat actors knew about and might soon exploit.

These are vendor and research reports, and several of the products remain previews. They do not establish complete coverage. They do show that context-aware, tool-using agents can perform useful security research that used to require scarce specialist time.

I would not build the programme around any one of these vendors. Vulnerability discovery is becoming a repeatable machine activity. Enterprise defence has to decide how to turn that capability into dependable security work.

Periodic assurance sees a moving system at rest

A penetration test can provide strong independent evidence at a point in time. A release gate can stop a known bad dependency or a policy violation. A code scanner can find a large class of repeatable mistakes quickly. I would retain all of them.

The gap appears between those events.

Business applications change through code commits, dependency updates, identity-policy changes, cloud configuration, integrations, vendor releases and operational workarounds. A test performed against last quarter’s architecture may be professionally executed and still say little about a trust boundary introduced yesterday.

The practical opportunity is straightforward. When the cost of creating and exercising a useful test falls, more of the application can be checked more often.

More tests do not automatically create more assurance. A generated test may repeat the implementation’s assumptions, exercise the wrong environment or prove only that a mocked path behaves as expected. Security work also needs an attacker model, explicit scope and evidence that the suspected path is possible in the system that matters.

This is where agentic security becomes useful. The agent can keep enough context to ask a harder question than whether one line matches a known rule. It can trace attacker-controlled input through several services, compare a change with the application’s threat model, use tools to test the hypothesis and explain why the result matters in this system.

The economic case for continuous verification rests on that combination: cheaper investigation and better context, connected to a remediation process that can absorb the findings.

A business-critical application is larger than its repository

Repository access is a good place to start. It is not the boundary of enterprise application security.

An important business service may depend on custom code, a SaaS platform, cloud resources, identity configuration, APIs, data pipelines and human operating procedures. The source repository may reveal only one part of the path. For a purchased application, the enterprise may not have source access at all.

The same is true for connected products. A customer-facing outcome can cross a mobile application, cloud APIs, device identity, firmware and support tooling. Each component may pass its own local checks while the complete path contains an authorization error, stale assumption or unsafe recovery route.

Continuous coverage therefore has to be defined at the level of a business service. The evidence will differ by surface:

• owned software can be reviewed from design and source through build and deployment;
• SaaS and commercial applications need configuration, identity, integration, data-flow and supplier-assurance evidence;
• cloud environments need infrastructure, policy, exposure and credential checks;
• public interfaces need controlled dynamic testing against deployed behavior;
• critical operating workflows need recovery tests and observation of the real outcome.

No single agent sees all of that by default. The IT function has to maintain the inventory, ownership and permission model that lets several kinds of evidence describe one service.

This also changes what “continuous” means. It does not mean attacking every production system every minute. It means that the portfolio has defined triggers and frequencies based on consequence, change and current threats. A critical identity change may trigger a check immediately. A deeper adversarial exercise may run in an isolated replica on a schedule. A supplier release may call for configuration and integration tests rather than source review.

Build a security evidence loop

I would organize the capability around a security evidence loop for each critical business service. The loop has six parts.

1. Scope by consequence

Name the service, its owner and the business process it supports. Record the sensitive data, privileged actions, critical dependencies, acceptable outage and recovery expectations. Classify the service before choosing the testing technology.

This prevents a common portfolio error: applying the deepest tooling to the easiest repositories while high-consequence SaaS configuration, identities and integrations receive less attention because they are harder to scan.

2. Keep the threat context current

Each service needs a living account of its entry points, trust boundaries, important assets, attacker objectives and control assumptions. Architecture records, interface contracts, identity roles, data classifications and relevant requirements should be available to the security workflow in a controlled form.

An agent can draft and update this context. The application owner and security function need to correct it. A model cannot infer business consequence from code alone, and an outdated threat model will focus capable analysis on the wrong system.

OWASP ASVS can supply concrete verification requirements for web applications and services. NIST’s Secure Software Development Framework provides the wider development and response practices. Neither should be reduced to a prompt. They are stable control references against which agent-generated hypotheses and tests can be checked.

3. Trigger investigation from change and threat

The loop should run when the attack surface changes instead of waiting for a calendar event. Useful triggers include changes to code, dependencies, infrastructure, permissions, exposed interfaces and security policy. New threat intelligence, a production incident or a finding in a shared component may justify a portfolio-wide investigation.

Scheduled deep reviews still have a place, especially for areas that do not produce a clean change event. The important design choice is that a trigger creates bounded investigative work with a named target, current context and a defined evidence requirement.

Traditional scanners should feed this stage. Their deterministic findings are cheap and repeatable. A reasoning agent can correlate them, investigate paths that cross components and test cases that depend on application logic. Replacing reliable rules with a model would discard useful certainty.

4. Validate away from production authority

A plausible vulnerability report is a hypothesis. Before it reaches an application team as urgent work, the workflow should try to reproduce it and retain the evidence.

For a capable cyber agent, the validation environment is itself a critical security control. Use an isolated copy where possible, synthetic or carefully controlled data, explicit target allowlists, restricted tools and credentials, tightly controlled network egress, resource limits, complete action logs and defined stop conditions. Any production testing needs a separate authorization and a method designed not to damage availability or data.

This cannot be enforced by telling the model to stay inside scope. OpenAI recently disclosed two third-party cyber evaluations in which models acted beyond the intended test boundary when internet access and environment controls were configured incorrectly. In one case, a model mistook a real website for part of the simulated exercise and exploited it.

The lesson is direct: the test harness must be able to deny what the instructions prohibit.

A security agent can run continuously. Its authority should remain narrow.

5. Remediate through the normal change system

A validated finding should arrive with the affected service, attack path, reproduction evidence, likely consequence and a proposed minimal repair. It should also have a named owner and an explicit disposition: fix, contain, accept for a stated period or reject with a recorded reason.

Generated patches belong in the same review, test, approval and release process as other changes. High-consequence fixes may need independent technical review, staged deployment and a tested rollback. The discovering agent should not be the authority that approves its own change or declares the business risk closed.

Speed still matters. The loop should shorten the time between a credible finding and a safe containment or repair. It should not obtain that speed by hiding the decision.

6. Revalidate and preserve the lesson

After remediation, rerun the exploit path in the controlled environment. Confirm that the specific failure is gone and that the surrounding behavior still works. Then turn what was learned into durable evidence: a regression test, a policy check, a safer shared component, an updated threat assumption or a new monitoring rule.

Confirmed findings and false positives should improve the next run. That learning belongs in versioned threat context and evaluation cases rather than being left in a model’s opaque memory. If the model, tools or prompts change, the organization should be able to replay the important cases and see whether capability improved or regressed.

That is the improvement loop. Each useful investigation leaves the application easier to verify than it was before.

Use the latest models only after they earn promotion

The strongest models will be attractive for this work. Cyber capability appears to be advancing quickly, and difficult findings may depend on the reasoning depth of the current frontier.

“Always use the latest model” is still the wrong enterprise policy.

A new model can change finding quality, tool behavior, cost, latency, boundary compliance and the kinds of evidence it produces. It may reduce one class of false positive while becoming less dependable in an application-specific task. Model release cadence can also be faster than the organization’s ability to understand a material change in its security control system.

Treat the model as a replaceable component behind an evaluation gate. Maintain a protected set of historical findings, known-safe cases, application-specific abuse paths and boundary tests. Compare the candidate with the current model on discovery, reproducibility, noise, resource use and adherence to scope. Promote it deliberately, pin the version where the platform permits and retain a rollback path.

The framework should survive a change of model provider. System inventory, threat context, authorization, validation evidence, review and risk ownership belong to the enterprise.

Keep independent controls and independent judgment

Agentic testing extends an application-security programme. It does not make the rest of the programme obsolete.

Static analysis, dependency and secret scanning, infrastructure policy, dynamic testing, runtime protection, logging and manual penetration testing each provide different evidence. A skilled security researcher can question the model’s assumptions and recognize business consequence that is absent from its context. Independent testing remains especially important when the same model family helped generate the application code, the security tests and the proposed repair.

The division of work should follow the kind of certainty available. Use deterministic controls for rules that can be stated and checked reliably. Use agents for contextual investigation, cross-system reasoning, test construction and evidence assembly. Use people for scope, authorization, difficult trade-offs, risk acceptance and accountability.

None of those layers has to carry the full problem alone.

What the IT director should be able to see

This becomes an IT operating model when the portfolio is legible to leadership.

For every business-critical service, I would want a concise answer to five questions:

1. Who owns the service, its business consequence and its unresolved security risk?
2. Which code, identities, configurations, integrations, suppliers and runtime surfaces are inside its security scope?
3. Which events trigger verification, and when was each important surface last checked?
4. Which findings were reproduced, which remain hypotheses and which were explicitly accepted, with an owner and review date?
5. What can the testing agents access, which independent boundary contains them, and did the latest remediation produce a permanent check?

I would begin with a small number of genuinely important services. Establish the evidence model, safe validation environment, owner workflow and baseline evaluations there. Expand only when the findings are useful, the remediation path can keep up and the testing system has demonstrated that it stays within scope.

Coverage can then grow by business consequence rather than repository convenience.

NIST has described continuous monitoring and continuous improvement as risk-management practices for years. Frontier models do not create that principle. They may finally make a deeper form of continuous application investigation affordable across more of the enterprise portfolio.

If Astra develops as OpenAI’s early evaluation suggests, waiting for the next periodic review is a poor plan. Giving the same capability uncontrolled production access is worse.

For each critical service, the IT director should be able to show what changed, when it was last examined, what the agent proved, who accepted the decision and which check will now stop the same weakness returning. The model can keep looking. The organization still decides what it may touch and what counts as fixed.

Sources and further reading

1. OpenAI, “Responding to the next frontier of critical cyber capabilities”, describes Astra as an upcoming model and explains why preliminary evaluations led OpenAI to respond as though Critical cybersecurity capability may be present.
2. OpenAI, “Codex Security: now in research preview”, and the current Codex Security documentation describe its threat-model, validation, patching, human-review and revalidation workflow.
3. OpenAI, “Third-party cyber evaluations involving OpenAI models”, reports two evaluation incidents involving activity beyond intended test boundaries and the containment lessons OpenAI drew from them.
4. UK AI Security Institute, “Our evaluation of OpenAI’s GPT-5.5 cyber capabilities”, reports rapid improvement on advanced cyber tasks while distinguishing controlled capability evaluations from ordinary public access.
5. UK AI Security Institute, “The Inspect Sandboxing Toolkit: Scalable and secure AI agent evaluations”, explains isolation across tools, hosts and networks for agents executing security work.
6. Anthropic, “Making frontier cybersecurity capabilities available to defenders”, describes a separate context-aware vulnerability discovery, validation and human-approved remediation workflow.
7. Google, “A summer of security: empowering cyber defenders with AI”, reports Big Sleep’s vulnerability research and the SQLite case informed by threat intelligence.
8. NIST, Secure Software Development Framework and SP 800-137, Information Security Continuous Monitoring, provide established foundations for risk-based secure development, ongoing assurance and continuous improvement.
9. OWASP, Application Security Verification Standard 5.0, provides a stable, testable control reference for web applications and services.