Can AI Agents Exploit One-Day Vulnerabilities? What the UIUC Study Actually Found

The research question

The UIUC preprint LLM Agents can Autonomously Exploit One-day Vulnerabilities ↗ asked a narrow but important question: can a tool-enabled language-model agent exploit real vulnerabilities that have already been disclosed but remain unpatched in a deployed system?

This is the one-day setting. The vulnerability is no longer secret. A CVE description or equivalent advisory exists, but a system may still be running an affected version. The paper did not test whether a chatbot could answer trivia about cybersecurity. It evaluated an agent that could browse, use a terminal, search, create files, run code, and react to tool output inside controlled environments.

What the researchers tested

The authors assembled 15 reproducible vulnerabilities from open-source software and prior academic work. The benchmark included websites, container software, and Python packages. Eight of the 15 were described as high or critical severity, and 11 were published after the tested GPT-4 model's stated knowledge cutoff.

Every experiment ran in a sandbox so real users and external systems would not be harmed. Success was manually checked. The primary pass@5 metric asked whether at least one of five attempts succeeded; pass@1 represented the average single-attempt success rate.

That distinction matters. A system that succeeds in one of five attempts is useful evidence of capability, but it is not the same as an 87% chance that every single run, target, or real-world attack will succeed.

The headline result, stated precisely

With the CVE description available, the GPT-4 agent reached 86.7% pass@5 and 40% pass@1 on the 15-item benchmark. GPT-3.5, the eight open-source models tested, and the tested ZAP and Metasploit baselines did not successfully exploit a benchmark item under the reported setup.

The result is often shortened to “AI exploited 87% of vulnerabilities.” The accurate version is more limited: one specific GPT-4-based agent succeeded on 13 of 15 selected, reproducible benchmark vulnerabilities within five attempts when it received the vulnerability description and broad tools.

This does not mean ZAP or Metasploit are useless. The paper itself notes that several benchmark items were not amenable to those scanners and that conventional scanners are not autonomous exploitation agents. The comparison shows that the tested systems answered different questions.

The most important number may be 7%, not 87%

When the researchers removed the CVE description, the GPT-4 agent's exploitation success fell from 87% pass@5 to 7%. The agent identified the correct vulnerability class more often than it successfully completed the exploit, but discovery and long-range exploration remained difficult.

This gap separates two capabilities that are often mixed together in public discussion:

• Finding an unknown weakness in a complex system
• Acting on a weakness after a useful description has been published

The study found the second task was far easier for the tested agent. For defenders, the lesson is concrete: public disclosure can change the operational risk even when a vulnerability still requires several technical steps to exploit.

Why tool access changed the experiment

The agent was not only a language model. It could navigate interfaces, use a terminal, search, create and edit files, and run code. Some successful paths required dozens of actions. One WordPress case averaged 48.6 actions per run, and a successful trace required extensive navigation.

Tool access turns a generated answer into a sequence of external effects. That is why an AI security review should ask which identity the agent uses, which tools it can call, whether actions are read-only or write-capable, which environments are trusted, and where explicit approval is required.

Use the free AI Agent Security Checklist to map those boundaries without connecting CheckLink to the agent.

The cost estimate needs context

The paper estimated an average cost of $3.52 per run and $8.80 per successful exploit, based on the API pricing, token use, and 40% single-run success rate at the time. The researchers compared that estimate with a simplified human-labor assumption.

Those figures are not a current market price for hacking. Model pricing changes, real operations include discovery and infrastructure costs, failed attempts can trigger detection, and production environments differ sharply from a controlled benchmark. The durable conclusion is that automated attempts can be inexpensive and parallelizable, not that every real target can be compromised for a fixed dollar amount.

What this means for ordinary users

Most individuals do not manage CVE programs, but they still depend on software that does. The practical response is familiar:

• Install security updates from the official publisher
• Avoid delaying browser, operating-system, router, and widely exposed application patches
• Verify update links through the official application or vendor site
• Treat messages that use a CVE headline to pressure you into downloading an unknown “fix” as suspicious

The CVE Exposure & Response Planner is designed for teams, but its core idea applies to everyone: confirm the affected product and version before reacting, then use the official update path.

What this means for developers and security teams

The study supports a shorter disclosure-to-decision workflow. That does not mean patching every CVE blindly. It means knowing how to answer the important questions quickly:

1. Is the vulnerable product or component deployed?
2. Is the deployed version and configuration affected?
3. Is the asset internet-facing or reachable from a high-risk path?
4. Is exploitation reported by the vendor or CISA KEV catalog ↗?
5. Is a patch or vendor workaround available?
6. What business process, data, or access is exposed?
7. Who can approve, deploy, and verify the remediation?

Severity remains useful, but it should not replace affected-version evidence, exploitation status, exposure, and business context.

What the paper does not prove

The work is an arXiv preprint, not a guarantee about every model or system. The benchmark was small, deliberately reproducible, and focused on open-source software. Closed-source deployments, custom controls, network segmentation, modern model behavior, defensive monitoring, and different tool permissions can all change the result.

The researchers also withheld their prompt for ethical reasons and disclosed findings to OpenAI. Reproduction is therefore intentionally incomplete. CheckLink does not recreate the prompt, payloads, or offensive workflow.

A defensive workflow built from the finding

The right product lesson is not “launch an agent at every system.” It is to make patch response structured before the next disclosure:

• Maintain an asset and dependency inventory
• Subscribe to official vendor advisories
• Record affected-version evidence
• Use active exploitation as a strong prioritization input
• Assign remediation and verification owners
• Test patches and rollback paths
• Restrict exposure while a permanent fix is prepared
• Keep a decision record for unresolved risk

For higher-impact cases, the Rapid CVE Readiness Review offers a non-intrusive manual evidence and response review. It does not execute exploits or probe a target.

Bottom line

The study's strongest message is about the value of information. A capable agent with the vulnerability description and broad tools performed far better than the same basic approach without the description. That makes the period after disclosure operationally important.

Defenders do not need to mirror the offensive agent. They need faster asset knowledge, trusted advisories, clear authority, bounded tools, and a tested path from evidence to remediation. Explore both source papers and the newer defensive context in the AI Cybersecurity Research Hub.