AI models are accelerating the discovery of software flaws, while the window between vulnerability disclosure and exploitation has narrowed from days to minutes. Security teams are turning to AI to help keep pace with the defensive work those attacks create, but there is little evidence to judge how well models can perform different defensive tasks. Most public cyber benchmarks focus on offensive tasks, such as developing exploits or completing capture-the-flag challenges. Few measure whether models can help defenders write detections and investigate attacks.

Today, we’re launching the GreyNoise AI Cyber Index to evaluate those defensive capabilities. At launch, it tests 46 closed and open-weight models across three tasks: Network Detections, File Detections, and Attack Forensics. The Index reports task performance, refusal rates, time, and cost, giving security teams a benchmark for comparing AI models across different security tasks.

GreyNoise has spent eight years observing how attackers find, exploit, and compromise internet-facing systems through our Global Observation Grid. We collect and label the traffic and associated artifacts, such as files and commands executed. That labeled data is used in production by more than 100,000 defenders and provides the ground truth for these benchmarks. Explore the GreyNoise AI Cyber Index >

‍

How the Index works

Every problem asks a model to perform a specific defensive task. The rule or list of evidence a model returns is run against recorded data and checked against our labeled data. The initial benchmarking set has 145 problems across three evaluations.

‍

The three evaluations

The overall composite score is a weighted average of 3 different defensive evals. . Network Detection carries the most weight (55%) because it’s the most tested and validated problem set, File Detection follows at 30%, and Attack Forensics carries the least (15%) because it’s the least complex task and most models already score well on it.

Evaluation SOC task What the model does Problems Index weight
Network Detection Identify and block malicious traffic Writes a Suricata rule from one malicious session that must catch other deliveries of the same exploit without alerting on similar harmless traffic. 51 55%
File Detection Identify the spread of malware across systems Writes a YARA rule that recognizes new builds of a malware family without flagging unrelated files. 63 30%
Attack Forensics Link an outbound traffic alert to the remote exploit responsible for the compromise Traces an outbound traffic alert back to the exploit responsible, using recorded network and host evidence while rejecting planted decoys. 31 15%

‍

How problems are built and scored

Every problem is built the same way. The model gets one example of an attack and has to produce something we can test. We hold back other forms of the same attack, the same malicious payload delivered a different way. These held-out forms are what a good answer should catch. We also hold back a benign twin, a copy of the attack with the malicious content swapped out, and a false positive corpus of unrelated data. A good answer should leave both alone. The corpus includes other malicious activity on purpose, so a rule can’t score by matching anything that looks like an attack. Every problem also has an oracle, a reference answer that earns full marks and proves the problem can be solved.

  • Network Detection. The model gets one malicious session from a real exploit and writes a Suricata rule. The held-out forms are the same exploit but hitting a different path, using different headers, or having a different structure. The grader awards points for each held-out form the rule catches and takes them away for catching the benign twin or any of the sessions in the false positive corpus. A bad rule that matches only the request path will identify the benign twin and will lose points.
  • File Detection. The model gets one malicious file and must write a Yara rule that identifies the malware family. The held-out forms are other builds of that family the model never sees. The grader takes points away for matching the benign twin, which has the malicious family-identifying content removed, or any of the files in the false positive corpus. A rule that fails to compile scores zero.
  • Attack Forensics. The model gets our data lake tables, a copy of the data recorded at the time of the attack, and an alert that a server is sending requests to the internet. It queries the data to identify the egress traffic, the process that generated it, the dropped file or launcher behind that process, the exploit payload and attacker IP, and the CVE being exploited. The recorded data also holds unsuccessful attacks and planted benign versions of the payloads and processes, so naming a decoy as the attacker costs points.

Each model attempted every problem at least three times, and a problem’s score is the average of its attempts. A model’s score is the share of available points it earned across problems, so 100% means full marks on every problem. Time is clock time per attempt, including tool use. Cost is based on each provider’s published prices.

Every view has a Penalizes Refusals setting. With it on, a refused attempt scores zero, so the score reflects both capability and reliability. Our problems contain real malware, as real incidents do, and a model that refuses the job can’t help a security team. A model counts as capable when it scores at least 90% of the top score, which puts the threshold at 64.5% with refusals penalized. The results below use that view.

‍

What the results show

We evaluated 46 models. All figures reflect the Index as of November 11, 2026.

  • No model has a perfect score. Claude Opus 5.5 posted the highest raw score we recorded, 74.2%. It also refused 16.5% of its attempts. Scoring each refusal as zero drops it to 63.9%, just under the 64.5% capable threshold. Grok 4.5 refused nothing and led at 71.7% with refusals penalized.
  • Refusals came only from Anthropic and Google models, and they varied by task. Claude Opus 5.5 answered every Network Detection problem but refused 52.4% of its File Detection attempts, where the input is a malware file. Claude Fable 5.1 refused 90.8% of its Network Detection attempts and 72.2% of its attempts overall, which leaves it at 21.9% with refusals penalized. Gemini 4 Argon refused 16.7%.
  • Detection engineering is hard. The best Suricata score was 75.9% (Claude Opus 4.8) and the best YARA score was 58.4% (Grok 4.5). 
  • Strength on one task doesn’t carry over to another. Beam 1.0 scored 92.5% on Attack Forensics and 8.3% on Network Detection. GPT-5.4 scored 70.3% on Network Detection and 4.6% on Attack Forensics, indicating that model capability varies widely by task.
  • Among capable models, cost varies far more than score. Seven models cleared the capable threshold, led by Grok 4.5 at 71.7% and followed by Grok 4.6 (70.0%), Claude Opus 4.8 (69.0%), Claude Haiku 5.5 (67.9%), Claude Sonnet 4.6 (67.7%), GPT-6.1 Sol (64.9%), and MiniMax M3 (64.7%). Claude Haiku 5.5 was the cheapest and fastest of them, at $0.020 per problem and 44 seconds per attempt. 
  • Open-weight models trail frontier models. MiniMax M3 is the only open-weight model above the capable threshold, at 64.7%.

‍

What comes next

These results cover Suricata rule writing, YARA rule writing, and Attack Forensics investigations, we are far from covering all security work. The launch set is a starting point, and we plan to keep dated snapshots so readers can see how rankings change over time. We are building new problems for the three existing evaluations, each held to the same QA gates, and developing new environments that cover more defensive tasks.

Explore the GreyNoise AI Cyber Index at benchmarks.greynoise.io.

This article is a summary of the full, in-depth version on the GreyNoise Labs blog.
Read the full report
GreyNoise Labs logo
Link to GreyNoise Twitter account
Link to GreyNoise Twitter account