Vals AI cyber benchmark: DeepSeek V4.1 Flash 1017 approaches closed frontier at roughly one-seventh the cost
According to a November 7, 2026 report by The Decoder, Vals AI's CyberOffense benchmark found that open-weight DeepSeek V4.1 Flash 1017 autonomously breached a simulated corporate network and completed 72.8% of its objectives at about $3 per run, versus 80.8% at about $22 for closed GPT-6 Sol Daybreak. DeepSeek scored zero during the first 25 million tokens, so short tests underestimate open models.

What changed
According to a November 7, 2026 report by The Decoder's Matthias Bastian, Vals AI used its CyberOffense benchmark to test whether agents could break into a simulated corporate network on their own. Each agent ran on a separate isolated Kali Linux attack machine, started with only a public URL and no human assistance, and had to move from the web application into a Windows server and obtain administrator privileges; its actions were recorded externally and scored automatically. The reported results: GPT-6 Sol, part of OpenAI's restricted Daybreak cyber program, completed 80.8% of the objectives; open-weight DeepSeek V4.1 Flash 1017 completed 72.8% at about $3 per run, versus about $22 for the former. Vals researchers said that once weights are released, no provider can control or limit their use. In the harder Lighthouse scenario, Daybreak completed 27 of 31 objectives in under three hours, while DeepSeek completed 21 at about one-tenth the cost, and was not merely held back by safety guardrails. The report emphasized run length: DeepSeek scored zero for the first 25 million tokens, with most progress coming after more than 100 million tokens. The article also cited a July analysis by the UK's AISI saying the cyber-capability gap between open and closed frontier models had narrowed to four to seven months, and Anthropic research on GLM-5.3's exploit-building ability approaching that of its internal models.
What it means for developers
For developers, the value of this benchmark is not who ranks first but that it lays out three parameters usually left vague: token budget, cost per successful objective, and who is responsible for guardrails. If the task is long-horizon and requires maintaining state across systems and retrying after failure, evaluating an open-weight model in short episodes of tens of thousands of tokens may produce a near-zero score, and that result may be mistaken for a capability gap. In the report, DeepSeek scored zero for the first 25 million tokens and showed most of its progress after 100 million tokens, which suggests the budget itself is an independent variable. On cost, about $3 and about $22 are per-run magnitudes, but what really matters is cost per successful objective: when completion rates differ by a few percentage points, the expense of closing the gap through retries may erode the unit-price advantage (this is an analytical inference from the reported figures, not a conclusion of the report). On security, the report says that once open weights are public, no provider can limit their use, and DeepSeek's score was not merely suppressed by guardrails. That means when such a model is connected to an agent with real network access, sandboxing, egress allowlists and action auditing can only be handled by the integrator; one cannot count on model-side refusals. Conversely, the same capabilities can be used for self-checks: Vals noted that a model capable of exploiting vulnerabilities can also first help an enterprise find and fix them, and said attacks on real networks may increase in the coming months. In addition, the report does not address the license for the weights; open weights are not the same as an open-source license, and ownership and commercial terms need to be checked separately.
Integration and evaluation
The following are conditional recommendations, not facts stated in the report. First, decide the evaluation budget before choosing a model. The change highlighted by the report is that short runs systematically underestimate open-weight models. If your task really is long-horizon and multi-step, raise the token budget per evaluation to near the magnitude mentioned in the report, and record how objective completion rises with token consumption rather than looking only at the final score. The test is whether the model stays nearly flat until close to 100 million tokens and only then rises clearly. If the task is single-turn question answering or two or three tool calls, this does not apply; look at performance under a small budget. Second, choose by cost per successful objective, not by per-run price. The change highlighted by the report is 72.8% versus 80.8%, and about $3 versus about $22 per run. The method is to fix the same task set and number of retries, then divide total spending by the number of successes. The precondition is that your scenario allows retries or parallel attempts; if a failure is irreversible—for example, production has already been changed—choose by completion rate rather than unit price. A measurable test: over 20 or more repetitions, does the cost per successful objective for the two models still differ by an order of magnitude? Token-based prices change; use the pricing page of the platform you use for specific numbers. Third, put guardrails in your own infrastructure. The change highlighted by the report is that once weights are public, no provider can control their use, and DeepSeek was not merely limited by safety guardrails. If this agent has real network or credential permissions, you need egress allowlists, an isolated sandbox, step-by-step action logging, and human confirmation for privilege-escalation operations; if it is only an offline, read-only task, the urgency can be lower. The test: run a set of internal red-team tasks with and without these controls, and see how much the number of completed objectives falls; the controls should be accountable for that difference.
Limits and open questions
Caveats to preserve: the scores and costs above come from Vals AI's benchmark and this report by The Decoder; they are not tests performed by this article, nor have they been confirmed by the model providers. The report says only that the weights are freely available and does not state a license, so one cannot infer the scope of usage rights from open weights. A caption in the same article says DeepSeek overtook other models in the Lighthouse scenario, but its table lists DeepSeek at 21, GPT-6 Sol Daybreak at 27, and Meta Wonder at 28; the caption and table do not match, and per-model rankings should defer to Vals AI's original benchmark page. The Nova column is an average of three runs; Daybreak's range is 12 to 19, so small score differences fall within run-to-run variation. The figures of about $3 and about $22, as well as AISI's July figures of $1.19 and $85, come from different times and different test setups, are not directly comparable, and do not represent the current token prices of any platform. The AISI and Anthropic material is second-hand citation in this article. The report does not say which API or hosting service DeepSeek V4.1 Flash 1017 ran through, so the per-run cost cannot be mapped to any particular price list; that version name has a corresponding entry in KimiSeek's current catalog, but all figures cited here come from Vals AI and The Decoder, not from KimiSeek measurements; check the pricing page of the platform you use before integration.
Illustrations from the original report



