For practitioners who have applied AI-assisted pentesting, or who are building their own harness to do the same thing, the open question is this: can we just prompt the most advanced model available and let it do the job, or is there something better to think about?
For readers who have not worked on this: a penetration test is a controlled attack. The customer signs an agreement that lists exactly which systems may be tested. The tester then tries to break into those systems the way a real attacker would, and reports what worked. Anything outside that list is off limits, and testing it would be a crime. An autonomous pentest system does the same work without a person driving it. The model decides what to try, the system runs the command, the model reads the result, and the model decides what to try next.
Most commercial models today ship with built-in safety guardrails that stop them producing harmful content. Those same guardrails also limit security testing work. Ask a model to run a step that is necessary and authorised, and it may decline: instead of the command we asked for, the reply explains why the model will not proceed. A common example is asking for a payload — a short piece of input designed to make a target behave incorrectly — for a login page the customer has asked us to test. Instead of the payload, the reply says it cannot help with attacks on login systems. In a chat window that is an annoyance. In an automated system it stops the job.
So should we take an open-source model and strip the guardrails out? Or should we use different models at different stages of the test?
PAIStrike is our system for doing this. What follows is how we choose the models that run inside it.
The short version: a pentest already runs in stages — plan, execute, browse, judge, report. Each stage can use a different model. We group those choices into three profiles, one per type of engagement, and each profile fixes one model per stage.
For each stage we ask three questions, in this order:
Volume is worth spelling out. One job is not one question to a model. Every tool the system runs produces output that the model has to read before choosing the next step, so a single job can involve several hundred model calls. The planning stage might run five times in that job. The execution stage runs every time.
The order of the three questions matters. A cheap model that stops halfway through an authorised test is no use to us however cheap it is, so price is never the first thing we look at. An excellent model can still be unaffordable at that volume, so capability is not the last word either.
We keep the three as separate numbers and never blend them into one score. A blended score hides the trade-off we are actually making, and the trade-off is the whole point.
Start with what is available. Anthropic's June 2026 release is the clearest illustration. Fable 5 and Mythos 5 are the same underlying model. Mythos 5 has the cybersecurity classifiers switched off — a classifier here is a separate checking layer that reads each request and can refuse it before the model answers. Mythos 5 is not generally available: access runs through Project Glasswing, an invitation-only programme for cyber defenders and infrastructure providers, with no self-serve sign-up. Fable 5 is the version everyone else can call, and it ships with those classifiers active. Anthropic's own guidance for developers says to expect refusals and to plan a fallback to another Claude model when a request is declined.
Availability also moves. Both models were suspended in June 2026 to comply with US export controls, and restored on 1 July.
Capability and permission to use it are sold separately, and neither is guaranteed to still be there next quarter. That is a fact about supply, not an argument about safety, and it applies whichever vendor we pick.
This is also the answer to the open-source question. Removing the guardrails from an open model buys one thing: fewer refusals. It does not keep the system inside the agreed scope. That job belongs to the harness — the code around the model — and it has to be there whichever model runs inside:
None of that comes from the model, and none of it disappears if the model's guardrails do. Meanwhile the open models we could strip are behind the hosted ones on the second question.
The problem we actually care about is narrower and more practical. A model that writes a correct plan and then declines to run one of its own steps fails in a messy way. The system retries, the job finishes half-done, and the same job run twice gives different output. That is a reliability problem, not a policy debate.
So we test each model on our own work before anything else. We give it the same authorisation prompts, the same engagement context and the same tool descriptions it would see in a real job, and we count three things:
We tested 52 models this way. Twelve of them could do the work. Eleven were left after we removed older versions that had been replaced.
Two things follow. Most models on the market cannot do this job at all, so the plan of picking whichever model is strongest this month and writing a good prompt does not work. And passing this test is not a score. It only means the model can be used somewhere in the system. It does not say where.
None of the eleven is best at everything, so we pick a model per task. Each task needs something different.
Planning produces the test sequence: scan the host first, then look at the web application, then try the login page. The model has to produce a sequence that still works when step three fails and returns nothing.
Execution picks the right tool for the current step and reads what comes back. A port scanner and a database injection tool answer different questions, and choosing the wrong one wastes a step. Execution also has to return its answer in the same shape every time. If the tenth call describes an open port differently from the first nine, the code that reads the answer breaks.
Browser work means logging into a web application and clicking through it — filling a form, following a redirect, checking whether the page now shows another user's data. We test this separately, because a model that handles text well may still misread a rendered page.
Judging separates what the system observed from what it assumed. "The server returned a database error" is an observation. "The database can be read by an attacker" is a conclusion, and it needs the reproduction steps to back it up. Reports lose their value quickly when the two get mixed.
Utility work is summarising a long scan output, pulling up something from earlier in the job, and formatting text. It needs very little from the model.
Guardrails hit these tasks unevenly, and this is the part that makes the split worth doing. What matters is not the topic but how much the request looks like an attack in progress:
So we apply the behaviour test per task rather than once for the whole system. A model that refuses too much during execution can still do the judging and the utility work. That is why eleven models are enough to fill every stage.
The cost of splitting is that different models format their answers differently, so every handoff between stages is a place where things break. If the executor reports port 8080 open one time and Port 8080: OPEN the next, the judging stage has to cope with both. We fix the format between stages so that each model always receives the same shape of input, whichever model produced it.
Models are billed by the amount of text going in and coming out, counted in tokens, where one token is roughly three quarters of a word. The bill for a stage is therefore the price per token, multiplied by how much text passes through that stage, multiplied by how often the stage runs. The third factor is the one people forget.
Data from one of our own runs, a 14-day window covering 125 pentesting jobs, shows the split:
EXECUTION 70.4%BROWSER16%PLANNING12.2%everything else, under 1.5%SHARE OF MODEL SPEND · 14 DAYS · 125 JOBSA 10% cheaper model here cuts the whole bill by 7% → six times the effect of the same cut on planningFig 4 The bar is to scale. The ratio, not the amount, is what carries over to a larger deployment.
Finding a 10% cheaper execution model would have cut the whole bill by about 7%. The same 10% cut on the browser model takes about 1.6% off the total, and on the planner about 1.2%.
The amounts in a two-week sample are small. The ratio between them is the part that carries over. Effort spent on the execution stage is worth roughly six times the same effort spent on the planner, and it is still worth six times as much when the system runs a thousand jobs a month instead of 125. The ratio comes from how often each stage runs, which does not change with volume.
So cost work belongs on the stage that repeats. A strong planner is affordable precisely because it runs a handful of times per job while the executor runs on every single tool call. Moving execution and utility work to cheaper models, while keeping strong models for planning and review, is where nearly all of our saving came from.
Customers do not all want the same test. One asks us to push as far as we can along a real attack path, because they want to know what an intruder could actually reach. Another wants a careful test with tight boundaries and a check that the basic protections are switched on, because they have an audit coming. Most want something in between.
A profile is one of those engagement types, together with the set of models that runs it. Answering the three questions for every stage of every job would be unmanageable, so we answer them once per engagement type and store the result. When a job is booked, the operator picks the profile that matches what the customer asked for, and the system loads the whole set in one step — planner, executor, browser, judge, utility and fallback.
The industry names for these three are Red Team, Blue Team and Purple Team, and we use them too. One point of confusion is worth clearing up: our Blue Team profile still attacks the target. It is not security monitoring or incident response. It runs the same offensive test with tighter boundaries, and also checks whether the basic protections are correctly configured.
What differs between profiles is not only the model names. It is how many refusals we are willing to tolerate, how hard the system pushes, what counts as a finding worth writing up, and how much we are prepared to spend on one job. The model choices follow from those decisions, not the other way round:
What each profile optimises forProfileWhat the customer wantsWhat we accept in exchangePush hardest (Red Team)Proof of what an intruder could reachHigher spend per job, looser reporting of small configuration issuesStay conservative (Blue Team)Tight boundaries, evidence for an auditMore refusals, slower progress along the attack pathBalanced (Purple Team)A useful test without tuning anythingNeither the deepest attack path nor the widest configuration check
One term for reading the table further down: by tier we mean the vendor's own product levels: the flagship model, the mid-range one, and the small fast one. Within a single vendor's line-up the flagship often costs ten times the small model or more for the same amount of text.
Red Team — push hardest along the attack path. An attack path is the chain of steps from the first way in to whatever the test is trying to reach: a weak login leads to a user account, which leads to an admin page, which leads to the customer database.
Blue Team — stay conservative and check the basics. One model runs planning, execution and judging. Splitting them would save a little money at the cost of consistent behaviour at the edge of what is allowed, and here that consistency is the point. This profile also checks whether basic protections are correctly configured — password rules, encrypted connections, security headers, admin pages that should not be reachable from the internet — and it reports required controls separately from recommended improvements, so that every small deviation from best practice does not arrive looking like a vulnerability.
Purple Team — balanced default. Most deployments use this one. It does not push as hard as Red Team and does not check as broadly as Blue Team.
The three combinations, with version numbers left offStageRed TeamBlue TeamPurple TeamPlanningDeepSeek ProxAI GrokOpenAI GPT TerraExecutionDeepSeek FlashxAI GrokOpenAI GPT LunaBrowserQwen PlusZ.ai GLMZ.ai GLMJudgingDeepSeek ProxAI GrokxAI GrokUtilityOpenAI GPT LunaOpenAI GPT LunaOpenAI GPT LunaFallbackQwen PlusOpenAI GPT TerraQwen Flash
Version numbers are left off on purpose. They were accurate for one month, and a version string in a blog post goes stale faster than anything else in it. Internally each combination records the date it was chosen, because prices change, refusal behaviour changes, and models get retired. If you want the exact versions we are running now, get in touch.
Three things are worth reading off it. Red Team keeps planning, execution and judging inside one vendor's line-up, paying for the Pro model where judgement matters and dropping to its cheaper Flash sibling for the stage that repeats. Every profile sends utility work to the same small model, because none of them needs more than that. And the browser model is never the same as the planner, because browser work was assessed on its own and the model that won it came from elsewhere.
The fallback row matters more than it looks. If the main vendor has an outage or changes its refusal behaviour overnight, a job that is halfway through still has to finish. So the fallback is always a different vendor from the model it stands in for, close enough in capability to carry on, and no more expensive.
The profiles above have been run against three public benchmarks. XBEN covers web application attacks: 97 of 104 scenarios passed, or 93.27%. CVE-Bench asks the system to reproduce real published vulnerabilities: 37 of 40. CyBench is a set of end-to-end problems: all 31 were completed, though only 22 on the first run, and the rest needed faults in the test environment fixed first. We report both CyBench numbers, because the difference between them is a measure of our setup rather than of the models.
PUBLIC BENCHMARK RESULTSXBEN97 / 104CVE-Bench37 / 40CyBench31 / 3122 SOLVED ON THE FIRST RUNREST AFTER FIXING OUR TEST ENVIRONMENTFig 5 Bars are to scale. The CyBench split is shown because the gap between the two numbers measures our setup, not the models.
All three are public, so the runs can be checked rather than taken on trust. The cases we watch most closely are the ones where an attack only works if several steps happen in the right order — log in, change a value in a request, reload the page, check whether the change took effect — because that is where a single-model setup seems to lose ground.
A benchmark target holds still; a customer's system does not, so we read these as a baseline for what the configuration can do rather than a promise of the same result on a live engagement.
We track cost per stage from the first day rather than working it out later from an invoice. Without that split, the obvious move when the bill arrives is to downgrade everything, which saves little and costs a lot in plan quality.
When a job fails, we check the environment before blaming the model. Most of our early failures were faults in our own setup — a target that had changed since the job started, a tool that returned nothing, a login session that had expired. Recording those as model failures would have sent the next round of selection in the wrong direction.
A complex penetration test is not a one-prompt problem. Every next action depends on what happened before, and no single model is strongest on all of reasoning, cost, refusal behaviour and holding context across a long job. Splitting the work by stage is what let us stop trading those against each other. Two weeks and 125 jobs is still a thin sample, so if you are running something similar we would like to hear what your own cost split looks like. More about the platform is at paistrike.ai.