AI Penetration Testing: Can Autonomous Agents Replace Human Pentesters?

AI penetration testing is a security test where an AI agent attacks your systems the way a real hacker would. It doesn’t stop at a list of weaknesses. It exploits a weakness, uses what it finds to reach the next one, and keeps going until it runs out of room. That’s the difference between a scanner that flags an open door and an attacker who walks through it, finds your keys on the counter, and opens the safe.

This is now a real product category with a market forming around it. On episode 97, Alexis Lingad, founder and CEO of KinoSec, walked through building an autonomous offensive platform he calls “Palantir for offensive cyber operations,” and explained who he sells it to and why. This post covers that side: what the agents are good at, where the humans still earn their fee, and how a startup should buy a penetration test in 2026. The threat side, the same kind of agent pointed at you with no scope and no kill switch, has its own post: AI Cyberattacks Are Going Autonomous.

What is AI penetration testing?

The term gets used two ways. The first is using AI to run the test: an agent plays the attacker against your apps, cloud, network, and devices. The second is testing an AI system, such as checking a chatbot for prompt injection; that is AI red teaming, with its own attack surface and cadence. This post is about the first: AI doing the hacking.

The underlying thing hasn’t changed. The UK’s National Cyber Security Centre defines penetration testing as “a method for gaining assurance in the security of an IT system by attempting to breach some or all of that system’s security, using the same tools and techniques as an adversary might,” and compares it to a financial audit: a check that the way you find and fix weaknesses actually works. A test that proves your process is sound is worth more to a buyer than a long list of bugs.

For years the automated version of a pentest meant a vulnerability scanner: check your systems against a list of known bugs, hand you a report. An AI agent goes further. It proves the problem is real by exploiting it, then asks the next question a human attacker would ask: now that I’m in, what else can I reach? The upside is reach; the risk is an agent that goes further than you expected, which is why so much of this post is about scope.

Approach	What it does	Chains exploits?	Human effort
Vulnerability scanner	Flags known weaknesses against a signature list	No	Low (review the list)
Scripted “automated pentest”	Runs fixed attack playbooks	Rarely	Medium
AI / autonomous agent	Reasons about what it just saw, exploits, chains	Yes	Low to medium (scope, approvals, validation)
Human pentester	Creative, context-aware, full manual attack	Yes	High

How do autonomous pentest agents work?

An autonomous pentest agent runs a loop that copies how a skilled attacker thinks: reconnaissance, then exploit a weakness, then chain that exploit, using the access it just gained to set up the next attack. The key word is chaining. One small bug rarely matters on its own; the damage comes from linking several together, a point NIST’s testing guide, SP 800-115, made in 2008 when it noted that several low-risk vulnerabilities can present a higher risk when combined.

Lingad gave a concrete example. One agent broke into a client’s web app and pulled out the API keys. One key belonged to the client’s email service. The agent used it to reach the admin inbox, then sent internal phishing to the whole company, one attack surface feeding the next, with no human typing the commands. KinoSec’s pitch leans on this: it doesn’t just say a door is unlocked, it shows the full path an intruder would take, live, with a proof-of-concept report and fixes. What that chain means for defenders is the companion post’s subject; here it is a description of the product.

The public proof arrived in mid-2025, when an autonomous system called XBOW reached the top of HackerOne’s US leaderboard. Its own write-up has the numbers most coverage skipped: nearly 1,060 reports submitted, of which 130 were resolved and 303 triaged at the time of writing, while 208 were duplicates and 209 informative. And one sentence worth reading twice: “All findings were fully automated, though our security team reviewed them pre-submission.” Machine breadth, human sign-off, even at the company built to prove the machine could go it alone.

Open source caught up quickly. Shannon, from Keygraph, reads your source code and then executes real exploits against the running app, and its README carries the sentence every buyer should internalize: “Run it only against applications and environments you own or have explicit written authorization to test. Do not run Shannon against production systems.”

Who is buying AI pentesting? Why KinoSec sold to pentest firms first

Anyone building a product for enterprises should listen to how Lingad picked his first customer. He didn’t start with the enterprises. He started with the people who already do the work.

“We started with the pen testing companies, because these pen testing or security testing or hacking companies already know what we’re building, already know the tools. So technically they are the easiest ICP for us: in order for them to have more revenue, more profit, more customers without hiring more pen testers, they can just have our tool in order to do it with just two, three, five people and have a thousand customers. And after the pen testing company, or the security team within the pen testing company, we’re also providing this to enterprises that have a security team inside.”

Alexis Lingad, founder and CEO of KinoSec, on episode 97

That is a market-shaped answer. The pentest industry runs on billable hours from a scarce pool of skilled testers, and a tool that lets three people serve the accounts of thirty changes a firm’s economics before it changes anyone’s security. Then the tool moves in-house at companies with their own security teams. Lingad said KinoSec was in talks with Puma and that Puma’s CISO had just joined as an advisor to open doors to other CISOs in Europe.

The sequence also tells you where the tools reach you first as a buyer: inside the firm you already hire. HackerOne’s ninth Hacker-Powered Security Report, published in October 2025, found that 70% of the security researchers it surveyed now use AI tools in their workflow. So “AI or human” is mostly a decision already made for you: your human tester is bringing an agent. The live questions are how they use it and what you get told about it.

What AI agents do well in a pentest, and where human pentesters still earn their fee

What the agent does better. Breadth: it enumerates every host, endpoint, parameter and leaked key without getting bored. Speed: in XBOW’s own benchmark (a vendor’s numbers, so treat them as directional), the most experienced of five professional testers solved 85% of 104 realistic web challenges in 40 hours; XBOW reached the same 85% in 28 minutes, and the other four humans scored 59% or less. Chaining: the mail-key-to-phishing path above is tedious for a person and trivial for a loop that never loses the thread. Persistence: it runs at 3 a.m., and the week after your big release, which is when the new bug shipped.

What the human still owns. Four things, none of them going away soon.

Scoping. Someone decides what is in bounds, which environment, which techniques are off limits, and what an acceptable outage looks like. Lingad’s answer to Sasha’s question about safeguards began here: a defined scope for every attack, an agent that checks whether it is straying beyond it, and a kill switch behind that. The buyer’s version is three questions: who wrote the scope, who can hit the switch, and has anyone tested that it works.

Judgment on the risky step. Exploit that? Touch that data? Send that email? An agent with a working mail key will send the phish; a human decides whether the engagement should. It is the same problem as giving any agent write access: the permission is only safe when a person owns the consequence.

Novel business logic. “Should a user on the free plan ever be able to export another tenant’s invoices?” There is no CVE for that, no signature, no payload list. XBOW’s own reading of the split is fair: agents excel at common vulnerability classes and easy-to-medium challenges, while humans still lead on complex business-logic attack paths and unusual architectures.

Meaning. An auditor, a buyer’s security team and your board each need a different page from the same engagement, and someone has to validate and rank the raw findings first. XBOW’s 208 duplicates and 209 informative reports are what unvalidated volume looks like; you pay a pentest firm so that you don’t triage it yourself.

The pentest lifecycle: scope, recon, exploit, report, retest

The lifecycle is old and well documented. NIST SP 800-115 breaks a penetration test into planning, discovery, attack and reporting, with attack looping back to discovery as each foothold reveals new targets. The Penetration Testing Execution Standard stretches that into seven phases, from pre-engagement interactions to reporting, and the OWASP Web Security Testing Guide supplies the test-by-test detail for web apps and APIs. None of those documents changed when agents arrived. What changed is who leads each step.

1. Scope (human). Which systems, which environment, which techniques are off limits, who to call. Write down whether the agent may send email, create accounts, or touch customer data, because it will be able to. Insist on a named human who owns the engagement and can stop it.
2. Recon (agent). Mapping every host, API, credential and device is the tireless work agents are built for. The human’s job is aim: point it at the surfaces your buyers ask about and the ones you’d rather not think about, including the devices nobody put on the org chart.
3. Exploit (both). The agent runs the known classes and chains footholds; the human approves the steps that could hurt and hunts the logic flaws that have no signature. Lingad’s own view is that frontier models handle information gathering and exploitation reasonably well and struggle with deeper chaining; the vendors who claim otherwise have built a lot of orchestration, or are describing a scanner.
4. Report (human, agent drafts). The agent produces reproduction steps and evidence quickly. The human removes false positives, rates severity in business terms, and writes the page an executive will read.
5. Retest (agent re-runs, human signs). After your fixes, the agent re-runs every finding in minutes. A human verifies and signs the retest report, the version procurement wants.

Then the loop closes. The NCSC is candid that a test can only confirm the absence of known issues on the day it ran, and shipping changes the day after. Agents make the honest cadence affordable: a human-led engagement each year, agent-driven re-testing after significant releases, and a verified retest closing every cycle.

What should a pentest report include for SOC 2 auditors and enterprise buyers?

The report is the product. Lingad learned that early: “When I was in college, I hacked my college and gave them the report of how I hacked in so that I can help them. But they didn’t like it. So I got kicked out.” Asked what KinoSec actually delivers, he described a report, a proof of concept, and a window into the attack:

“We give the report to them, the POC. And they can also see in our platform how the hack works, so they can watch it live hacking. That’s one of the most entertaining parts for our clients, by the way. All of the things that you’re seeing in the movie, where it escalated, it added a user, it deleted some kind of things, or it escalated a privilege: all of those things, they can see it. It’s transparent. And in the report, they can see also some fixes in there that they can put.”

Alexis Lingad, KinoSec, on episode 97

Two audiences read it, and they want different things.

The SOC 2 auditor. SOC 2 does not name a penetration test as a required control. The Trust Services Criteria ask you to evaluate whether your controls work, and penetration testing is one of the evaluation methods they point to, which is why nearly every auditor asks for a recent test and every enterprise buyer assumes you have one. The auditor checks that scope and dates line up with your system description, that the tester was independent, and that every material finding has a documented response: fixed and retested, mitigated, or formally accepted. Where the pentest sits in the larger arc is in our plain-English SOC 2 guide.

The buyer’s security team. They read the findings. Give them an executive summary a non-engineer can follow; the scope, dates and methodology; each finding with severity, business impact, reproduction steps and evidence of exploitation; remediation guidance; and the retest record with fix dates. That verified retest version is the document that clears security review, and it is how we build our own reports: one document for the auditor’s checklist and the buyer’s engineer, followed by a walkthrough and a retest.

An AI-assisted engagement adds three lines to that list. Which findings a human validated, and which are raw agent output. A log of what the agent was permitted to do and what it did, in the spirit of Lingad’s “it’s transparent.” And what the vendor keeps afterward: Lingad said every strategy his agents use gets fed back into KinoSec’s own model so it becomes “much more intelligent and continuously learning.” Good for the model, and a fair question for procurement about what, exactly, from your engagement ends up in someone’s training set.

How should a startup buy a penetration test in 2026?

Start from the deal, not the tool. The buyer’s security questionnaire tells you which surfaces need coverage; your roadmap tells you when the next big release lands; your auditor tells you the date the report needs to carry. With those three in hand, the vendor conversation gets short, and the checklist below is most of it.

Human-led with agents, or agent-led with humans? Both are legitimate products at different prices. For the report you hand to an enterprise buyer, insist on the first: a named tester who scoped the work, supervised the agent, validated every finding and will get on a call with the buyer’s engineer. For continuous re-testing between engagements, the second is fine, as long as its output is treated as leads to validate rather than findings to publish.

Fixes, not tickets. A pentest that ends with a PDF has done half the job. Ask how findings become merged fixes, because that is where remediation usually stalls. Our own answer is an AI-accelerated remediation program that turns findings into reviewed pull requests, plus a managed bug bounty between engagements. If your engineers ship with coding agents, budget for extra findings; AI-generated code leaves more of them behind.

Timeline. A serious engagement takes about a week to scope, two to four weeks of active testing and a week to report, plus a retest after your fixes: four to six weeks end to end, and a week when a deal genuinely demands it. The right test costs more than a scan and far less than the deal it clears.

Ask the bonus question. Sasha’s argument for buying any pentest is the one to keep in mind while shopping for one:

“This is a cat and mouse game. The attacker is always very creative and tends to move a lot faster than the defense against that attack. So, in other words, you would much rather be friends with the attacking force that has all of the knowledge and expertise in order to avoid your controls that are meant to protect, and still find the gaps.”

Sasha Sinkevich, YSecurity co-founder and co-host, on episode 97

The corollary is that a vendor selling a realistic attacker should be able to explain how they make sure the buyer is who they say they are. Lingad’s answer was that KinoSec hacks its own users first, “so that we can really make sure that the users are not the high profile threat themselves.” Yours can be more conventional: written authorization, scope sign-off, a customer-verification step. This was the year agents ended up on both sides of the breach, and anyone selling one should have an answer.

The decision: hire the machine, keep the human

The pattern is simple: the more autonomy a tool has, the more it must prove it can be reined in. Agents have won the arguments about breadth, speed and re-testing; want them in your next engagement. Humans still own the scope, the risky decision, the novel logic bug and the page that a board, an auditor and a buyer all accept; insist on a named one. Buy with those two facts in mind and you get a better test than you could have afforded three years ago, on a cadence that matches how fast you ship.

If a buyer or an auditor has asked for your pentest, tell us the deal, the stack and the date and we’ll scope an engagement that satisfies both readers, agents included and a human’s name on the report. If you’d rather hear the offensive side describe its own product first, Alexis Lingad does exactly that, hotel doors and all, on episode 97.