With the rise of AI pentesting, a lot of people are wondering if it’s as good as a human. Having started as a pentester twelve years ago, and having moved through similar roles to working on Intruder’s AI pentester,
I've come to understand the advantages and disadvantages of both. Here’s how they compare.
The strict methodology was never the good bit
You may expect AI pentesters to follow strict methodologies in the way that many human pentests do.
At Intruder, we experimented with this approach early on - structured systems where each agent had a specific vulnerability type to look for. What we found was that agents would miss things outside their remit entirely, or miscategorize what they did find as the one thing they'd been told to look for, causing confusing reports.
A complete lack of structure also failed to produce good results, as completely free agents without any guidance couldn’t break down and manage large applications properly. They could find interesting things, but at the cost of missing a load of other vulnerabilities in our targets.
We’ve put a lot of testing into finding the balance between these approaches, and settled on giving agents freedom with broadly defined roles to help them approach and manage large applications, and stricter guardrails to stop them causing damage.
This does mean that our AI pentester doesn’t tick off the parts of a methodology document that you’d typically see in a human pentest. That may make some people uncomfortable - a strict methodology gives people assurance that someone has gone through your application and tested everything on the list. An unauthenticated SQL injection needs to be found, and a checklist is there to make sure it does.
The methodology gives a lot of people the idea that a human pentest is a perfectly repeatable product, as any tester will be able to follow the methodology, and everything important will get found. The reality is a lot more nuanced than that though, as pentesting is a skilled job, and many of the most impactful findings on a pentest will come from the tester being experienced in noticing weird behaviours and following up on them.
For our AI pentester, we ended up with much more interesting and impactful results by not trying to guide our agents through specific vulnerability types. A good example is a complex authentication bypass the agents found that had been sitting in a codebase through five years of pentests without any human testers picking it up.