Podcast Episode 16

Episode 016 56 min

Will Butler, Red Team at Robinhood and Co-Founder of TruffleSecurity, From Hacking Airports to Leading Red Teams

with Will Butler · Robinhood & Truffle Security

0:00 / 56:00

Also on Spotify, Apple Podcasts, YouTube, Podbean, Player FM

Episode 16 of The Security Podcast of Silicon Valley, published May 1, 2022. Host Jon McLachlan talks with Will Butler, then the red team tech lead at Robinhood and a co-founder of Truffle Security, the company behind the open source secret scanner TruffleHog. They cover an airport security system given away by a small session ID, economics, finance and philosophy as lenses on security, attackers who work like dev teams, findings that break workflows, a favorite bug that went from a phishing domain to engineers' laptops, SSH CA and the exceptions that keep an attack chain alive, outages a red team can cause, internal and external red teams, red teams as security product managers, reports for buyers, an interview question with no floor, insight under constraint and the escape fire, an insecure binary you can't fix, and why an engagement ends at the attacker's goal.

Episode notes

In this thrilling episode of a YSecurity.io production, we sit down with Will Butler, a seasoned security expert whose journey into the world of red teaming began with an audacious hack of airport security systems in his youth. This pivotal moment not only sparked Will's passion for security but also set him on a path to becoming a formidable figure in the field of penetration testing and red teaming.

Will shares his fascinating transition from a curious youngster with a knack for finding security loopholes to a professional dedicated to safeguarding organizations against cyber threats. He delves into the various facets of red teaming, explaining the different types of red teams and the unique value they bring to businesses through penetration testing. Listeners will gain an insider's perspective on what constitutes a successful attack and the creative, unconventional thinking required to unearth vulnerabilities.

The conversation also touches on the critical importance of diversity within red teams. Will argues that a broad range of perspectives and experiences is crucial for success in this field, enhancing the team's ability to think outside the box and anticipate the myriad ways an attacker might breach security defenses.

Moreover, Will addresses the ethical and moral dilemmas that penetration testers often face. He discusses the fine line between hacking for security enhancement and the potential for misuse of skills, emphasizing the importance of ethical guidelines and professional integrity in the practice of red teaming.

This episode is a treasure trove for anyone intrigued by the art and science of cybersecurity, offering deep insights into the challenges, strategies, and ethical considerations of red teaming and penetration testing. Whether you're an aspiring security professional, a seasoned expert, or simply fascinated by the world of cybersecurity, join us for an engaging and enlightening conversation with Will Butler.

Tune into this YSecurity.io production to explore the captivating journey of a red team leader who transformed a youthful curiosity into a career dedicated to advancing security practices and protecting organizations from cyber threats.

Hosted by Jon McLachlan — co-founder of YSecurity.

The deep dive

A tether to reality

Will Butler, red team tech lead at Robinhood and a co-founder of Truffle Security: an airport session cookie, attackers who work like dev teams, findings that break workflows, a favorite bug that began with a phishing domain, SSH CA and ground truth, outages, internal and external red teams, red teams as product managers, an interview question with no floor, and why the engagement ends at the goal.

Episode 16 of The Security Podcast of Silicon Valley came out on May 1, 2022, with Jon McLachlan hosting. The guest was Will Butler, then the red team tech lead at Robinhood and a co-founder of Truffle Security, the company behind the open source secret scanner TruffleHog (Podbean, Apple Podcasts, Spotify, Truffle Security, TruffleHog on GitHub). The episode is called “Will Butler, Red Team at Robinhood and Co-Founder of TruffleSecurity, From Hacking Airports to Leading Red Teams.”

Two threads run through the talk. One is what a red team finds: the gap between how a system is described and how it runs. The other is what happens next, and why Will wants red teams close enough to the rest of security to get the fixes shipped.

This page follows the conversation in order: a section for each part of it, eighteen figures, eight clips from the tape, two videos from Truffle Security, the full transcript, and five tools: a quiz, a scorecard for red team follow-through, a calculator for hosts a rollout missed, a timeline, and a to-do list. Where the decodes of the tape disagree on a word, the transcript puts its best guess in brackets, and the page doesn’t quote or retell those words.

From consulting to Robinhood

Jon opened with the path. Will started as a red team consultant at PwC, working for companies of every industry and maturity. At Apple, Will was on the AppSec team for service products such as Apple Music, iCloud, Apple Pay and Siri. Jon had been at Apple too, and the two compared notes.

Next came management. Will started the red team at Cruise, the self-driving car company in San Francisco, and grew it from zero to eight people.

Then back to hands-on work, as a staff red teamer at BitMEX, a small company facing real stakes and real adversaries. At the time of the episode, Will was the red team tech lead at Robinhood: designing assessments, presenting the results, and using them to shape the security roadmap.

Wikipedia describes Cruise as a General Motors subsidiary that worked on driverless taxis, until GM stopped funding it in December 2024 (Wikipedia). BitMEX is a cryptocurrency exchange and derivatives trading platform, founded in 2014 (Wikipedia). Robinhood is an electronic trading platform based in Menlo Park, which listed on Nasdaq on July 29, 2021 (Wikipedia).

Jon also introduced Will as a co-founder of Truffle Security, which builds an enterprise product around TruffleHog. Truffle Security’s chief executive, Dylan Ayrey, was the guest on episode 9 (Truffle Security). Will named two more co-founders, Dustin Decker and Julian Dunning.

Six cards. PwC: red team consultant, for companies of every industry and maturity. Apple: AppSec for service products: Siri, iCloud, Apple Pay and Apple Music. Cruise: built the red team from zero to eight people. BitMEX: staff red teamer, back to hands-on work at a small company with real stakes. Filled, Robinhood: tech lead for red team work, designing assessments, presenting results, shaping the roadmap. Truffle Security: co-founder, as Jon introduced Will, the company behind the open source TruffleHog.
Will's path, as Jon read it out on the show. Sources: the episode and Wikipedia.

The story Will likes to tell starts with a summer job at a local airport. It turned into an internship in the airport’s IT department. While chasing a bug for a coworker, Will found a web application on the network.

“And I noticed that it was returning a very suspiciously small session ID cookie.”

Will Butler, at 4:59

The session ID was a number. After logging out and back in, it had gone up by five. A short script tried thousands of session IDs and landed on an administrator session. Wikipedia calls this session hijacking: using a valid session to get unauthorized access to a system (Wikipedia).

“It turns out that it controlled almost all physical security at the airport, everything from the hand readers to the door locks to the cameras.”

Will Butler, at 5:37

Listen · Will Butler on episode 16, from 4:59
About 35 seconds: Will on the session ID that gave away an airport. The clip stops on its own. Read the full transcript.

An independent contractor wrote the software. It ran at many airports across the country, and it faced the internet for maintenance. The report went up to the head of the local port authority, and the Secret Service, already at the airport for a security assessment, heard about it.

“getting it patched, and they hired me to review some of their software after that.”

Will Butler, at 6:28

That review was the start of Will’s consulting work.

Six numbered steps. 1, a web application on the airport's network hands out a session ID cookie that is a small number. 2, log out and back in, and the number has gone up by five. 3, a short script tries thousands of IDs and lands on an administrator session. 4, the application runs almost all physical security at the airport: hand readers, door locks and cameras. 5, a contractor wrote it, it runs at many airports, and it faces the internet for maintenance. 6, the contractor patches it and hires Will to review more of its software. Below: Wikipedia calls this session hijacking, using a valid session to get unauthorized access to a system.
From a small number in a cookie to the airport's door locks, in the order Will told it. Sources: the episode and Wikipedia.

Three lenses on security

Will reads security through economics, finance and philosophy. On philosophy, Will quoted a line from someone else:

“philosophy is the science of thinking clearly”

Will Butler, at 8:04

Will credits it with more rigor. Jon added a moral angle.

“And I suppose being a red team expert, it changes your relationship to power, right?”

Jon McLachlan, at 8:23

Economics shows what drives attackers. Finance shows the business impact of a vulnerability. Jon carried the point on: a finding can feel scary from a security point of view and still not touch the core business, so the question is whether reaching 100% is worth the cost, or the risk is one to accept. Will added that attackers face incentives too, and Jon agreed that some of them make the same economic choices as businesses.

Three cards. Economics: what drives attackers, who weigh costs and payoffs as a company does. Finance: what a finding costs the business, and whether closing it all the way pays. Philosophy: clear thinking, for more rigor, and Jon added ethics, since red teamers can make systems do almost anything. Below, a quote from Will on philosophy.
Economics, finance and philosophy, and what each one adds to red team work. Source: the episode.

Attackers who work like dev teams

Will likes reading leaked chat logs from threat actors. Many groups run like an average developer team, with site reliability engineers, software engineers and product managers. A security podcast Will listens to makes the same point. Jon had pictured a faceless adversary.

“They’re human beings, just like the rest of us.”

Jon McLachlan, at 10:39

The division of labor also makes attribution hard.

“One crew might be like the malware developer for several other crews that actually run a campaign.”

Will Butler, at 10:52

Another crew might get initial access and hand it to a different crew that uses it.

Two rows of boxes joined by arrows. One crew builds the malware, and several crews, filled, run campaigns with it. An access crew gets the first foothold, and another crew, filled, uses that access. Below: leaked chat logs show many groups organized like a developer team, with SREs, software engineers and product managers. Split work makes attribution hard, since the crew that built a tool may not be the crew that used it.
Crews that split the work, as Will described them, and why it makes attribution hard. Source: the episode.

Findings that break workflows

Will’s best days are when a red team changes an organization. Many readouts go nowhere, because the findings are inconvenient.

“We tend to find vulnerabilities at the boundaries between different systems and abstractions, and things that are fundamentally insecure by design.”

Will Butler, at 12:07

A finding like that can’t be handed over as a configuration change, a patch or an output encoding fix. The example Will gave was a CI system that is insecure by design, where the fix breaks many other teams’ workflows. Sometimes an assessment starts a large tiger team, a group of specialists put together to solve one problem (Wikipedia), and the breaking changes ship. Core infrastructure changes, the attack surface shrinks, and people are happy with it. Those are Will’s best days.

Two rows of boxes joined by arrows. First row: a finding that is insecure by design, then a dashed readout, then nothing. Second row: the same finding, a tiger team of specialists on one problem, and a filled box, the breaking change ships. Below: these findings sit where systems meet, so a setting, a patch or an encoding fix won't close them, and Will's example was a CI system. When the change ships, the attack surface shrinks, and that makes for Will's best days.
Two paths for a finding that is insecure by design: a readout that stalls, or a tiger team that ships the breaking change. Sources: the episode and Wikipedia.

A crawler that ran attacker JavaScript

The other kind of best day is a good bug, and Will told the story of a favorite. It started before an engagement, while the team registered phishing domains for a client.

“And I noticed that approximately every hour on the hour after we registered these domains, we would get a very specific set of HTTP requests from some odd user agents.”

Will Butler, at 14:29

The requests came from way outdated Chrome and an outdated Python requests user agent. The client ran software that compared new domains with the company’s own domains and pages, to catch lookalikes early. Wikipedia calls this kind of lookalike domain typosquatting (Wikipedia). Will liked the tool, and saw a way in: anything that renders attacker-controlled JavaScript is attack surface.

The software loaded each page in a headless browser, a browser with no graphical interface that renders HTML and runs JavaScript (Wikipedia). The team sent back a payload that used the WebRTC API to read the local IP addresses of the machine running it. WebRTC gives browsers real-time communication, and Wikipedia notes that its IP address leak comes from the way it works (Wikipedia). More JavaScript took screenshots of the web services on that local /24 network.

Listen · Will Butler on episode 16, from 15:20
About 30 seconds: Will on the payload that mapped the network behind a headless browser. The clip stops on its own. Read the full transcript.

From a dashboard to every laptop

One of the screenshots showed a dashboard, a vendor product that displayed statistics from a database. Its URL’s query string was the name of the table, and the product evaluated that string as code in an esoteric programming language.

“It’s literally an RCE that’s triggerable from a simple GET request.”

Will Butler, at 16:52

The JavaScript in the headless browser could trigger it. The host it reached was a letdown: little network access, no route to the AWS metadata API, and a read replica of a non-production database with nothing of interest in it.

Then Will looked at how the product handled async results. A database client could name a stored procedure for the server to call on the client when the results were ready. A sandbox was supposed to stop arbitrary code. Will found a way out of the sandbox, and every engineer working on the database was connected to that test database.

“So I was able to exploit their laptops with this sandbox breakout.”

Will Butler, at 18:26

A few iterations later, most hosts in the client’s network were compromised. Jon compared it to tugging on a loose thread at the end of a sweater until the whole thing unravels.

It took a few days. The team was already working with the client’s security team by then, so it asked what the product was and skipped the fingerprinting.

Ten numbered steps. 1, register phishing domains for a red team exercise, before the engagement starts. 2, every hour, requests arrive from outdated Chrome and Python user agents, the client's lookalike checker. 3, the checker loads each page in a headless browser, so it runs the red team's JavaScript. 4, WebRTC gives away the local IP addresses of the machine running that browser. 5, screenshots of the web services on that /24 turn up a vendor dashboard. 6, the dashboard evaluates its query string as code, remote code execution from a GET request. 7, the host is a dead end, with little network access and a non-production read replica. 8, the database product calls stored procedures on its clients, behind a sandbox. 9, a sandbox escape reaches every engineer whose laptop is connected to the test database. 10, a few iterations later, most hosts in the network are compromised, within a few days. Below, a quote from Will about exploiting the laptops.
The favorite bug, step by step, from a phishing domain to most of the client's hosts. Source: the episode.

More art than science

Jon sees red teaming as more art than science: intuitions, and a flicker at the edge of vision that a normal engineer writes off as an edge case.

“almost more of an art than a science”

Jon McLachlan, at 19:37

Will reached for Paul Graham’s essay Hackers and Painters, from May 2003, which, as Will put it, says hackers are more like painters than engineers. In the essay, “What hackers and painters have in common is that they’re both makers” (Paul Graham). It also gave its title to a 2004 essay collection (Wikipedia).

“But I actually think a lot of his arguments can extend to offensive security engineers.”

Will Butler, at 20:16

There’s a lot of creativity in the work, Will said, and little formal reasoning about exploits and lateral movement. Jon called that a strength. Red teamers don’t bring preconceptions, so they find out how a system works for themselves.

“you are actually looking at the real bits and pieces that are not what they should be, but what they are.”

Jon McLachlan, at 20:43

A tether to reality

Will picked up Jon’s point. Security and engineering teams often describe their systems the way they should be, the way the documentation says, or the way someone told them. Few of them have operated those systems.

“And so there’s a lot of like little subtle differences between what the documentation says or what somebody tells you and what’s actually going on.”

Will Butler, at 21:26

A red team brings back that ground truth. The story: over beers, Will walked a head of infrastructure through an idea for an attack chain. Two thirds of the way in, the head of infrastructure stopped it. The chain needed SSH into a box, the boxes used an SSH certificate authority, and a certificate required a second factor that the attacker wouldn’t have at that point.

“But like any diligent red teamer, I’m going to try anyway.”

Will Butler, at 22:23

It worked.

“And it’s not because he was wrong.”

Will Butler, at 22:31

SSH CA ran on most systems. The team didn’t know about some systems, so SSH CA never reached them, and they still took passwords and keys. On others it broke some automation, got rolled back, and never came back. A third set took SSH CA certificates and other forms of SSH authentication too.

Listen · Will Butler on episode 16, from 22:29
About 30 seconds: Will on the three kinds of hosts an SSH certificate rollout missed. The clip stops on its own. Read the full transcript.

“I would call that operational rigor.”

Jon McLachlan, at 23:16

Will agreed: exceptions like these are what get a company breached.

The calculator below counts the hosts a rollout like that leaves open.

Three boxes joined by arrows: an attack chain that needs SSH to a box, a dashed SSH CA that needs a second factor, and a filled box, it worked through the exceptions. Below, three cards. Unknown hosts: hosts nobody knew about never got SSH CA and still took passwords and keys. Rolled back: SSH CA broke some automation, came off, and never went back on. Both ways: hosts took SSH CA certificates and other SSH logins too. Below, a quote from Jon: operational rigor.
An SSH CA rollout on paper, and the three groups of hosts Will found outside it. Source: the episode.

When the red team breaks something

A worst day, for Will, is causing a production issue.

“So red teams inherently have to do some risky things.”

Will Butler, at 24:06

Red teams change systems they don’t understand as well as they should, and some of those systems serve production traffic. Process and tooling cut the risk, and once in a while something slips through. Will’s teams have kept outages contained, because an internal red team works closely with the other engineering teams.

The favorite bug had a cost. The database’s runtime was single-threaded and synchronous by default. Will’s script to list and exploit the connected clients ran synchronously, so nothing else on the system could make progress while it ran. It was a test system, so no production traffic was lost. By Will’s own account, few red teamers are as cautious, and the team works to understand a system before acting on it.

Jon noted that a denial of service is too easy to call a red team success. Some systems Will has worked on were fragile, under heavy load or in odd configurations. So the team limits the risk.

“Even if that means, by the way, taking another path.”

Will Butler, at 26:53

Will has had teams drop a promising exploit or lateral movement opportunity and backtrack, because it was too risky.

Two cards. Why it happens: red teams change systems they don't fully understand, and some of them serve production traffic. The favorite bug's cost: a synchronous script on a single-threaded runtime stopped every other request on the test system. Below, three rules: outages stay contained when the red team works closely with engineering. Understand a system before acting on it. Filled, when a path looks too risky, take another one, even if it means backtracking.
How a red team causes an outage, what happened with the favorite bug, and the rules Will's teams use to limit risk. Source: the episode.

How the team runs an engagement

Jon asked how the team splits the work. Will’s way: meet, list the research tasks, such as the initial recon, and split them up. When the team is ready to act on what it found, everyone meets on Zoom or in a conference room.

“we actually get together and do that stuff live together.”

Will Butler, at 28:05

Then the research splits up again, and the loop repeats until the objective is done.

Operators pair up when research overlaps or needs someone’s expertise. During the independent stretches, updates go to Slack, and each operator works on vulnerability research or recon. Jon heard good feedback loops in that.

Four boxes joined by arrows: plan, list the research. Split up, recon and research. Filled, execute, live and together. Repeat, to the objective. Below: live means one Zoom call or one conference room. Between sessions, updates go to Slack, and operators team up when research overlaps or needs an expert. Below, a quote from Will on getting together to do the work live.
The loop Will's teams run: plan, split up, execute together, repeat. Source: the episode.

Inside or outside

Jon turned to internal red teams and external ones, also called contracted pen testers. A penetration test is an authorized simulated cyberattack on a system (Wikipedia). Will sees people manage one kind as if it were the other and end up disappointed.

An external red team is closer to the real adversary, and it faces the same constraints an attacker does.

“They have a time box.”

Will Butler, at 30:40

It knows little about the company’s security, its controls, its projects or its technology, and has to work all of it out. An internal red team will notice things anyway. Will wants it integrated with the rest of security, knowing what detection and response can do and where the common vulnerabilities are.

“Now, a disadvantage of being so closely integrated is you are biased.”

Will Butler, at 31:41

An internal team can share everyone’s blind spots. In return it can drive change, because it knows each team’s constraints and earns their trust in the aftermath of an assessment. It can help detection and response improve their processes, help the SRE team redesign what it broke, and patch the bug it found.

“You can help patch a bug that you found.”

Will Butler, at 32:23

Jon pointed out that every fix makes the red team’s next attempt harder.

Two cards. External red team: close to a real adversary, with a time box and little knowledge of your controls, projects or technology, less biased, with specialists to draw on. Filled, internal red team: works inside security and knows what detection and response can do, biased, with shared blind spots, and able to help ship the fix. Below: a penetration test is an authorized simulated cyberattack on a system, per Wikipedia. Will's advice is to test the same system with both kinds of team.
An internal and an external red team side by side, as Will compared them. Sources: the episode and Wikipedia.

Red teams as product managers

How independent should an internal red team be? Will’s answer comes from experience.

“Personally, I’ve had the greatest success when my internal red teams were not independent at all.”

Will Butler, at 32:41

Integration lets a red team act almost like a security product manager. In a product organization, product managers decide what to build and engineering decides how. In security, InfraSec, AppSec and detection platform teams know how to build: log ingestion pipelines that scale, fixes across a code base, infrastructure that is secure by default.

“But they might not necessarily be the ones who know the most about what security controls, if in place, would make a real attacker’s life most miserable.”

Will Butler, at 33:39

Red teams, and threat intel teams, are the ones qualified to say that.

Listen · Will Butler on episode 16, from 33:39
About 25 seconds: Will on red teams as security product managers. The clip stops on its own. Read the full transcript.
Two cards. In a product team: product managers pick the features, and engineering works out how to build them. Filled, in security: the red team and threat intel say which controls would hurt attackers most, and the InfraSec, AppSec and detection platform teams build them. Below, a quote from Will on red teams that were not independent at all.
What a product manager does for a product team, and what a red team can do for security. Source: the episode.

A report for the buyer

Jon added a use for external reports. A company can ask, or pay, for a customer-facing summary: an initial assessment, a few engineering cycles to fix the criticals and highs, then a second assessment that shows the delta. In a B2B deal, the buyer’s security team asks the hard questions.

“Do you have any internal pen test reports that you can share with us?”

Jon McLachlan, at 35:08

Handing over those reports, Jon said, breaks down sales barriers in B2B deals.

One report, or a patch

Reports mark another difference. An internal team can add value after an assessment in many ways: submit a patch, write the detection rule it wished existed, or join an architecture meeting.

“An external red team usually only has one, which is the report.”

Will Butler, at 35:30

Will learned the other side of this at Apple, reading consultants’ reports on the AppSec team.

“I never understood how useless my own recommendations were as a consultant until I started being on an internal security team.”

Will Butler, at 35:58

Consultants don’t have the internal context, so their recommendations come out generic. In Will’s first days at Apple, the team heard Will’s standard consulting answer and listed 16 edge cases, limitations and restrictions that ruled it out.

“Now help us.”

Will Butler, at 36:57

Listen · Will Butler on episode 16, from 36:29
About 30 seconds: Will on a first week at Apple and the limits of a consulting report. The clip stops on its own. Read the full transcript.

Will had to learn those restrictions and design a fix that mitigated enough. Internal teams find more, with the advantage of knowing the stack and the people. Their reports are lighter, because the findings go into tickets, meetings and pull requests.

“So I actually find the external team’s reports tend to be of significantly higher quality.”

Will Butler, at 38:31

Jon’s take: the report is an external team’s only channel, and its name goes on it.

Two cards. External: one channel, the report, and the firm's name is on it, so it tends to be thorough. Filled, internal: a bug patch, a detection rule, a voice in the architecture meeting, tickets and pull requests. Below: in Will's first days at Apple, a standard consulting answer ran into 16 edge cases, limits and restrictions. Internal teams find more, and their written reports are lighter. Below, a quote from Will: now help us.
What each kind of red team leaves behind after an assessment. Source: the episode.

Experts you can’t hire

External firms can hire subject matter experts that a single company can’t justify, such as a full-time cryptocurrency expert.

“But Trail of Bits does.”

Will Butler, at 39:10

Trail of Bits can sell that expert’s time to every client who needs it. An internal red team is made of generalists. So Will likes both on the same system. The external team is less biased and brings deeper specialists, and the internal team brings context.

At one company, Will’s team and an external red team ran an assessment together.

When should a startup start?

Jon asked when a startup, especially in a regulated space, should get its first assessment. Will expects few early startups to hire a red team, and still sees value in starting early.

“hiring a red team early, even if it’s just a single person”

Will Butler, at 40:53

Even someone with the background who doesn’t red team full time can bring the product management view and the ground truth, and help design the first infrastructure. Will expects that to pay off for years. Red teams usually get invited in after infrastructure has so much momentum that it’s hard to change.

An interview question with no floor

Will’s favorite interview question for an offensive security role is a hypothetical. The first assignment is to recover one file worth a lot, with no access and no inside knowledge.

“some like individual file of immense value.”

Will Butler, at 42:22

The candidate starts with recon: IP space, domains and subdomains, where they point, which cloud, whether there are S3 buckets. Will supplies assumptions to narrow it down, such as what the recon or a port scan returned. Then Will drills into each topic the candidate raises until someone’s knowledge runs out, the candidate’s or Will’s.

“because the candidate is almost entirely in control of the technical topics that we discuss.”

Will Butler, at 44:18

The example was command injection.

“explain to me a command injection vulnerability as if I’m the developer who wrote it.”

Will Butler, at 44:39

The follow-ups go deeper: the root cause, the fix, finding it across many applications, debugging error conditions, exploiting it blind, and getting results out when egress is locked down. Wikipedia covers command injection under code injection, where a program treats external input as executable commands (Wikipedia).

“What if egress was heavily restricted?”

Will Butler, at 45:10

Listen · Will Butler on episode 16, from 44:31
About 40 seconds: Will on drilling into command injection in an interview. The clip stops on its own. Read the full transcript.

Jon heard the creativity from earlier in it: a depth-first search into small glimmers of odd behavior in software written by people who mean well.

Four boxes joined by arrows: the ask, one file worth a lot. Recon: IP space, domains, S3. Assumptions: what the scan found. Filled, drill down until someone stops. Below, six numbered follow-ups on command injection: explain it as if to the developer who wrote it. Name the root cause, and the fix. Find it across a large set of applications. Debug the exploit through error conditions. Exploit it without seeing the output. Get results out when egress is locked down.
The interview question and the drill-down on command injection, in Will's order. Sources: the episode and Wikipedia.

Insight under pressure

That reminded Jon of a book, Gary Klein’s Seeing What Others Don’t: The Remarkable Ways We Gain Insights, from 2013. Klein is a research psychologist who studies how experts, such as firefighters, decide under time pressure, and Wikipedia credits him with a multi-path model of insight (Wikipedia). Jon described two of the paths.

“One of them is noticing a contradiction.”

Jon McLachlan, at 47:44

The other is creativity under constraint, and time is often the constraint. Jon told the story of firefighters on a hillside, with the wind turned against them and a wildfire they couldn’t outrun.

“And the best way to protect themselves from this raging fire was to actually start a fire.”

Jon McLachlan, at 49:00

Ground that has already burned can’t burn again. Wikipedia’s article on the Mann Gulch fire, in Montana in 1949, describes the same move: the foreman lit an escape fire and lay face down in the burned area while the main fire passed (Wikipedia). Jon tied it to red team work: a time box and the limits of the system push people toward creative answers.

Two cards. A contradiction: you think the world works one way, notice something that says it doesn't, and dig into why. Filled, a constraint: time and limits force a creative answer, as with an escape fire. Below, a timeline. August 5, 1949, Mann Gulch: in Montana, wind and slope push a wildfire faster than 16 men can retreat. The same day, filled, an escape fire: the foreman lights a fire and lies face down in the burned area while the main fire passes.
Two of the paths to insight Jon described, with the escape fire as the example of a constraint. Sources: the episode and Wikipedia.

The binary you can’t fix

Will has a second interview question that flips to the defender’s side. A vendor ships an executable known to be insecure. The business needs it, and it has to accept input from across the internet.

“You can’t reverse it or change it for legal reasons.”

Will Butler, at 50:23

So how do you run it as securely as possible? Will asks because, in Will’s view, most security people focus on the vulnerabilities and less on the other layers of defense.

Constraints also shaped Will’s time at BitMEX: a small team, high stakes and real adversaries.

“Go to a place with high stakes, constraints, and real adversaries, and you will be forced to learn a lot.”

Will Butler, at 51:28

Listen · Will Butler on episode 16, from 51:12
About 25 seconds: Will on what a small team with high stakes taught. The clip stops on its own. Read the full transcript.

Jon added that growth happens outside your comfort zone, one of the best reasons to join or start a startup.

Four cards. Insecure: known to be full of vulnerabilities. Required: the business needs it. Untouchable: no reversing, no changes, for legal reasons. Exposed: takes input from across the internet. Below, filled: the question is how to run it as safely as possible, and with the bug off the table, the answer lies in other defenses around it. Below, a quote from Will on the legal constraint.
The defender's question: four constraints, and the layers that remain when the bug can't be fixed. Source: the episode.

The engagement ends at the goal

Jon asked for parting words, and Will gave two. The first was red teams as product managers.

“First word of wisdom is red teams can make really interesting PMs.”

Will Butler, at 52:10

The second was about layers. Between an attacker exploiting a vulnerability and reaching a goal, a defender can put many layers of defense in the way, and each one is a chance to make attackers miserable. Wikipedia calls the idea defense in depth (Wikipedia).

“The engagement doesn’t end with the vulnerability. It ends when they accomplish their goal.”

Will Butler, at 52:47

“I think a lot of people tend to over-index on vulnerabilities specifically.”

Will Butler, at 52:58

Listen · Will Butler on episode 16, from 52:29
About 35 seconds: Will on the layers between a vulnerability and an attacker's goal. The clip stops on its own. Read the full transcript.
Four boxes joined by arrows: a bug, exploited. A dashed layer, a chance to stop it. Another dashed layer, another chance. Filled, the goal, what they came for. Below: defense in depth means several independent methods against the same attack, per Wikipedia. Filled: many teams focus too much on the bugs themselves, and every layer is a chance to make the attacker's work miserable. Below, a quote from Will on where an engagement ends.
An exploited vulnerability, the layers between it and the attacker's goal, and the goal. Sources: the episode and Wikipedia.

Becoming a red teamer

Jon’s last question: what should someone do to get into red teaming? Will’s answer starts with breadth. Red teamers have to be generalists. A specialty is fine, but the list is long: social engineering and phishing, how the internet works, how modern software is built, several programming languages, finding and exploiting application security bugs, how IT organizations work, and how software gets onto a laptop.

“You should really thoroughly understand the defender.”

Will Butler, at 54:01

That means the defender’s monitoring tools, the footprint your activity leaves in them, and the person reading the alerts.

“Understand that there’s a human sitting behind that screen, looking through thousands of alerts”

Will Butler, at 54:12

Will’s advice is to work on different kinds of security teams, such as detection and response, secure infrastructure and application development, and to build things and break them.

Contributing to open source helps too. Find security tools that interest you, learn how they work, fix bugs and add the features you’ve wanted. Jon and Will both pointed listeners to TruffleHog, which is open source (GitHub).

Three cards. Know a lot: social engineering and phishing, the internet, modern software, several languages, AppSec bugs, IT organizations, and how software reaches a laptop. Know the defender: their monitoring tools, the footprint you leave in them, and the person reading the alerts. Filled, learn by doing: work on other security teams, build things and then break them, and contribute to open source tools.
What a red teamer has to understand, and the ways Will suggested to learn it. Source: the episode.

What happened after the tape

A Series B for Truffle Security. On November 6, 2025, Truffle Security announced a $25 million Series B led by Intel Capital and a16z. The release counted more than 23,000 GitHub stars for TruffleHog, 15 million downloads and more than 250,000 runs a day (Truffle Security). The Series A, $14 million led by a16z, came on December 8, 2021 (Truffle Security).

Revoking leaked secrets for free. On November 4, 2026, Truffle Security launched a free Secrets Revocation Program for companies that issue credentials, starting with 1.4 million verified, live credentials found in public code and packages. In a 2025 study, 81% of 21,981 live keys reported to their owners and issuers still worked six months later (Truffle Security). Reporting alone didn’t close the gap, which is the point of One report, or a patch.

Two videos from Truffle Security. Four weeks before this episode, Truffle Security posted an introduction to TruffleHog v3, “an open source secrets detection engine.” The day after the episode, it posted a NahamCon 2022 talk on how red teams and state-sponsored attackers get into companies. Truffle Security made both videos.

Introducing TruffleHog v3, from Truffle Security's channel, 9 minutes, posted on April 4, 2022. Watch on YouTube.
No Malware Needed, a NahamCon 2022 talk from Truffle Security's channel, 27 minutes, posted on May 2, 2022. Watch on YouTube.

More is on Truffle Security’s YouTube channel, LinkedIn page and X account.

Cruise wound down. Cruise suspended operations in October 2023 after a series of incidents, and GM stopped funding it in December 2024, moving the work into driver assistance for personal cars (Wikipedia).

Truffle Security on this show. Dylan Ayrey was the guest on episode 9, on how open source makes the world more secure. On episode 84, Jack Cable of Corridor named Dylan among the angel investors in Corridor’s first round.

A timeline. December 8, 2021: Series A, $14 million for Truffle Security, led by a16z. April 4, 2022: TruffleHog v3, an introduction video from Truffle Security. May 1, 2022: this episode. May 2, 2022: NahamCon 2022, Truffle Security posts a talk on how red teams get in. Dashed, October 2023: Cruise suspends operations after a series of incidents. Dashed, December 2024: GM stops funding Cruise. Filled, November 6, 2025: Series B, $25 million, led by Intel Capital and a16z. Filled, November 4, 2026: a free program to revoke 1.4 million live leaked credentials.
After the tape: Truffle Security's funding and a free program for leaked secrets, with the two videos and Cruise. Sources: Truffle Security, YouTube and Wikipedia.

Did you catch it? A seven-question quiz

Seven questions from the episode. Each answer links the moment in the transcript or its section.

Quiz · seven questions

Did you catch it?

Pick an answer to see whether you caught it.

  1. What gave away the administrator session in the airport's security system?

    a. "And I noticed that it was returning a very suspiciously small session ID cookie." Transcript, 4:59.

  2. Where do the findings that stall tend to sit, in Will's experience?

    c. "We tend to find vulnerabilities at the boundaries between different systems and abstractions, and things that are fundamentally insecure by design." Transcript, 12:07.

  3. Why was the client's lookalike-domain checker attack surface?

    b. The checker ran the red team's JavaScript in a headless browser. See A crawler that ran attacker JavaScript.

  4. Why did the attack chain work, even with SSH CA in place?

    a. SSH CA ran on most systems, and the exceptions were enough. See A tether to reality.

  5. What's the main drawback of an internal red team that works closely with the rest of security?

    b. "Now, a disadvantage of being so closely integrated is you are biased." Transcript, 31:41.

  6. What can a red team tell security engineering that the builders may not know?

    c. "But they might not necessarily be the ones who know the most about what security controls, if in place, would make a real attacker's life most miserable." Transcript, 33:39.

  7. When does an engagement end, in Will's words?

    b. "The engagement doesn't end with the vulnerability. It ends when they accomplish their goal." Transcript, 52:47.

Do your red team’s findings get fixed?

Eight questions on whether your testing turns into fixes, built from the episode. Each tip links the section behind it.

Scorecard · eight questions

Do your red team's findings get fixed?

Answer for the systems you'd least like an attacker to reach, whoever tests them today. If two answers fit, pick the lower one.

  1. What happens to findings after an assessment?

    • A report, and little changes
    • Tickets for the easy fixes
    • The testers help ship the fixes: patches, detection rules, design reviews

    "An external red team usually only has one, which is the report." See One report, or a patch.

  2. When a finding is insecure by design and the fix breaks other teams' workflows, what happens?

    • It waits
    • It has an owner but no date
    • A tiger team ships the breaking change

    "We tend to find vulnerabilities at the boundaries between different systems and abstractions, and things that are fundamentally insecure by design." See Findings that break workflows.

  3. Does anyone test that a control runs everywhere the documentation says?

    • We trust the documentation
    • Spot checks now and then
    • Someone hunts the exceptions: unknown hosts, rollbacks, fallback logins

    "And it's not because he was wrong." See A tether to reality.

  4. How closely does your red team work with detection and response?

    • Separately, with results at the end
    • They share readouts
    • The red team knows what they can detect and helps them improve

    "But the advantage is that it's much easier for you to influence change" See Inside or outside.

  5. Who decides which security controls to build next?

    • Whoever builds them
    • The next audit
    • The red team and threat intel weigh in on what would hurt attackers most

    "Personally, I've had the greatest success when my internal red teams were not independent at all." See Red teams as product managers.

  6. If an attacker exploits one vulnerability, what stands between it and your most sensitive data?

    • Not much
    • One or two controls
    • Several layers, each one tested

    "The engagement doesn't end with the vulnerability. It ends when they accomplish their goal." See The engagement ends at the goal.

  7. When did an outside team last test the same systems?

    • Never
    • More than a year ago
    • In the last year, with the specialists the system needs

    "But Trail of Bits does." See Experts you can't hire.

  8. Does your red team have rules for production risk?

    • No
    • Informal ones
    • Written limits, and a habit of backing off risky paths

    "Even if that means, by the way, taking another path." See When the red team breaks something.

Findings that sit in a report

The testing happens, and little changes. Bring the testers into tickets, pull requests and design reviews, and give the hardest finding an owner and a date.

Easy fixes only

The quick findings get fixed. Next, hunt the exceptions to the controls you count on, and staff a tiger team for the finding that breaks workflows.

Findings that change things

Keep an outside test on the calendar for fresh eyes, and keep adding layers between a bug and your most sensitive data.

How many hosts did the rollout miss?

The calculator counts the hosts outside a control like SSH CA, using the three groups from A tether to reality.

Calculator

How many hosts did the rollout miss?

Five numbers: the hosts in your fleet, the share the team doesn't know about, the share where the rollout was rolled back, the share that still takes other logins, and how many hosts an attacker can reach.

100hosts that still take a password or key
10%of the fleet outside the rollout
65%chance an attacker lands on one

In a fleet of 1,000 hosts, 10% sit outside the rollout: about 100 hosts that still take a password or key. An attacker who reaches 10 hosts at random has a 65% chance of landing on at least one.

The three groups come from the tape. The percentages are placeholders. The math assumes the groups don't overlap and that the attacker reaches hosts at random. In Will's story, the exceptions were enough to keep the attack chain going.

From an airport to TruffleHog

The companies on Will’s path, Truffle Security, the field around them, and this show. Filter by topic.

Timeline

From an airport to TruffleHog

  1. Mann Gulch

    A wildfire in Montana, where the foreman survives by lighting an escape fire (Wikipedia, on this page).

  2. Tiger teams

    A paper on program management uses the term for a team of specialists sent after every possible failure (Wikipedia).

  3. Hackers and Painters

    Paul Graham's essay on hackers as makers (Paul Graham, on this page).

  4. The essay collection

    Hackers & Painters comes out from O'Reilly Media (Wikipedia).

  5. Cruise

    The self-driving car company is founded (Wikipedia).

  6. Robinhood

    The trading platform is founded (Wikipedia).

  7. Seeing What Others Don't

    Gary Klein's book on how people gain insights (Wikipedia, on this page).

  8. BitMEX

    The cryptocurrency derivatives exchange is founded (Wikipedia).

  9. The WebRTC leak

    TorrentFreak reports that WebRTC can expose a user's real IP address (Wikipedia, on this page).

  10. GM buys Cruise

    Cruise becomes a General Motors subsidiary (Wikipedia).

  11. Robinhood goes public

    Robinhood lists on Nasdaq as HOOD (Wikipedia).

  12. Episode 9

    Dylan Ayrey of Truffle Security on open source (episode 9).

  13. Series A

    $14 million, led by a16z (Truffle Security).

  14. TruffleHog v3

    Truffle Security posts an introduction to the new version (YouTube, on this page).

  15. This episode

    Will on the show (Podbean).

  16. No Malware Needed

    Truffle Security posts a NahamCon 2022 talk on how red teams get into companies (YouTube).

  17. Cruise pauses

    Cruise suspends operations after a series of incidents (Wikipedia).

  18. GM stops funding Cruise

    The work moves into driver assistance for personal cars (Wikipedia).

  19. Series B

    $25 million, led by Intel Capital and a16z (Truffle Security).

  20. Secrets Revocation Program

    A free program to revoke leaked credentials, starting with 1.4 million live ones (Truffle Security, on this page).

What to do this week

Five things, in the order of the episode.

  1. Hunt the exceptions to your controls. Pick one control you count on, such as SSH certificates or single sign-on, and look for the hosts that skip it: unknown systems, rollbacks and fallback logins, as in A tether to reality. The calculator shows how fast a few percent adds up.
  2. Give the hard finding an owner. For a finding that is insecure by design, put a small tiger team on it with a date, as in Findings that break workflows.
  3. Bring the red team into the fixes. Invite it to tickets, pull requests and design reviews, and ask which controls would hurt an attacker most, as in Red teams as product managers. The scorecard shows where to start.
  4. Scan for leaked secrets. Run an open source scanner such as TruffleHog over your code and its history.
  5. Get help with it. Our Penetration Testing team tests the way an attacker would, Product Security helps get the fixes into the code, and a vCISO can own the program. If you want to talk it through, book 15 minutes with us.

Key takeaways

  1. Small session IDs give sessions away

    A session ID that was a small number, up by five with each login, let a short script land on an administrator session in an airport's physical security system.

  2. Attackers work like teams

    Leaked chat logs show crews with engineers and product managers, and crews that split the work, which makes attribution hard.

  3. Hard findings need a tiger team

    Findings at the boundaries between systems are insecure by design. No configuration change or patch closes them, so a tiger team has to ship the breaking change.

  4. Anything that runs attacker code is surface

    A lookalike-domain checker that rendered pages in a headless browser ran the red team's JavaScript, and that led to most hosts in the network.

  5. Test the exceptions

    SSH CA ran on most systems. Unknown hosts, rollbacks and hosts that took other logins kept the attack chain going.

  6. Limit the risk of testing

    Red teams change systems they don't fully understand. Understand before acting, and take another path when one looks too risky.

  7. Integrate the internal red team

    An internal red team is biased, and it can drive change: patches, rules, design help, and trust built in the aftermath.

  8. Red teams can act as product managers

    Security engineering knows how to build. Red teams and threat intel know which controls would hurt attackers most.

  9. Use both kinds of red team

    External teams are less biased, bring specialists and write better reports. Internal teams have context. Have both look at the same system.

  10. The engagement ends at the goal

    Put layers between an exploited vulnerability and the attacker's goal, and look past the bugs themselves.

Chapters

Tap a chapter to play from there.

Pull quotes

“And I noticed that it was returning a very suspiciously small session ID cookie.”

Will Butler ·

“It turns out that it controlled almost all physical security at the airport, everything from the hand readers to the door locks to the cameras.”

Will Butler ·

“One crew might be like the malware developer for several other crews that actually run a campaign.”

Will Butler ·

“We tend to find vulnerabilities at the boundaries between different systems and abstractions, and things that are fundamentally insecure by design.”

Will Butler ·

“It's literally an RCE that's triggerable from a simple GET request.”

Will Butler ·

“So I was able to exploit their laptops with this sandbox breakout.”

Will Butler ·

“But I actually think a lot of his arguments can extend to offensive security engineers.”

Will Butler ·

“And so there's a lot of like little subtle differences between what the documentation says or what somebody tells you and what's actually going on.”

Will Butler ·

“But like any diligent red teamer, I'm going to try anyway.”

Will Butler ·

“I would call that operational rigor.”

Jon McLachlan ·

“So red teams inherently have to do some risky things.”

Will Butler ·

“we actually get together and do that stuff live together.”

Will Butler ·

“Now, a disadvantage of being so closely integrated is you are biased.”

Will Butler ·

“Personally, I've had the greatest success when my internal red teams were not independent at all.”

Will Butler ·

“An external red team usually only has one, which is the report.”

Will Butler ·

“I never understood how useless my own recommendations were as a consultant until I started being on an internal security team.”

Will Butler ·

“But Trail of Bits does.”

Will Butler ·

“hiring a red team early, even if it's just a single person”

Will Butler ·

“One of them is noticing a contradiction.”

Jon McLachlan ·

“You can't reverse it or change it for legal reasons.”

Will Butler ·

“The engagement doesn't end with the vulnerability. It ends when they accomplish their goal.”

Will Butler ·

“You should really thoroughly understand the defender.”

Will Butler ·

About the guest

Will Butler, Red Team Tech Lead, Robinhood

When this episode came out, Will Butler was the red team tech lead at Robinhood, designing assessments, presenting red team results and using them to shape the security roadmap. Jon also introduced Will as a co-founder of Truffle Security, the company behind the open source secret scanner TruffleHog. Before Robinhood, Will was a red team consultant at PwC, worked on the AppSec team for Apple's service products, started and grew the red team at Cruise, and was a staff red teamer at BitMEX. Will's path into security started with an airport's physical security system and a session ID that was a small number.

About the host

Jon McLachlan is co-founder of YSecurity, the on-demand cybersecurity team for startups, and of Cyberbase.ai. The Security Podcast of Silicon Valley is hosted by Jon McLachlan and Sasha Sinkevich. Jon McLachlan on LinkedIn ↗

Context and sources

  • Episode 16 on Podbean: Will Butler, Red Team at Robinhood and Co-Founder of TruffleSecurity, From Hacking Airports to Leading Red Teams. Source ↗
  • Truffle Security, December 8, 2021: Truffle Security raises $14 million Series A led by a16z. Source ↗
  • Truffle Security, November 6, 2025: a $25 million Series B led by Intel Capital and a16z, with more than 23,000 GitHub stars and 15 million downloads for TruffleHog. Source ↗
  • Truffle Security, November 4, 2026: a free Secrets Revocation Program, starting with 1.4 million verified live credentials. Source ↗
  • TruffleHog on GitHub: find, verify and analyze leaked credentials. Source ↗
  • Truffle Security on YouTube, April 4, 2022: Introducing TruffleHog v3. Source ↗
  • Truffle Security on YouTube, May 2, 2022: No Malware Needed, a NahamCon 2022 talk on how red teams get into companies. Source ↗
  • Wikipedia, Robinhood Markets: an electronic trading platform based in Menlo Park, public on Nasdaq since July 29, 2021. Source ↗
  • Wikipedia, Cruise: a self-driving car company owned by General Motors, which suspended operations in October 2023 and lost GM's funding in December 2024. Source ↗
  • Wikipedia, BitMEX: a cryptocurrency exchange and derivative trading platform, founded in 2014. Source ↗
  • Wikipedia, Red team: a group that simulates an adversary and reports back so an organization can improve its defenses. Source ↗
  • Wikipedia, Penetration test: an authorized simulated cyberattack on a computer system. Source ↗
  • Wikipedia, Session hijacking: using a valid computer session to get unauthorized access to information or services. Source ↗
  • Wikipedia, Typosquatting: lookalike domains that rely on typing mistakes. Source ↗
  • Wikipedia, Headless browser: a web browser without a graphical user interface. Source ↗
  • Wikipedia, WebRTC: real-time communication for browsers, and the IP address leak reported in January 2015. Source ↗
  • Wikipedia, Tiger team: a team of specialists assembled for one goal or problem. Source ↗
  • Wikipedia, Code injection: a program that treats external data as executable commands. Source ↗
  • Wikipedia, Defense in depth: several independent methods against the same attack. Source ↗
  • Paul Graham, Hackers and Painters, May 2003. Source ↗
  • Wikipedia, Gary A. Klein: a research psychologist of naturalistic decision making, and author of Seeing What Others Don't (2013). Source ↗
  • Wikipedia, Mann Gulch fire: August 5, 1949, in Montana, where the foreman survived by lighting an escape fire. Source ↗

Transcript

Automatic transcription. Jon McLachlan hosts this episode with Will Butler, red team tech lead at Robinhood and a co-founder of Truffle Security. There is no diarization on the tape, so speaker labels were assigned from the content and checked against reference recordings of each voice from this episode, and a few short lines may sit with the wrong speaker. Names and terms were normalized where the recognizer misheard them (Jon McLachlan, Will Butler, Robinhood, BitMEX, Cruise, Truffle Security, TruffleHog, WebRTC, SSH CA, Trail of Bits, Risky Business). Words in [brackets] are best guesses where the audio is unclear, and [...] marks words no decode made out. Speech is transcribed as spoken, though many repeated words and false starts are left out, and so are short backchannels such as "yeah" where they overlap the other speaker.

Read the full transcript 9,764 words

Jon McLachlan

Welcome everyone to the Security Podcast of Silicon Valley. I'm here today with a very special guest, Will Butler. Good to have you on the show.

Will Butler

Thanks so much for having me, Jon.

Jon McLachlan

So Will brings with him a great deal of security experience, and I would love to go through just a quick summary. He got into security first by hacking airport physical security systems at the age of 15. He has degrees in economics and finance, oh, and philosophy, because he figured he could teach himself security but wanted to start his own company someday. He started actually as a red team consultant at PwC, a wide variety of companies in terms of industry [size], maturity. Then he moved over to Apple, where he was on the AppSec team for service products like Apple Music, iCloud, Apple Pay, Siri. I think your Siri might have heard that.

Will Butler

I think so. That's so funny. Siri, go away. It happens to me all the time. Yeah, it does.

Jon McLachlan

No, I was on the iTunes security team and the DRM team. So maybe we.

Will Butler

Oh, I'm sure you'll have some good stories.

Jon McLachlan

I bet we overlap. I was underneath that crazy Frenchman, [name unclear].

Will Butler

I know exactly who you're talking about. Really?

Jon McLachlan

Yeah. Okay. So they kept us like tucked away in the back corner and wouldn't let us talk to anyone else. But I knew other people existed. They must have been doing some really good stuff over there too. So anyway, after you were with Apple, you moved into management. You started a small red team focusing on the same areas. You started the red team at Cruise, the self-driving car company here in San Francisco. Looks like you grew that team from zero people up to eight people. You hired some illustrious folks like

Will Butler

[names unclear]. That's my boss here, Chris Gates.

Jon McLachlan

That's an amazing team.

Will Butler

[unclear] I love it too.

Jon McLachlan

And so you were doing some red teaming for Cruise, and next you decided that you do really enjoy the [foray] into people management. You're going to likely do it again, but your real passion was actually technical work and that you were just not done learning. And so [you] moved to BitMEX, the largest Bitcoin exchange in the world. You were a staff red teamer. That was also a huge learning experience because it was a small company facing very real stakes and probably some very interesting adversaries. And then you followed your manager from BitMEX, who's Chris Gates, over to Robinhood, where you are the red team tech lead. And at Robinhood, you do things like designing your assessments, delivering red team presentations, using your perspectives to influence the security roadmap where it makes sense.

Will Butler

Yeah, that's good.

Jon McLachlan

And you also are the co-founder of Truffle Security. And Dylan, he was actually on a previous podcast here on the [...] Security Podcast of the Silicon Valley. Gotcha. Gotcha both. Are there any other co-founders on Truffle Security that we should have on the podcast?

Will Butler

There are a few, yes. There's Dustin Decker and Julian Dunning, both extremely interesting. I'm sure that you'd have a great time talking to them.

Jon McLachlan

Amazing. We'll have to [pull them into] the podcast as well. And so it looks like at Truffle Security, you're taking an open source product, TruffleHog, and you're building out sort of an internal secret scanner that you've seen a lot of people struggle with. So instead of watch them struggle, you're offering an enterprise SaaS solution, I would imagine, to help drop in and get folks up and running. Hey, thank you so much for joining.

Will Butler

Yeah, thank you for having me.

Jon McLachlan

Okay, so maybe you'd like to share that initial story with all of our listeners of how did you get into security?

Will Butler

Yeah, absolutely. Yeah, this is the story I always like to tell people. So I got into security pretty young. I was a mischievous little kid and I went to a Montessori school with a kind of like basic computers class where they taught you how to make a web page and HTML and JavaScript and things like that. I gravitated towards security because I loved messing with the kids in my school and had read a lot of the hacker lore. And so I naturally gravitated towards that. So then at one point, when I was 15, I wanted to get a summer job and I found a job at the local airport cutting grass. But so my very first day, they were sending me to training on how not to get run over by airplanes when cutting the grass at an airport, which is, you know, good training.

Jon McLachlan

That's important. That sounds very important.

Will Butler

Yeah, so then right in the middle of the training, it breaks and it's some like old Adobe Captivate presentation. I thought I could fix it, so I tried and it worked. And they were like, oh, computer things, you should be an intern at our IT department. We need people. So I ended up being an intern all summer at their IT department. And one day I was looking into a bug for a coworker when I came across the system on our network. And I started looking at the web application. And I noticed that it was returning a very suspiciously small session ID cookie. It was just a number and it was about three orders of magnitude. And so you're thinking, like, huh, that's suspiciously low. I wonder what happens when I log out and log back. And it had only incremented by five. So I'm thinking, okay, I know how this algorithm works. So I wrote a quick little script to just try thousands of session IDs and see what happened. Turns out I landed on an administrator session and I could control almost everything in this application. So I decided to look around and figure out what this application did. It turns out that it controlled almost all physical security at the airport, everything from the hand readers to the door locks to the cameras. And so at that point, I go to my boss and I'm like, oh, this is bad. We should look into this. And he tells me something interesting, which is that the airport actually didn't write this as I thought they did. It was some independent contractor. And this software is deployed at many airports all throughout the country. And he also tells me that for maintenance reasons, this software is actually facing the internet as well. And it is [the deployment model].

Jon McLachlan

Wow.

Will Butler

Yeah, it's no good. So then we report it up to the head of our local little port authority that runs the airport. And he tells me that it turns out Obama's flying into this airport in two weeks. So the Secret Service is out conducting a security assessment of the airport and heard about it. So we ended up getting on the phone with the contractor, getting it patched, and they hired me to review some of their software after that. So then that's kind of like how I got into the consulting side of things. And I did end up going to college, but I figured, like you said, that I could maybe teach myself security and computer science, but it would be more difficult to teach myself the principles of economics and finance and accounting and things like that. I always wanted to start my own business someday. So I thought it'd be good to take this opportunity to learn something that I might not otherwise be able to in college. And so we'll see how that works out for me, because I don't actually have a CS degree and I do occasionally feel that pain. But it's, I'm glad that I ended up with those degrees because they've definitely been useful on the business side of things.

Jon McLachlan

Well, I could imagine it complements all of your technical skills [unclear]. So if you have the drive to go off and dig into those technical pieces, that sounds like a perfectly good use of your time in college. So you also listed philosophy as an interest, a point of interest. I also love philosophy. Do you have a favorite philosopher?

Will Butler

Ooh, so I like the Greeks. I read a lot of the ancient Greek philosophers. I got to say, I'm relatively out of practice. I have not read a lot of philosophy lately, but it was something that I was always interested in at the time. And so I figured I should just write a few papers and take the classes. It wouldn't be that much more work than I'd otherwise be doing. But I found it rewarding because, I forget who said this, but someone smarter than me, there's a quote I like that's, philosophy is the science of thinking clearly. And I really think it's helped up my rigor quite a lot.

Jon McLachlan

Yeah, I get that 100%. I really enjoyed it. I have to say, like understanding where meaning comes from or morality in some sense. And I suppose being a red team expert, it changes your relationship to power, right? You can make these systems do pretty much anything with enough time and energy and focus. So having a strong moral compass, I'm sure, is part of the territory in your line of work, specifically in a red team, huh?

Will Butler

Yeah, I think that's a really good point. I think that my degrees have interesting security applications that haven't always been obvious to me and certainly weren't at the time. Understanding economics and its effects on attacker incentives, or like you said, moral philosophy and how we navigate our careers and things like that. Yeah, it's very interesting. Or finance to really understand the business impact of some of the vulnerabilities that we find.

Jon McLachlan

Exactly. Sometimes something can feel very scary from a security point of view. But then when you look at it from a business point of view, you're like, well, it doesn't impact our core business. And so is that really worthwhile to spend all of these resources to get us to that 100% mark, or is that an accepted risk for the business?

Will Butler

Yeah. And similarly, the incentives that attackers face.

Jon McLachlan

Exactly. So some of the attackers, they're making the same economic choices as the businesses. That's a really interesting perspective. I've never flipped that before.

Will Butler

Yeah, something I love doing is listening to stories about leaks from various threat actors, like their chat logs or something, and learn how much they actually function like your average developer team. They have SREs and they have SWEs and they have PMs and it's very [...] they have PMs? What sort of things do they PM? [unclear] Yeah, I guess. I, you know what? I admittedly haven't looked into this a ton, but I listen to the [Risky Business] podcast a lot. They're constantly talking about how similar some of these crews are to your average security or software engineering team. And it's fascinating.

Jon McLachlan

I imagine. That's a really interesting perspective. I hadn't thought of them as full-functioning teams before. Sometimes it's just easier to think of them as that nebulous, faceless adversary out there, but they're not. They're human beings, just like the rest of us. And they operate, I'm sure, pretty well in small teams of high trust, and they move quickly together, just like everyone else.

Will Butler

Yeah, and interestingly, that can make attribution difficult too. One crew might be like the malware developer for several other crews that actually run a campaign. Or there might be one that tends to get initial access and then another one that uses that initial access. And so it can make attribution kind of challenging.

Jon McLachlan

Yeah, I could imagine. So if you think about your entire career as a red team member and as a leader, and even as a team builder at Cruise, would you care to pick out like a very best day that you've ever had through your entire journey?

Will Butler

Oh man. So I have a lot of good days, to be honest with you. I really enjoy my job. Some of the ones that come to mind, first, I really love days when I, as a red teamer, can influence significant amounts of change in an organization. And I think that there have been several at recent companies that I'm going to combine almost into one story, so I'm not revealing too much about any of them. But basically, sometimes we'll do assessments and we give readouts and they don't go anywhere because our findings are really inconvenient. We tend to find vulnerabilities at the boundaries between different systems and abstractions, and things that are fundamentally insecure by design. And so they're not something you can just hand a team and say, go change this configuration setting, or patch this software, or encode this output as it goes out onto an HTML page in your browser. What we find tends to be, hey, your CI system is insecure by design, and you need to make some major changes to it that are likely to break many other teams' workflows. And so [...] it's pretty easy, I think, to get discouraged when you have findings like that and companies don't want to fix them. But occasionally, we've been a part of assessments that have really influenced change significantly, where we will be impetus behind these very large scale tiger teams that can actually make significant breaking changes across the organization. And I think that we've convinced organizations that it's worth investing the time to fix these things properly. And seeing that is extremely exciting and motivating, and you feel like your work has a lot of value. So those are some of the best days that I've had, is seeing stuff really fixed, like breaking changes rolled out, core infrastructure changed, and people really being happy with the attack surface reduction as a result of your work.

Jon McLachlan

No, that's something really meaningful, like just totally fulfilling to be part of something bigger and larger than just our day-to-day grind.

Will Butler

Yeah. And then, of course, the other best days are when you find cool bugs. There's just, there's no feeling quite like that. It's, it's a real rush. One of the best days that I can remember is when I found this bug. It's probably my favorite bug. So if you don't mind, I'll give you a really quick [unclear] about it.

Jon McLachlan

I will, I will love to hear about your favorite bug.

Will Butler

Okay, so this is my favorite bug that I found. Not my favorite bug ever. I'm definitely, other people have found way better bugs than this, but this is

Jon McLachlan

your favorite bug that you have discovered.

Will Butler

This is the favorite bug that I've discovered. So one time I was registering new phishing domains for a red team exercise with a client. And I noticed that approximately every hour on the hour after we registered these domains, we would get a very specific set of HTTP requests from some odd user agents. So like outdated Chrome, like way outdated Chrome, or like an outdated Python requests user agent or something like that. This was actually before the engagement started. So we reached out and tried to figure out what this was at the client. And it turns out they had a very cool piece of software that would [ingest] domains, try to figure out which ones were sufficiently similar to domains that the company owned, and then see if the pages were sufficiently similar to any of the company's pages. And if so, they could get ahead of it, which I thought was awesome. I'm like, that is a really cool piece of technology to have. And I was thinking, though, if you're rendering attacker-controlled JavaScript, that's an interesting bit of attack surface for me. I'm wondering if there's anything I could do with that. So we ended up sending back a payload that used the WebRTC API to get the local IP addresses of network interfaces on this instance running the headless browser. And we then pushed another bit of JavaScript that did a little bit of screenshotting of various HTTP services that were running on that local /24. And we found a very interesting application. It was a simple dashboard application that was just displaying data from a database. It did, it appeared static, didn't really have much functionality other than you could just click a couple buttons, and it would show different stats. I looked into it because it turned out it was a vendor product, and I looked into this a little more. And the way that it works is hilarious. So the query string of the URL was just the name of the database table that it was showing us statistics about. Like it wasn't a query parameter, it was literally just the name of the table. And the way it worked is it literally, it evaluates the string in an esoteric programming language. I won't get into language because it might tell you too much about this, but it's a very weird esoteric programming language. And it functionally is just [unclear] query string. Yeah. Even worse. Even I love [Lisp], but way more esoteric than Lisp. And so it's calling eval on your query string. So this is perfect. It's literally an RCE that's triggerable from a simple GET request. So I can use this JavaScript that I'm writing in the headless browser to trigger the RCE. So I'm super pumped about this at this point. And I look into this language. I learn it enough that I can write a payload, and I get code running on this host. But it turns out it's not that useful, because the host that I'm running on has very little access to anything. It doesn't have a whole lot of network access. It has very low [privilege] Kubernetes service [account]. It can't talk to the AWS metadata API. It's connecting to some read replica of a non-prod database that nobody cares about, has no interesting data in it. So it's not like even that database connection is really valuable. So I was thinking, oh man, was all that just to compromise this silly little unprivileged host? But then I started looking more into this product. And it turns out that the way this product does async is very odd. It allows a database client to basically specify the name of a stored procedure that the server can call on the client when the results are ready. And so I was thinking, oh, that's fascinating. So can I just call arbitrary code on this client? Not quite. There's actually a sandbox that this technology uses to make sure that you can't just call arbitrary code on the client. But I found a way to break out of this sandbox. And so now you actually could execute arbitrary code from the server to the client. So I looked at what clients were connected to this database. Turns out, like every engineer working on this database in production is connected to the test database. So I was able to exploit their laptops with this sandbox breakout. And then a few iterations later, had compromised most hosts in this network at the client. And it was all just from this JavaScript payload in a headless browser.

Jon McLachlan

Wow, so that's incredible. It was so deep. It's like tugging on the little thread at the end of the sweater, and the whole thing came unraveling before your very eyes. So how long did that take to go from like when you first noticed to all the way to all of those compromised hosts?

Will Butler

It was on the order of a few days. Now, to be honest with you, we did cheat a bit. At this point, we were already talking to the client and we were fairly engaged with their security team. And so we did just ask them what this technology was. I didn't actually spend the time to like fingerprint it and [reverse] and figure out what it was. We just asked. It took two days instead of three days. Yeah. So it was on the order of a few days, but we did have a little bit of help from the client's team.

Jon McLachlan

No, that's very [...] just finding those little pieces. I think of red teaming as, it's almost more of an art than a science, because you have to have these intuitions, and you notice something kind of flicker at the peripheral vision of that just normal engineer, just like, oh, don't worry about that, that's just an edge case. But like, you go down, you go all the way down those edge case paths. You just want to see what's there.

Will Butler

I completely agree with you. Yeah, I think that the old Paul Graham essays about hackers and painters come to mind there, where he's talking about how hackers tend to be more like painters than they are like engineers. When he says hackers, he means makers, creators, and not necessarily security people. But I actually think a lot of his arguments can extend to offensive security engineers. There's a lot of element of creativity. And there's almost a weird amount of lack of formal understanding of reasoning about exploits and lateral movement [unclear] work that's still to be done.

Jon McLachlan

That's part of your strength, because you don't bring with you all of these preconceptions about how a system's working. You have to discover it for yourself. And so when you go down that road of discovery, you are actually looking at the real bits and pieces that are not what they should be, but what they are. And there's difference. And that's what you exploit, right?

Will Butler

That is such a good point. And I actually want to talk about that point for a second. So that's something that I really try to impress upon people, is that one of the bits of value that a red team can provide is that security teams, and engineering teams too, they often talk about their systems the way that they think they should be, or the way that the documentation says it should be, or the way someone told them it should be. But they haven't actually gone through and operated on those systems. And so there's a lot of like little subtle differences between what the documentation says or what somebody tells you and what's actually going on. And one of the value that red teams provide is understanding that ground truth and giving your teams a tether to reality, almost. And I have a good story about this. One of the places that I was working, I happened to be good friends with their head of infrastructure. And we were out having beers one night, and I was going over an idea that I had for an attack chain. And about two thirds of the way through, he stops me and says, I actually don't think that's going to work, because at that point, you'll have to SSH into this box, and we have SSH CA configured in all those boxes. And in order to get a certificate, you have to have second factor [auth]. And you, at that point in the attack chain, wouldn't have any access to somebody's MFA. And I thought about that for a second and was like, okay, yeah, I guess that would be a big obstacle. But like any diligent red teamer, I'm going to try anyway. Of course. So I try anyway, and it works. And it's not because he was wrong. SSH CA was actually deployed on most systems, but it turns out that there were a few that it wasn't deployed on, because some systems the team didn't know about, so they didn't deploy SSH CA over there, and it was still accepted passwords and keys. Other systems, they deployed it, but it broke stuff. There was some automation that was using it. And so they rolled it back, and then they never redeployed it. And then there was a third set of systems that did accept SSH CA, but also accepted other forms of SSH authentication. And so [...] I think that kind of real ground truth understanding is very valuable. And there aren't a whole lot of teams that are paid to go around and find that kind of stuff.

Jon McLachlan

I would call that operational rigor. Like you are so well grounded into what's actually happening that you're just forced to see all of the exception cases. And it just turns out that those exceptions lead to some pretty interesting security situations.

Will Butler

Yeah, absolutely. I think it's definitely that kind of stuff that is going to get you popped.

Jon McLachlan

So if you think of your entire career as a red teamer again, but this time, instead of thinking through your best days, could you share with us maybe any of your worst days?

Will Butler

[unclear] Good question. So one type of [worst thing] is when you cause production issues. So red teams inherently have to do some risky things. You're generally manipulating systems that maybe you don't understand as thoroughly as you should. And sometimes those systems are serving production traffic. And as much as we try to have lots of processes and technology in place to mitigate risk, occasionally something will slip through and it'll cause an outage. And so sometimes I've been on teams where this kind of thing has happened. Luckily, our outages have been very well controlled, because as an internal red team, we tend to work very closely with the other engineering teams. And so we're able to get on top of these issues when they come up. But for example, remember that database story that I was telling you earlier?

Jon McLachlan

Oh, yeah.

Will Butler

So there was a weird situation that happened with that, where the runtime with this database was single-threaded. And they, it's like the JavaScript runtimes, except instead of being asynchronous by default, it's actually synchronous by default. And you can specify that it's asynchronous if you want. But so the script that I wrote to enumerate all of the connected clients and then exploit them was synchronous. And because it was running, that means that nothing else could actually make progress at that time. And so now, luckily, as I said before, this was a test system. It wasn't serving production traffic. But those are the kinds of subtle gotchas that you can run into, where I didn't realize, but I was actually preventing all other traffic on that system from making progress, because I didn't quite as thoroughly understand how the runtime worked as I thought I did. And you can run into these situations a lot. So we do, we do actually go to great lengths to mitigate risk. I'm probably one of the most cautious red teamers you'll talk to, but I really try to think about that a lot and make sure we thoroughly understand what we're doing before we do it.

Jon McLachlan

No, that sounds like a very responsible approach to red teaming.

Will Butler

But we, boy, I'll tell you, it gets your blood pressure up sometimes. You know, you're exploiting something and all of a sudden the connection starts hanging, or weird stuff, or sev pops up and stuff.

Jon McLachlan

I could imagine. I guess the denial of services is just too easy to pull off and call, like, a red team success. Especially when you're in a production system.

Will Butler

Yeah.

Jon McLachlan

I could imagine.

Will Butler

Yeah. Absolutely. And we've worked with some really fragile systems where, you're absolutely right, they were under such heavy load or in such odd configurations that it didn't take a whole lot to cause problems. But yeah, we really try to make sure that we limit the risk as much as we can. Even if that means, by the way, taking another path. I can't tell you how often I've had my teams do that, where we have a promising looking exploit or a promising lateral movement opportunity, and I say, no, let's look for something else, even if that means backtracking a bit, because that's too risky.

Jon McLachlan

No, that's really interesting. And this gets into the operational, like day-to-day grind of a red team. Do you guys often work together on the same project, on the same system? And everyone takes a piece and goes off and investigates and recons and then regroups?

Will Butler

Yeah, that's a really good question. I've seen it work different ways, but the way that I like to do it generally is we will meet and [...] so we'll meet initially, and we'll come up with our list of kind of research tasks, like the initial recon, for example. Then we'll split up and have each of us some recon tasks assigned, and then we'll work through them. And when we think we are at a point where we can make enough [transitions] that we're ready to actually run them, we actually get together and do that stuff live together. So if we're remote, we'll all hop in a Zoom, or if we're in person, we'll all go into a conference room, and we'll do that together. Then there's also points at which research overlaps, or somebody's expertise is required for research that another operator is doing. And a lot of times they'll collaborate independently of the rest of the team, seek advice as necessary. But yeah, that's usually how it works, is we get together and we execute. Then once we're at a good stopping point and we don't know what to do next, we divide up research tasks and then work independently. Repeat, repeat, repeat until the objective's accomplished.

Jon McLachlan

Nice. Yeah, I know that it sounds like good feedback loops and a good amount of independent time versus also working together so that everyone's differences can be brought to the table and shared. And I'm sure that creativity, like in an open and welcoming environment, can really get going in those small groups.

Will Butler

Yeah, absolutely. And even when we're working independently on our own tasks, like we're usually posting updates in Slack and folks are collaborating there. And so there's still plenty of collaboration that happens, but we're usually exploring our own tasks mostly independently, doing things like vuln research or recon or things like that.

Jon McLachlan

Yeah. And the other thing that you mentioned too, that I'm sure maybe some of our listeners caught, is that you're describing this from the perspective of an internal red team, which is really distinct from an external red team, [who are] sometimes referred to as contracted pen testers.

Will Butler

Yes.

Jon McLachlan

And so what are the differences that you get? What are the advantages and, I guess, disadvantages too of being an internal red team versus just a one-time shot external pen tester?

Will Butler

That is a great question. There are a lot of differences, and I think it affects how you use the team. And I think a lot of people don't understand the differences, and so they will try to manage one as if it's the other and then be disappointed in the results. Then I feel like if you just tweaked your use of the team slightly, you could have gotten infinitely better results. So I'll delve into a few of the differences and how I think they affect how you use the team. So the first difference is that an external red team is going to be much closer to the real adversary in terms of their knowledge of your environment. And specifically, they're going to be facing similar constraints that a real adversary will. They have a time box. They have very little knowledge about what your company is like from a security perspective. They don't know what controls are in place. They don't know what projects are currently in the works. They don't know what technology you're using. And they have to figure all that stuff out themselves. So an internal red team, they're generally just going to notice stuff. And I know some internal red teams try to be more like an independent red team, but I think that is not probably the best way to use an internal red team. I think an internal red team should actually be fairly well integrated with the rest of the security team. They should work very closely with everybody else. They should get an idea of what the capabilities of the [D&R] team are, for example. They should get an idea of where the common vulnerabilities are. Now, a disadvantage of that [...]

Jon McLachlan

[unclear] detection and response, just for our listeners.

Will Butler

Exactly. Yep. Now, a disadvantage of being so closely integrated is you are biased. And so you might miss things that an external red team would find, because you're kind of involved in the day-to-day operations, and you might have the same blind spots that everybody else does. But the advantage is that it's much easier for you to influence change, because you understand all of the constraints on these various teams. You understand how things work. You're able to build up credibility and trust with these teams, because you can be involved in the aftermath of your assessment. You can help D&R improve their processes. You can help the SRE team architect the new design of the thing you told them was broken. You can help patch a bug that you found.

Jon McLachlan

[I'd love to see that] you can actually make your future life a little bit more challenging.

Will Butler

Absolutely. As a red team, as an internal red team. Yeah, so I think that's the internal conflict, is how independent should your team be? Personally, I've had the greatest success when my internal red teams were not independent at all. They were very tightly integrated with the rest of the security team. And I think that actually enables them to act almost like security product managers too, which could be a really interesting, unusual role for an internal red team, but one that I've seen work really well. Just as your typical product org has product managers that are supposed to help your team decide what to build, like what features to build, and the engineering team is supposed to decide how to build it. I think security is in a similar position, where your security engineering disciplines, like InfraSec or AppSec, they are the ones, or like Detection Platform, for example, they're the ones who understand how to build things. They're the ones who know how to build scalable log ingestion pipelines. And they'll be the ones who understand how to fix vulnerabilities across your whole code base and design secure by default infrastructure and stuff like that. But they might not necessarily be the ones who know the most about what security controls, if in place, would make a real attacker's life most miserable. And I think that is what red teams and maybe also like threat intel folks are most qualified to opine on. And so I think that red teams can actually be really useful from that perspective and be like pseudo product managers. And that's something that I think is much easier to do when you're an internal red team and very closely integrated with the rest of the security team.

Jon McLachlan

Oh, I imagine. Yeah, for sure. For sure. I guess from my own experiences, one of the things that I've also seen external red teams, I don't know why this falls on more external red teams, but those reports that they produce, sometimes you can ask or even contract out an external-facing, like customer-visible summary of a report. And then they'll come in and they'll do like an initial assessment. You go through a couple of engineering cycles, fix a whole bunch of the criticals and the highs, and then come back with another assessment and show the delta, that things are moving in the right direction. And then, at least in B2B plays, where it's a large business selling to another large business, maybe it's an enterprise deal or a large company, you show these customers, these security teams that the sales teams are bumping into that are asking the tough questions around, hey, is this product really secure? Do you have any internal pen test reports that you can share with us? You can hand them these reports and break down sales barriers that sort of show up in the B2B play.

Will Butler

I'm actually really glad you brought up reports too, because I think that's another really critical difference between an internal and an external red team. So an internal red team has lots of different options for how they add value after the assessment. An external red team usually only has one, which is the report. An internal red team can do everything from submitting a patch for the bug themselves, or writing a rule that they wish existed, or sitting in on an architecture meeting and providing an opinion. And so I think that gives them all sorts of additional opportunities for having a lot of impact. But then also, another thing that I think is so interesting about the internal-external dynamic is that I never understood how useless my own recommendations were as a consultant until I started being on an internal security team. When I went from consulting over to Apple, I was originally in the AppSec team. And so I would get some consulting reports and have to do things with them. And I was just thinking, [...] I don't expect the consultants to understand our internal context, but because they don't, their recommendations tend to be way overly generic and not useful to us. So I remember one of my first days actually at Apple, we got some report and I sat down with the team and I pulled out my typical consulting person explanation for how they should remediate it. And they were like, oh, yeah, we've already thought of that, but there are 16 different weird edge cases and limitations and restrictions that make all of those solutions completely impractical. Now help us. And I'm like, oh no. That made me really significantly up my security [game] quickly, because I had to really sit down and understand the context [and] restrictions these teams are operating under and design them a solution that mitigated enough. And so I think that really taught me a lot about the engineering economics of [unclear] and fixes. And I feel like external teams miss a lot of that stuff.

Jon McLachlan

They do miss that.

Will Butler

Yeah. But I don't fault them for it. They're under constraints, they're time constraints.

Jon McLachlan

Yep.

Will Butler

Yeah, they're under a lot of constraints themselves that make it super difficult to get this kind of context. And so I'm not blaming them at all [for not] having this context.

Jon McLachlan

This is a safe space for not blaming, for just understanding the different positions that folks are put in. Have you ever been in a situation where you've seen assessments coming in from both external red teams and internal, like on the same systems?

Will Butler

On the same systems. I sure have, yeah. And I will say internal teams tend to [...] they tend to find more stuff, but it's because they have an undeniable significant advantage of understanding the tech stack and having a lot of familiarity with the teams and environments and stuff like that. So I've definitely seen both sides of it. But the other thing, though, is the internal team's reports tend to be a lot lighter, because they know that they're probably just going to end up in a meeting with these teams explaining it anyway. And a lot of that's going to go into [vuln] tickets, and a lot of that's going to go into pull requests and things. So I actually find the external team's reports tend to be of significantly higher quality.

Jon McLachlan

Oh, okay. If that's your only venue of communication, I suppose they have to put whatever they got in there. And that's the reputation their name goes on.

Will Butler

Yeah. And another thing too about external teams is that they often are able to hire subject matter experts that it doesn't make any sense for you to hire. So you might not have enough time, or, I'm sorry, you might not have enough headcount for some full-time cryptocurrency expert on your red team. But Trail of Bits does. And they can sell that expert time to all sort of clients who need it. And so they are able to get some really deep subject matter experts looking at a system that your red team, who has to be comprised of more generalists, can't. So I actually think a lot of times it's really useful to have both look at it, because they both have their pros and cons. The external one's less biased. They probably have deeper subject matter experts that they can go rely on for different things. And the internal team has more context and knowledge. So I think that having both look at the same system is actually really valuable. I remember we actually worked with an external red team at a company that I was at, and we both operated on the assessment together. So we had a few of [their] operators and a few of ours, and we went after it together. [unclear]

Jon McLachlan

Oh, that's amazing. Yeah, I could imagine that also an [external] red team would be a good option for folks that know that they're at a point where they need a little bit of help. They want that second pair of eyes on their system that is really bringing that security perspective to the table, but they're just, the company's not big enough or whatever to have an all-out red team in-house already. And that's an interesting question, even for startups in general. If you're in a regulated space, when is the right time to do your first assessment? Yeah. Is that [unclear] when you close your first big customer? I don't know.

Will Butler

That's a really good question, because the economic constraints of startups are really interesting. And so I can't imagine many [...] early startups hiring a red team. But I must say, hiring a red team early, even if it's just a single person, or even if it's a person that doesn't red team full time but has that background, can be really useful for that product management perspective that I was talking about before, and kind of the ground truth perspective. Like having a red teamer help design some of your initial infrastructure and things like that, I think, will pay enormous dividends over time. Because [...] because a lot of times we're invited in, understandably, but when there's so much momentum built up behind the current infrastructure that it's really difficult to change things, like I mentioned before. And if you get a red teamer in who's comfortable being more of a product manager and not red teaming all the time, I think they can provide immensely valuable perspective that will pay for itself a lot later.

Jon McLachlan

Amazing. No, that's really good advice. Great perspective. So speaking of bringing on red team members, do you have a favorite interview question?

Will Butler

Oh man. Okay. So my favorite interview question is actually a hypothetical that I like to give. So if I'm interviewing for an [offsec] position, the question that I always ask that takes up most of the time is just a hypothetical. And it's, say I hire you and your first assignment is to recover [a] private Bitcoin wallet [seed] key or something like that, some like individual file of immense value. Now, obviously, that's not how it works, but like, we may assume it is. Sure. And just a fun start with. Yeah, and you start with no access, and you start with no internal knowledge. And to make it easier, yeah, you can assume that [the key] is just a single file [on] a single EC2 instance. Then I asked them, walk me through how you'd approach this problem to begin with. So it's pretty vague right now, and I'm hoping that they're going to do things to try to make it a little more concrete. So they would say, like, all right, I'm going to start by doing some recon and figuring out things like, do they own any IP space? Do they, what domains do they use for their infrastructure? What subdomains of those domains exist? Where do they point? Is it to some cloud infrastructure? If so, what is it? Things like that. Do they use S3 buckets? Is [their] registrar Route 53, stuff like that? And then I give them assumptions to work with to narrow it down a little bit. So I'll say, all right, assume your dummy recon returned X, or assume that your port scanning returned the following set of [open] ports, stuff like that. And then we go through the hypothetical. And then what I try to do is, as the candidate brings up various technical topics, I like to drill down into those topics as far as I can to see where the candidate's knowledge stops, or my knowledge stops, actually. That happens all the time. And that's great. I love it. It's a spectacular feeling.

Jon McLachlan

Because then you're learning in the interview yourself too, right?

Will Butler

Yes, I love learning in the interview, and it happens all the time. And I feel like either way, regardless of who stops first, I understand where your [minimum] level of depth is. You know what I'm saying? Like, you can go deeper than me, maybe, but where the minimum is. If I can go deeper than you, I still understand where the [...] maximum is. That's very useful information to me. And the reason I like this is because it feels a lot less like technical trivia than just asking them these questions, because the candidate is almost entirely in control of the technical topics that we discuss. I just try to discuss it in a decent amount of depth. So an example of me diving deeper is, I would say, like if a candidate told me that they would look for a specific type of [vuln], like a command injection vulnerability, I would stop them and I would say, okay, explain to me a command injection vulnerability as if I'm the developer who wrote it. How would you do that? And then I would ask them questions like, what's the root cause of that vulnerability? How would I, as a developer, fix that vulnerability? How would you discover this vulnerability in a very large collection of applications with a lot of functionality? Say you were trying to exploit it and the application returned all these different error conditions. How would you try to troubleshoot it and get the exploit working? But let's see that you were, let's say you were trying to exploit it, but you couldn't see the results of your commands. How would you get the results, or how would you try to infer them? What if egress was heavily restricted? How would you infer your, you know, or get your results then? And then how would you fix it, assuming different weird, like, constraints? What if you had to fix it, but the application still had to start a subprocess calling single binary with attacker control arguments? How could you do that safely? Or things like that. And so I try to dig into a lot of depth with each technical topic that [the candidate] brings up. And I feel like that gives me a really good understanding of both their breadth, depth, and areas of focus, based on the stuff that they bring up.

Jon McLachlan

No, that sounds like a really enjoyable interview. And it also sounds like the signal that you're going after, it cuts right to the heart of that creativity that we were talking about earlier, where there's a little bit more of an art to this than a prescribed science. And part of that [is] not being afraid to [do a] depth-first search down [these] little, what do you call, the little glimmers of odd behaviors found within different versions of systems that all mean well. Like this software is all stuff that a human being has written that's well-intentioned. Here's like what we want this thing to do, here's how it works, here's how it's presented to the world. But we're human, and so there are bugs and there's mistakes. And that's okay. To notice them, to notice those little flickers, let's call them flickers, digital flickers across the system, really takes a keen eye and attention to detail and patience with yourself as you dig deeper. And as you were describing that and the signal that you look for inside the interviews, it almost got me thinking, that's really the signal that you're looking for, is that creativity. Man, that's spectacular. There is a book here. Let me look it up real quick. It is called Seeing What Others Don't.

Will Butler

Ah, it's on my list, but I haven't read it.

Jon McLachlan

Oh, this is on your list? I'd highly recommend it. You're going to love it. It is by Gary Klein. I'll just plug it real quick. It's by Gary Klein. It's called Seeing What Others Don't, [the] Remarkable Ways That We Gain Insight. And what this guy does is he takes a huge case study of all of these really great instances of creativity that have bubbled up over the years. And he just looks into the context where the creativity came from, and he organizes it into this really elegant sort of situation where there's three primary ways that folks, [the] context of being creative. One of them is noticing a contradiction. We think the world is X, but really you notice something that indicates [otherwise], and then you dig into why. Like, why is it [not]? Yeah. Why is it not what everyone is saying it is? That's one form of creativity. But there's another one, which really resonates with how I think that interview flows, and that is constrained-based creativity. So when you constrain a system, problems get more and more specific. And with the constraints introduced to a system, like you have to think different. And oftentimes, one of those constraints is the time [vector]. And so one of the situations that he highlights in the book is there's these firefighters, they're fighting a fire, and the winds shift, and all of a sudden the fire is now chasing up after these firefighters out in the middle of nowhere. It's actually out here in California. And they realized, these firefighters realized, that they could not outrun this wildfire that was rushing up a hill, because the winds had changed to work against them. That they were either going to die in this fire, or they were going to somehow have a moment of ingenious insight and creativity to save themselves. And so one of them realized that they're not going to, the leader actually realized, oh, we're not going to make it. We need to protect ourselves. And the best way to protect themselves from this raging fire was to actually start a fire.

Will Butler

Oh, right.

Jon McLachlan

And to protect themselves from the fire, because if it's already burnt, like, it's not going to get burnt again. And so you'd have a little space where you would not get burned. So that's what they did, and they were saved. But it's that constraint of time, and your life is on your mind, and there are certain things at stake, and you got to find a creative path forward. And maybe it's the type of stress that you've experienced yourself in a red team, where there's a time constraint to how long you're willing, or how long you have to look at something, along with like other constraints in the system that are defined by the parameters of the system, that results in some really interesting creative things happening.

Will Butler

Absolutely. Yeah. I have another interview question that I actually love to ask around constraints like that, which is flip the hypothetical on the defender side, and let's take a typical vulnerability that you might see and add a bunch of constraints to it and see how you solve the problem. So the one I like to ask is, let's say that you're given an executable from a vendor, and through some back channel that it's a very insecure executable, has all sorts of vulnerabilities in it. But you have to run it for business reasons. You can't reverse it or change it for legal reasons. And so how do you run it as securely as you can? And the other constraint is it has to accept, maybe not directly, but it has to accept some [attacker-controlled] input across the internet. And so I feel like most security people, they focus a lot on the vulnerabilities and less on the other layers of defense. And so I really like to impose that constraint on you, can't fix this. It's just going to be vulnerable. So now what do you do about that?

Jon McLachlan

I [unclear] this question. So that's super interesting. I bet you hear all sorts of stuff.

Will Butler

Oh, yeah. All sorts of interesting answers to that question. And another thing, speaking of constraints, it's not an interview question, but just something I wanted to bring up, is that we, that's actually something that I learned a lot back at BitMEX, which is, we were a relatively small team. We had really high stakes and real adversaries, and those constraints forced us to be very creative. And I learned so much. I would really recommend, if anybody's looking to significantly up their learning, go to a place like that. Go to a place with high stakes, constraints, and real adversaries, and you will be forced to learn a lot.

Jon McLachlan

Amazing. I love being pushed outside my comfort zone. That's the only place you do any learning or growing that's worthwhile. I think that's one of the best reasons to do a startup, to join a startup, to do a startup, to build something, to push yourself outside your comfort zone, is just, you're going to learn and you're going to grow. If nothing else, you're going to come out the other side, like, much improved. [unclear] This has been spectacular. I've really enjoyed our conversation. Would you like to leave our audience with any words of wisdom? Parting words of wisdom?

Will Butler

Any words of wisdom. Okay. First word of wisdom is red teams can make really interesting PMs. I know we've talked about this a lot, but it's a somewhat unusual role, and I think there should be a lot more of it. So if any of you are interested in experimenting with that, I would highly encourage it. Second word of wisdom is that there are many layers of defenses that you can put in place between an attacker exploiting a vulnerability and an attacker accomplishing their goal. And if you really want to make our lives miserable, exploit every one of those opportunities. The engagement doesn't end with the vulnerability. It ends when they accomplish their goal. So I would encourage folks to focus on that. I think a lot of people tend to over-index on vulnerabilities specifically. And let's see, what else?

Jon McLachlan

If someone were inspired by listening to [...] all of your stories and wanted to dabble in red teaming or explore that as a career opportunity, as a career path perhaps even, what would you recommend that they do?

Will Butler

Yeah, that's a great question. So [...] red teamers are very broad. I think they have to be very broad. They have to be generalists. Now, it's not to say you can't have a specialty, but it is to say that you really need to understand a lot about a lot of different types of systems. And like, examples of things that you might need to understand, everything from social engineering, sending phishing emails, to how the internet works, to how modern software is built, to how to program in different languages, to how to find and exploit application security vulnerabilities, to how IT organizations work in a company and how software makes its way on your laptop. You should really thoroughly understand the defender. Understand what tools they have to monitor their environment. Understand what footprint your activities leave in those tools. Understand that there's a human sitting behind that screen, looking through thousands of alerts, looking for a reason to say that this is a false positive or something. And so I think understanding all of those aspects of red teaming is really important. And a good way to do some of that is to work on a lot of different types of security teams. Go try your hand at D&R, at designing some secure infrastructure, or even an application development. Go try to build something and break it. Also, contributing to open source, I think, is a really great way to get broad experience. Go find projects like security tools that [red teamers use] that you're really interested in. Find out how they work. Fix bugs. Add features that you've always wanted. Those are really good ways to thoroughly understand how the systems that you're attacking, the tools that you use, work. And I think

Jon McLachlan

Go look at TruffleHog. It's open source.

Will Butler

You should definitely go look at TruffleHog. You should go look at TruffleHog.

Jon McLachlan

You should also go look at Peacemaker too. [That's a nice security tool.] All right. Awesome. Hey, it's been an absolute pleasure. This has been a really interesting conversation. Thank you so much. I am filled with gratitude for the time and all the stories and you're sharing your experience.

Will Butler

Hey, I've really enjoyed this too, Jon. Thanks so much for having me.

Jon McLachlan

And thanks for all our listeners out there. And stay tuned next time for another episode of the Security Podcast in Silicon Valley.

FAQ

Questions this episode answers

Who is Will Butler?
On this May 2022 episode, Jon introduced Will Butler as the red team tech lead at Robinhood and a co-founder of Truffle Security, the company behind the open source secret scanner TruffleHog. Before that, Will was a red team consultant at PwC, worked on AppSec for Apple's service products, started the red team at Cruise and was a staff red teamer at BitMEX.
How did Will get into security?
In an airport IT internship, Will found a web application whose session ID was a small number that went up by five with each login. A script that tried thousands of IDs landed on an administrator session in the system that ran the airport's hand readers, door locks and cameras. The contractor patched it and hired Will to review more of its software.
What is the difference between an internal and an external red team?
An external red team is closer to a real adversary, with a time box and little knowledge of your environment. It's less biased, can bring specialists, and its main product is the report. An internal red team knows the environment and the people, which biases it, and it can patch bugs, write detection rules and help redesign systems.
What does it mean for a red team to act as a product manager?
In Will's view, security engineering teams know how to build controls, and red teams know which controls would make a real attacker's life most miserable. A red team integrated with security can steer what gets built next.
What is Will's favorite red team interview question?
A hypothetical: recover one file worth a lot, with no access and no inside knowledge. The candidate starts with recon, Will supplies assumptions, and Will drills into each topic the candidate raises, such as command injection, until someone's knowledge runs out.
When should a startup bring in a red team?
Will expects few early startups to hire one. Even one person with the background, part time, can help design the initial infrastructure and bring ground truth, before momentum makes the infrastructure hard to change.
What is an escape fire?
A fire set on purpose to burn the ground ahead of a wildfire, so there's a burned area to shelter in. Jon told the story from Gary Klein's book Seeing What Others Don't, and Wikipedia's account of the 1949 Mann Gulch fire describes the foreman doing it.
What happened to Truffle Security after the episode?
It raised a $25 million Series B led by Intel Capital and a16z in November 2025, and launched a free Secrets Revocation Program in November 2026, starting with 1.4 million live leaked credentials.
New episodes every other Tuesday

Got a story like this one?

We're always looking for practitioners who've been in the room when it mattered.