Episode 3 of The Security Podcast of Silicon Valley aired on June 3, 2021, with Jon McLachlan hosting. The guest was Daniel Feldman, a cloud security architect at HPE. HPE had acquired Scytale, the startup he joined as an early employee, in February 2020 (Podbean, LinkedIn, SPIFFE, SPIRE on GitHub). The episode is called “Daniel Feldman, Cloud Security Architect at HPE, Service Authentication with SPIFFE and SPIRE.” It has no video version.
Later in the episode, Jon said HSMs are painful to set up, and that no secret at all is even better. Daniel answered with his motto:
“Get rid of your secrets. That’s my motto.”
Daniel Feldman, at 28:19
SPIFFE is short for the Secure Production Identity Framework For Everyone. In the words of the Cloud Native Computing Foundation, it “provides a secure identity to every workload.” SPIRE, the SPIFFE Runtime Environment, is the code that implements the specification on many platforms. Both projects graduated from the foundation on September 20, 2022 (CNCF). Daniel’s LinkedIn lists his HPE role as SPIFFE product technologist, from February 2020 to April 2023 (LinkedIn).
This page follows the conversation in order: a section for each part of it, eighteen figures, nineteen clips from the tape, the full transcript, and five tools: a quiz, a scorecard for whether your services can prove who they are, a calculator for how many identities your fleet issues a day, a timeline, and a to-do list. Where the decodes of the tape disagree on a word, the transcript puts its best guess in brackets, and the page doesn’t quote those words. Daniel says JWT as “jot,” which is how Wikipedia says the name is pronounced (Wikipedia). The transcript writes JWT.
A wild ride
Jon opened by asking how to pronounce the name of Daniel’s former company. Daniel said a search for the word gives a different pronunciation from the one the company used. A scytale is also an ancient cipher tool: a cylinder with a strip of parchment wound around it, used for a transposition cipher, and the Spartans are said to have used it on military campaigns (Wikipedia).
Jon said Scytale.io had recently been acquired by HPE.
“It’s been a wild ride over the last couple of years”
Daniel Feldman, at 0:33
Build the product, get some adoption, get acquired, then get a lot more adoption all of a sudden, Daniel said. HPE announced the deal on February 3, 2020. Its post says Scytale was “founded in 2017 by a group of seasoned engineers from cloud-native enterprises,” names the co-founders as Sunil James, Emiliano Berenbaum and Andrew Jessup, and says HPE is “fully-committed to continuing Scytale’s stewardship and contributions to SPIFFE and SPIRE” (HPE).
The dark joke of KubeCon
Jon said Daniel had recently been speaking at KubeCon Europe. Daniel had done a lightning talk there and a number of talks and workshops. HPE’s developer blog had announced the talk for Tuesday, May 4, 2021, on how to securely bridge cloud native and traditional workloads with SPIRE (HPE Developer).
KubeCon, Daniel explained, is the big conference for cloud native technology: Kubernetes and everything around it, like Istio, Prometheus, Jaeger and Thanos. Every cloud provider presents. A thousand startups show off their ideas.
“Of course, most of them won’t exist next year. That’s the dark joke of KubeCon.”
Daniel Feldman, at 1:21
KubeCon runs twice a year, and Daniel thinks he had taken part in every one for the last three years, six in all, presenting, staffing the booth or attending. Kubernetes was originally designed by Google, and the Cloud Native Computing Foundation holds the trademark (Wikipedia).
Who are you?
Jon set out the question for the episode: what is service identity management and authentication, and where is it going? User identity answers the question “who are you?” for people, with logins and everything people do to prove who they are. Services are harder:
“service identity, which is a completely different beast, because you can’t ask a service really to just save a password.”
Jon McLachlan, at 2:16
Every engineer who has built or worked with a microservice architecture has struggled with it, Jon said, and he asked Daniel what service identity is, cut right down.
Operation Aurora
Daniel started with history, from research he had done for his talks at KubeCon and other conferences. Around 2010, he said, attackers captured enormous amounts of data from Google’s data centers.
“That was really the genesis of all this technology.”
Daniel Feldman, at 2:53
The attack is called Operation Aurora. Google disclosed it in a blog post on January 12, 2010, and the attacks ran from mid-2009 through December 2009. Wikipedia attributes them to advanced persistent threats such as the Elderwood Group, based in Beijing (Wikipedia).
Google’s answer, Daniel said, was a service identity technology for every service at Google, called LOAS. He said he has never worked at Google. We found no public Google page that names LOAS. Google’s documentation does describe ALTS, its transport security for calls between Google’s machines: ALTS “began development at Google in 2007” and authenticates “primarily by identity rather than host” (Google Cloud).
The idea, as Daniel described it, has three parts. Give every service a unique identity. Trace that identity back to the cluster, the hardware and the data center it runs on. Manage the identities centrally and grant one to each service.
“So the idea that they came up with was to uniquely identify every service in their infrastructure”
Daniel Feldman, at 3:48
A certificate or a token
The identity a service gets is one of two things, Daniel said: an X.509 certificate or a JWT. With a certificate, any connection the service makes can be encrypted with mutual TLS. A JWT can carry extra metadata about the service’s request, and it can travel with a second JWT for the user.
“and then any request can have its authentication and authorization checked by all the other services in the microservices architecture.”
Daniel Feldman, at 4:38
X.509 is the ITU standard for the format of public key certificates, the kind TLS uses (Wikipedia). Mutual authentication means both parties prove who they are to each other at once (Wikipedia). JWT, JSON Web Token, is RFC 7519 from May 2015 (Wikipedia).
Alumni of the big companies
Companies like Google, Netflix and Facebook may run tens of thousands of services, Daniel said, and every request has to be authenticated and traceable to the service and the user it came from. They have the budgets and the people to build that themselves, and they did: each of those companies has teams working on service identity, he said.
Around 2017, when Scytale was founded, several alumni of those companies decided to build an open source service identity layer. That was the start of SPIFFE and SPIRE.
“So the initial document for SPIFFE was actually written by Joe Beda from Google”
Daniel Feldman, at 5:32
Daniel said Beda’s white paper is still on the website and dates back to 2017. An introduction on DevOps.com puts it a year earlier: “Joe Beda, a co-founder of Kubernetes, first proposed SPIFFE at GlueCon in 2016” (DevOps.com).
One API for every service
SPIFFE is an open standard, Daniel said. It gives any service an API to fetch its own identity, plus trust bundles to check the identity of other services. You get both an X.509 certificate and a JWT.
“I mean an X.509 certificate that lets you establish mutual TLS authentication, or a JWT token.”
Daniel Feldman, at 6:08
The standard exists so you don’t write each service to get its identity a dozen different ways, he said. One API, and each service implements it as a client. The plan was a whole bunch of implementations.
“What ended up happening is there’s the SPIRE implementation, which is the only real complete implementation.”
Daniel Feldman, at 6:43
Other projects take SPIRE’s open source code and build it into their own software, he said, and more were doing that as they spoke.
The parts that got adopted
Jon asked whether it was a shock that nobody else picked up the standard and wrote their own implementation. Daniel said the hard part is the workload API.
“The other parts of SPIFFE have actually been widely adopted.”
Daniel Feldman, at 7:30
Almost every service mesh uses SPIFFE-compatible identities, he said. They aren’t real SPIFFE implementations, because they don’t provide the API. In Istio, for example, a connection gets a mutual TLS tunnel with a SPIFFE identity on both ends, created for you.
“but it never provides the workload API so you can access your own identity.”
Daniel Feldman, at 7:58
Jon summed it up: “Very specific pieces of it got adopted.” Daniel agreed. Getting real adoption of the workload API, so a service can fetch its own identity and use it with other services directly, was what his team was working on. At the time, the only way to get it was to install SPIRE yourself.
Super open source
Jon said he loved that SPIRE, the full implementation, is also open source. Daniel went further: the code is owned by the Cloud Native Computing Foundation.
“So it’s super open source.”
Daniel Feldman, at 8:45
Anyone can contribute, he said, and anyone can join the meetings. Anyone can join the steering committee by being voted on, and anyone can have an official role in the project. The foundation is a subsidiary of the Linux Foundation, founded in 2015 (Wikipedia). SPIFFE and SPIRE entered its Sandbox in 2018 and moved to its Incubator in 2020 (CNCF).
How SPIRE knows a node
Jon asked how SPIRE identifies a service, and Daniel walked through it. A SPIRE server runs centrally and holds all the signing keys, for JWTs and for X.509 certificates. A SPIRE agent runs on each node: each Kubernetes node, virtual machine or bare metal box. The server keeps generating certificates and JWTs and pushes the right ones to the right agent.
A new agent has to log into the server and prove its identity first. In the cloud, it can use an instance identity document, a document signed by the cloud provider, from Amazon, Google or Azure.
“if you’re running in the cloud, you have an instance identity document.”
Daniel Feldman, at 9:51
The agent downloads the document on the node and hands it to the server. The server verifies the signature, checks which identities that agent may get, and passes them down.
How SPIRE knows a workload
Services on the node connect to the SPIRE agent over a Unix domain socket, which Daniel called a local network socket. Wikipedia describes it as an endpoint for exchanging data between processes on the same operating system (Wikipedia).
The agent then checks who is asking. If a certain identity belongs to services running in a particular Kubernetes service account, the agent verifies the process ID that opened the connection, asks Kubernetes whether that process has the service account, and pushes the right identity to it.
“So every 30 minutes or hour or two hours, we’re regenerating these certificates”
Daniel Feldman, at 11:01
Policies can change at any time, he said, and the identities keep refreshing.
“And that’s why you need an API, so it can be continuously pushed instead of having just a file that’s deployed alongside the service.”
Daniel Feldman, at 11:10
Jon called it an identity layer built on the infrastructure the cloud providers already give everyone, which is tricky to use well for identity on its own.
If everything runs in AWS
Daniel started with when you don’t need SPIRE.
“you probably don’t need this, because you already have those instance identities and you can just use them.”
Daniel Feldman, at 11:46
That holds if everything runs on AWS instances and nothing runs in Kubernetes, he said. What SPIRE adds is one identity layer that works in Kubernetes and on AWS, GCP and Azure instances and on-prem, with every identity generated by the same server and compatible with the others. Large organizations have heterogeneous infrastructure, and they want a service in Kubernetes on AWS to be able to talk to a database on an on-prem server.
Jon put the benefit in his own words. Move the service authentication layer away from the cloud provider, and you can run the same service across providers without re-implementing authentication that was tightly coupled to one of them.
An identity plane, even in one cloud
Single-cloud companies need it too, Daniel said. A lot of smaller companies standardize on one cloud, but they run in several AWS regions, and each region is effectively a different cloud that shares nothing with the others. They still have an on-prem resource now and then. They have workloads in Kubernetes and workloads outside it, like an RDS database or another managed SaaS service that can’t run inside Kubernetes.
“Even if you’re all in one cloud, you still need some kind of identity plane for everything to talk to each other.”
Daniel Feldman, at 13:23
SPIFFE’s site uses the same idea today: SPIFFE and SPIRE “provide a uniform identity control plane across modern and heterogeneous infrastructure” (SPIFFE).
Zero trust is a buzzword
Jon called zero trust the cool new thing in the security world, and asked whether SPIRE is zero trust.
“Zero trust is a buzzword. It’s a marketing term.”
Daniel Feldman, at 13:41
The original idea, he said, was a network with no perimeter, where every service and every user has a unique identity and every connection and request is verified against it. Then marketing took the term. VPN companies call their products zero trust. So do monitoring companies that check which services connect to which. One company claims zero data security, a new term to him: it stores vital data like credit card and social security numbers somewhere else, and calls itself zero trust. Like cloud and SaaS, the term blew up, and because it isn’t trademarked, everyone uses it to mean slightly different things.
“So as a consequence, I just try not to use the term zero trust.”
Daniel Feldman, at 14:55
Wikipedia defines zero trust architecture as a design where users and devices aren’t trusted by default, even on a privileged network, and says analyst John Kindervag of Forrester used the term zero trust model in 2010 (Wikipedia). On episode 42, Avery Pennarun says Tailscale is a zero trust product, though its homepage probably says nothing about zero trust.
Trusting your cloud provider
Jon tried a better question: do you have to trust your underlying cloud provider when you use SPIRE?
“You do have to trust whoever is running the SPIRE server.”
Daniel Feldman, at 15:10
There’s no way around it, Daniel said, since whoever runs it has your private keys and can generate the identities. SPIRE does help you spread trust. If you trust AWS and want to give Google Cloud a little trust, you can run a SPIRE agent on Google Cloud that talks to the SPIRE server on AWS. Or you can federate two SPIRE servers, one on each cloud, so they don’t have to trust each other completely.
“So it is one step towards not trusting your cloud provider.”
Daniel Feldman, at 15:40
He was looking forward to enclave technologies that let you run code on a cloud provider’s machines and isolate it from the provider, SGX among them, and called them a new frontier in security. Intel’s Software Guard Extensions let code define protected private regions of memory, called enclaves (Wikipedia).
Zero-days in a backup product
Jon asked how Daniel got involved. Before Scytale he worked at Veritas, which makes backup software. He called it the number one enterprise backup product, used by tens of thousands of companies.
Then a number of zero-day attacks were found against the product. Some were things the team already knew about, Daniel said. There were ways to convince the backup service to provide files it shouldn’t, and the backup server to let you restore files you shouldn’t. They had always trusted that customers would deploy the product inside a secure perimeter, where companies knew their users.
“Defense in depth,” Jon said. Daniel agreed. A security research team publicized the attacks:
“and suddenly all our customers demanded zero trust security for our backup product.”
Daniel Feldman, at 17:10
“There’s that buzzword again,” Jon said. Veritas built the first commercial journaling file system, VxFS. It merged with Symantec in 2004, was owned by the Carlyle Group from 2016 to 2024, and has been owned by Cohesity since 2024 (Wikipedia). Daniel’s LinkedIn has him as a principal software engineer in the CTO office there from January 2015 to May 2018.
Mandatory mutual TLS
The company put security first, Daniel said, and for several years it focused on security, ahead of other features, with hundreds of people working on it. As a principal engineer in the CTO office, he took on parts of the security strategy: certificate management and distribution, certificate revocation lists, and mutual TLS connections between services.
Each customer site ran about 50 services, depending on configuration. The product was old, with many protocols, many programming languages, and layers of technology going back probably two decades. They finished the project, and Veritas could say its backup product was secure even when it runs in an untrusted environment.
“That’s an incredible achievement,” Jon said. It took years of work for hundreds of people, Daniel said, and he guessed hundreds of millions of dollars in engineering. Then they shipped it.
“Tens of thousands of companies got that update, and it mostly went okay.”
Daniel Feldman, at 18:41
Some of those companies had hundreds of thousands of backup nodes. The update deployed certificates to all of them, and communication carried on: mandatory mutual TLS, with identities tied to the specific installation. When the project wound down, Daniel thought the technology had to have uses beyond one company and one project. That’s when he found Scytale and the problem it was working on: services in cloud native infrastructure with no secure identities and no way to establish trust between each other.
Employee three or four
“I was an early employee. I wasn’t one of the founders.”
Daniel Feldman, at 19:42
Jon guessed employee number two or three. Probably three or four, Daniel said. Even then, Scytale had a solid strategy and a good proof of concept to build on. Over the next couple of years, a number of the hyperscalers latched onto the technology and put whole teams on it. Jon asked which ones.
“I don’t think I can say that.”
Daniel Feldman, at 20:15
The marketing people had told him not to name customers. His hint was the list of conference presentations about SPIFFE and SPIRE on the website: look up who the speakers work for. Today the “Used By” section on spiffe.io shows 23 logos, among them Amazon, Google, Netflix, Uber and HPE (SPIFFE). His LinkedIn has him at Scytale.io from May 2018 to February 2020.
The week before the pandemic
“just actually the week before the pandemic, we got acquired by HPE”
Daniel Feldman, at 20:37
HPE brought a worldwide marketing and sales presence, Daniel said. It could show the technology to all kinds of companies that weren’t hyperscalers and weren’t cloud native yet. He had been consulting with those companies and adding to the product: Helm charts, installers, YAML files, white papers and documentation, to get adoption at more old-fashioned companies.
Jon said it sounded like Daniel had been instrumental in making it easy to consume. Daniel called a tiny startup with new technology plus a big company with marketing, distribution, sales channels and vendor relationships around the world a powerful combination. A big company struggles to develop and release new products internally, he said, through no fault of the company:
“if you release something, it needs to be nearly perfect. You can’t just experiment.”
Daniel Feldman, at 22:07
A tiny startup struggles with marketing and distribution. Every conference costs a lot of money, and so does every flight to a customer.
From idea to adoption
Jon said Daniel’s experience at Veritas led him to Scytale, and to the nuances of service identity and machine identity at a low level. Daniel said what he likes most isn’t service identity, or even security. It’s seeing the whole process: the idea, the prototype, the momentum from marketing, conference talks, white papers and discussions, and finally adoption.
“So I’ve been able to see it end to end.”
Daniel Feldman, at 23:09
He hoped to repeat it with more security products in the zero trust cloud native space.
OpenID Connect
Jon said a lot of successful movements rest on open source and open standards, and named OpenID Connect, the standard most likely behind “log in with Google” on other sites. He asked whether SPIFFE and OpenID Connect might merge.
“OpenID Connect is an amazing technology, and it meshes really well with SPIFFE.”
Daniel Feldman, at 24:29
OpenID Connect is a standard for creating JWTs so one service can authenticate to another, Daniel said. His example was AWS. You create an OpenID Connect provider for your company that serves something like a root certificate bundle over a secure HTTP endpoint. AWS keeps checking that endpoint, and a request to your AWS account with a JWT signed against that bundle is authenticated. You still need something to create the bundles and the JWTs. SPIRE already supports JWTs and serving OpenID Connect endpoints. That wasn’t SPIRE’s original purpose, he said.
“it’s a perfect application for SPIRE”
Daniel Feldman, at 25:39
The OpenID Foundation published OpenID Connect in February 2014 as an authentication layer on top of OAuth 2.0 (Wikipedia). The problem it solves here is public APIs, Daniel said, and the secret keys people use for them.
Who hasn’t accidentally checked an AWS secret key into a Git repo, he asked, public or private, or left one on a server the wrong people could reach?
“They’re probably the biggest barrier to cloud adoption.”
Daniel Feldman, at 26:12
Identities that SPIRE generates and serves over OpenID Connect can’t easily be stolen, he said, because any service using one has to have passed node attestation and workload attestation.
Static secrets
Jon wondered whether secret management might soon be a thing of the past. Daniel said he hoped for no more secrets. One competitor in the space is called Secretless, which he thinks is a great name, and he wishes he could use the term.
“because what we are trying to do here is get rid of static secrets.”
Daniel Feldman, at 27:12
No secrets, Jon said, means nothing to steal for an adversary who breaks into a cloud system. Daniel said the secret step holds back cloud adoption. Access tokens have to be managed securely for each cloud and each region, and a lot of companies can’t do that securely. A secret manager helps, but someone still has to set it up, manage it and grant access to it. That has held back regulated industries, security-conscious ones and the federal government, he said. Jon added that HSMs are painful to set up.
“secrets are like toxic waste. They just contaminate any infrastructure they touch.”
Daniel Feldman, at 28:29
On episode 92, Jasson Casey of Beyond Identity says the trouble with a secret is that it moves, and short lifetimes alone don’t fix that. SPIRE’s identities are short-lived and also tied to attestation of the node and the workload. We wrote up the inventory side in Who Holds Your Agent’s Keys? and The .env File Is Your First Security Program.
Can we use blockchain?
Jon asked about decentralized approaches, with cryptocurrencies pushing that way. Daniel said it’s a question he gets all the time: can a distributed ledger validate the secrets as they’re generated? The problem is volume. Customers run thousands or tens of thousands of nodes generating secrets continuously, and no distributed ledger yet keeps up reliably. If the ledger fails to update, communication starts to fail.
“There is one baby step towards decentralized service identity, and this is the same thing that’s used on the public internet, which is certificate transparency.”
Daniel Feldman, at 30:15
As secrets are generated, they’re added to a Merkle tree, a hashed ledger of all the secrets generated, and each one carries a pointer into the tree. A couple of servers host the log, a couple of clients validate it, and you get a little decentralization. Every website certificate already goes into a certificate transparency log, so the performance is there. His team was working on it for SPIFFE. You still trust the logs, he said, but you trust that at least one hasn’t been compromised, where today you trust one source.
Certificate transparency is “an Internet security standard for monitoring and auditing the issuance of digital certificates.” Each log entry references the hash of an earlier one, which forms a Merkle tree (Wikipedia). The calculator below shows how many identities a fleet issues, the number a ledger would have to keep up with.
Operating in the red zone
Jon said the hard part with enterprises may be the sale. You have to meet SLAs and convince a company it still controls its network. Daniel agreed. His market, he said, is a conservative one: enterprise data center infrastructure. When he worked on backup software, a company might test a release for a year and a half or two years before putting it into production.
“We’re operating in the red zone.”
Daniel Feldman, at 32:42
If a bank goes down even for 10 minutes, it could cost hundreds of millions or billions of dollars. If SPIRE fails to issue an identity it should issue and a service crashes:
“that could bring down your bank, that could bring down your hospital.”
Daniel Feldman, at 32:48
Jon added that decentralized means depending on other people’s computers. Daniel mentioned Helium IoT, which does decentralized identity for IoT devices in a consumer space, where a failure is bad but won’t crash the airplane. The Helium Network was begun by Helium, Inc. in 2013 as a network of LoRa gateway hotspots (Wikipedia). There are a lot of opportunities in this space, Daniel said, and it’s still early days for service identity.
Jetliners in the 1950s
Jon asked where service identity and security are going. Daniel said computer security in general is in its early days.
“If this were a time in aerospace engineering, we’d be in the 1950s”
Daniel Feldman, at 34:15
There were jetliners then, he said, and planes crashing all the time. Some organizations today fly a 737 exactly like the ones you’d see now. Others crash the plane constantly. The first commercial jet airliner, the de Havilland Comet, debuted in 1952. Within a year, three Comets were lost, two of them from metal fatigue in the airframe, which wasn’t fully understood at the time. Rival manufacturers learned from the Comet (Wikipedia).
What Let’s Encrypt got right
The most promising direction, Daniel said, is availability and usability. Let’s Encrypt didn’t invent anything new. It made certificates much easier to get, free where they had cost a few hundred dollars, and instant where they had taken a few weeks, and everyone adopted them. It did need a new protocol, ACME, to get certificates automatically.
“the primary innovation of Let’s Encrypt was usability and availability of the service”
Daniel Feldman, at 35:50
And cost, he added.
Jon said the security technology to solve these problems exists. It has to be put together so it’s easy to consume, understand and use. Daniel said that’s the direction he wanted to work in. SPIFFE and SPIRE exist and are free, so they meet the cost test. They’re hard to set up: you have to figure out how to attest your nodes, how to attest your workloads, and how your workloads will use the workload API.
“the real frontier is going to be in making this automatic and easy to use and instant, and also free.”
Daniel Feldman, at 36:41
Let’s Encrypt launched on April 12, 2016, and is now the world’s largest certificate authority (Wikipedia). The IETF published ACME as RFC 8555 in March 2019 (Wikipedia).
A GUI isn’t the answer
Jon asked whether anything would help with the setup overhead, like a simplified GUI. Daniel said they do have a GUI.
“I don’t think a GUI is the answer.”
Daniel Feldman, at 37:21
Let’s Encrypt succeeded because it took little effort, he said, and did something that had been difficult, really well. The company he would watch is Smallstep, which takes the Let’s Encrypt approach to service identity inside companies and focuses on usability. Smallstep focuses on small and medium businesses, where SPIFFE and SPIRE focus on large ones, and the two answer some security questions differently. He pointed to Smallstep’s documentation and ease of configuration.
Maybe they can be your next guest
Jon gave a shout-out to a friendly competitor. Daniel added that Smallstep has the best content marketing of any security company he has seen, with blog posts near the top of Hacker News and Lobsters and all over Twitter. Jon wondered whether they have a podcast.
“Maybe they can be your next guest.”
Daniel Feldman, at 38:47
Five months later, on November 15, 2021, episode 10 featured Michael Malone, Smallstep’s founder and CEO, on certificates, identity, X.509 and authentication. Smallstep’s site now calls it a device identity platform, with a heading introducing ACME Device Attestation (Smallstep).
On-prem and in person
Daniel thanked Jon, and said he had built SPIFFE and SPIRE along with “a big team of brilliant people at many different companies.” SPIFFE and SPIRE change how large companies give their services identity, he said, across clouds and between cloud and on-prem, with uniformly generated certificates and JWTs every service in the enterprise can use.
“If you’d like to chat with me, I’m always on the SPIFFE Slack channel”
Daniel Feldman, at 40:32
The Slack is open, he said, and you sign up from spiffe.io, which links to it (SPIFFE Slack). Jon spelled SPIFFE for the listeners: S-P-I-F-F-E. Daniel invited anyone at KubeCon US in the fall to grab a beer, at what he hoped would be the first on-prem conference many people had attended in a few years. Jon added that it would be in person, too.
KubeCon + CloudNativeCon North America 2021 ran October 11 to 15 in Los Angeles and online, the first in-person event after three virtual-only ones (CNCF). The foundation’s report counts 3,531 in-person and 19,633 virtual registrants (CNCF).
What happened after the tape
SPIRE 1.0 and a KubeCon keynote. Daniel’s LinkedIn description of his HPE role includes “During my time we released SPIRE 1.0, keynoted Kubecon.” He held the role from February 2020 to April 2023 (LinkedIn).
Zero trust in Minneapolis. At devopsdays Minneapolis 2021, he gave a talk called “Zero Trust: Security Without Boundaries.” The speaker page calls him “a member of the CNCF’s SPIFFE project for zero trust tooling” (devopsdays).
Graduation in September 2022. SPIFFE and SPIRE graduated from the Cloud Native Computing Foundation on September 20, 2022, joining 16 other graduated projects. The announcement names Anthem, GitHub, Netflix, Niantic, Pinterest and Uber as end users, and says SPIRE secures hundreds of thousands of workloads within ByteDance (CNCF).
A free book on SPIFFE. “Solving the Bottom Turtle,” the SPIFFE and SPIRE book, lists Daniel among its twelve authors. The PDF is free on the SPIFFE site (SPIFFE).
Sentima, Qusaic and WIMSE. Daniel co-founded Sentima, Inc. in January 2023 and consulted through his own company, Qusaic, in 2024. Of Sentima, his LinkedIn says “Some work made it into conference talks and IETF standards.” At KubeCon North America on November 13, 2024, he presented a poster as founder of Qusaic on what’s happening with SPIFFE and WIMSE, an IETF effort built on SPIFFE that adds a token format for identity across several hops (KubeCon schedule).
A stealth Fortune 500 company. Since December 2024, his LinkedIn lists him as a principal engineer at a company it calls a stealth Fortune 500 company (LinkedIn).
SPIFFE for AI agents. In February 2026, a NIST NCCoE concept paper on software and AI agent identity named SPIFFE and SPIRE as “one way in which agent workloads could be identified and authenticated” (NIST NCCoE). Google Cloud’s Agent Identity gives each agent a SPIFFE ID and an X.509 certificate that’s valid for 24 hours and renewed for you (Google Cloud). Our post Your AI Agent Said No. Your API Key Said Yes. (October 12, 2026) points back to this episode for it.
SPIFFE inside Flux 2.10. On October 26, 2026, Flux v2.10.0 added SPIFFE mutual TLS between its controllers (GitHub). Daniel shared the release post on LinkedIn.
More on this show. Taher Elgamal, known as the father of SSL, explains why SSL required a certificate from the server and only offered one to the client, on episode 61. Mutual TLS is that second certificate. Eric Olden of Strata Identity names SPIFFE among the standards that exist today for putting AI agents into production, on episode 81.
Did you catch it? A seven-question quiz
Seven questions from the episode. Each answer links the moment in the transcript.
Quiz · seven questions
Did you catch it?
Pick an answer to see whether you caught it.
-
Which attack does Daniel name as the start of service identity at Google?
b. "the attack, which is called Operation Aurora." Transcript, 3:23.
-
What two kinds of identity does SPIFFE give a service?
a. "I mean an X.509 certificate that lets you establish mutual TLS authentication, or a JWT token." Transcript, 6:08.
-
What do service meshes leave out, by Daniel's account?
c. "but it never provides the workload API so you can access your own identity." Transcript, 7:58.
-
How does a new SPIRE agent in the cloud prove itself to the server?
b. "if you're running in the cloud, you have an instance identity document." Transcript, 9:51.
-
How often does SPIRE regenerate certificates, by Daniel's account?
a. "So every 30 minutes or hour or two hours, we're regenerating these certificates" Transcript, 11:01.
-
What does Daniel compare secrets to?
c. "secrets are like toxic waste." Transcript, 28:29.
-
What does Daniel call the main innovation of Let's Encrypt?
b. "the primary innovation of Let's Encrypt was usability and availability of the service" Daniel adds cost. Transcript, 35:50.
Can your services prove who they are?
Eight questions on how your services identify themselves to each other and to your clouds, from the episode. Each tip quotes the tape or paraphrases it, and links its section.
Scorecard · eight questions
Can your services prove who they are?
Answer for your production environment, the services that talk to each other and to your cloud APIs. If two answers fit, pick the lower one.
-
Do you know every static secret your services hold?
- No, we haven't listed them
- Most of them, in a secrets manager
- All of them, each with an owner and a plan to retire it
"secrets are like toxic waste. They just contaminate any infrastructure they touch." See Static secrets.
-
How do your services prove who they are to each other?
- They don't, or they share a key
- A token for each service, set by hand
- Mutual TLS or signed tokens, with a unique identity for each service
"I mean an X.509 certificate that lets you establish mutual TLS authentication, or a JWT token." See One API for every service.
-
How long does a service credential last?
- Until someone rotates it
- Weeks or months
- Hours, and it renews itself
"So every 30 minutes or hour or two hours, we're regenerating these certificates" See How SPIRE knows a workload.
-
How does a workload get its credential?
- A file or environment variable deployed with it
- It fetches one with a long-lived key
- An API checks what the workload is, then hands it an identity
"And that's why you need an API, so it can be continuously pushed instead of having just a file that's deployed alongside the service." See How SPIRE knows a workload.
-
Would a stolen service credential work somewhere else?
- Yes, anywhere
- Only from inside our network
- No, it's tied to node and workload attestation
Daniel says a service using a SPIRE identity has to have passed node attestation and workload attestation. See OpenID Connect.
-
Do your clouds, regions and on-prem systems share one identity plane?
- No, each has its own way
- Some of them do
- Yes, one issuer for all of them
"Even if you're all in one cloud, you still need some kind of identity plane for everything to talk to each other." See An identity plane, even in one cloud.
-
How do your workloads call cloud APIs?
- With long-lived access keys
- With access keys in a secrets manager, rotated
- With short-lived tokens through OpenID Connect federation
Of secret access keys: "They're probably the biggest barrier to cloud adoption." See OpenID Connect.
-
Who can mint identities for your services?
- We don't know
- A few admins, with keys on a server
- One issuer, with protected signing keys and a log of what it issues
"You do have to trust whoever is running the SPIRE server." See Trusting your cloud provider.
Your services prove who they are with secrets they carry around. Start by listing every static secret, with an owner for each, and pick one service to give a real identity.
Some services have identities. Next, shorten credential lifetimes, move cloud API calls to short-lived tokens, and put clouds and on-prem under one issuer.
Services get short-lived identities tied to where they run. Keep the issuer's signing keys protected, log what it issues, and retire the last static secrets.
How many identities a day?
Daniel says SPIRE regenerates certificates and JWTs every 30 minutes, hour or two hours, and that no distributed ledger yet keeps up with that volume reliably. This calculator shows the volume for your fleet.
Calculator
How many identities a day?
Three numbers: how many workloads you run, how often each one gets fresh credentials, and how many credentials it gets each time.
With 2,000 workloads getting 2 credentials every 1.0 hours, your fleet issues 96,000 credentials a day, 4,000 an hour, or about 1.1 a second. Each one is a record a ledger would have to keep up with.
The renewal times come from the tape: every 30 minutes, hour or two hours. The 2 is one X.509 certificate and one JWT. The 2,000 workloads are a placeholder: put in your own count.
From Operation Aurora to agent identity
Identity standards, certificates on the web, SPIFFE and SPIRE, Daniel’s path, and the episodes on this site that pick up the thread. Filter by topic.
Timeline
From Operation Aurora to agent identity
- ALTS begins
Google starts developing Application Layer Transport Security, which authenticates by identity (Google Cloud).
- Operation Aurora
Google discloses the attacks in a blog post. They ran from mid-2009 through December 2009 (Wikipedia, on this page).
- The zero trust model
Analyst John Kindervag of Forrester Research uses the term (Wikipedia).
- OpenID Connect
The OpenID Foundation publishes it, an authentication layer on top of OAuth 2.0 (Wikipedia).
- Veritas
Daniel joins the CTO office at Veritas as a principal software engineer (LinkedIn, on this page).
- JWT
JSON Web Token becomes RFC 7519. The suggested pronunciation is "jot" (Wikipedia).
- The CNCF
The Cloud Native Computing Foundation is founded, as a subsidiary of the Linux Foundation (Wikipedia).
- Let's Encrypt
The free certificate authority launches after a public beta (Wikipedia, on this page).
- SPIFFE at GlueCon
Joe Beda, a co-founder of Kubernetes, first proposes SPIFFE, by DevOps.com's account. Daniel dates the white paper to 2017 (DevOps.com).
- Scytale
Sunil James, Emiliano Berenbaum and Andrew Jessup found the company (HPE).
- CNCF Sandbox
SPIFFE and SPIRE enter the foundation's Sandbox (CNCF).
- Scytale.io
Daniel joins as a principal software engineer, an early employee (LinkedIn, on this page).
- ACME
The protocol behind Let's Encrypt is published as RFC 8555 (Wikipedia).
- HPE acquires Scytale
HPE announces the deal and its commitment to SPIFFE and SPIRE (HPE, on this page).
- CNCF Incubator
SPIFFE and SPIRE move up to the Incubator (CNCF).
- KubeCon Europe
Daniel's lightning talk on bridging cloud native and traditional workloads with SPIRE (HPE Developer).
- Episode 3
Daniel on the show (Podbean, transcript).
- KubeCon in Los Angeles
Five days in person and online, after three virtual-only events (CNCF, on this page).
- Episode 10
Michael Malone of Smallstep on certificates, identity, X.509 and authentication (episode page).
- Graduation
SPIFFE and SPIRE graduate from the CNCF (CNCF).
- Sentima
Daniel co-founds Sentima, Inc. (LinkedIn).
- SPIFFE and WIMSE
A KubeCon North America poster on the two identity standards projects (KubeCon schedule).
- A stealth Fortune 500 company
Daniel's LinkedIn lists him as a principal engineer there (LinkedIn).
- NIST on agent identity
An NCCoE concept paper names SPIFFE and SPIRE as one way to identify agent workloads (NIST NCCoE).
- Your API key said yes
Our post on agent credentials points back to this episode (post).
- Flux 2.10
SPIFFE mutual TLS between Flux's controllers (GitHub).
- A SPIFFE ID for each agent
As of November 2026, Google Cloud's Agent Identity gives each agent a SPIFFE ID and X.509 certificates valid for 24 hours (Google Cloud, on this page).
What to do this week
Five things, in the order of the episode.
- List every static secret. Cloud access keys, API keys, database passwords and tokens in CI all count. Our 30-minute credential inventory has a template, and you can take the scorecard with your team.
- Give one cluster real identities. Try SPIRE on one Kubernetes cluster, starting from the SPIFFE site and the free book, and ask questions in the SPIFFE Slack.
- Replace one cloud access key. Pick a workload that calls a cloud API with a long-lived key, and move it to short-lived tokens through OpenID Connect federation, as in Daniel’s AWS example in OpenID Connect.
- Count what you’d issue. Put your workloads and renewal time in the calculator.
- Get a second pair of eyes. Product Security ships the security features enterprise buyers ask for, in your codebase, and Penetration Testing shows what an attacker can reach with a stolen key. If you want to talk it through, book 15 minutes with us.