AI Readiness Assessment: The Pre-Flight Check Before You Deploy an Agent

An AI readiness assessment is a structured check of whether your organization can actually support an AI deployment before you commit to it. It looks at the data the system will need, the context and systems it has to connect to, the access it requires, and the risk it introduces. The output is a plain answer to one question: are we ready to make this specific thing work?

Most published readiness assessments are broad. They score the whole company on strategy, culture, data, and talent, and hand back a maturity level. That has its place. But on episode 99, Ged Ossman, founder and CEO of Interf, made the case for a sharper version: a readiness check tied to one specific agent you are about to deploy. Not “is the company AI-ready” in the abstract, but “can this agent actually succeed here,” answered before the contract is signed.

Ged is refreshingly unromantic about the term itself:

“My advice would be first to understand: are you ready for this specific agent implementation? To do this, there is an activity, I hate to say, in the consulting world called a readiness assessment. Before deploying capital into something, you do this proactive activity of understanding what it takes for this initiative to succeed: what it takes from us to connect, to provide to the vendor, for this AI initiative to succeed. Answering: are we ready for this?”

Ged Ossman, founder and CEO of Interf, on episode 99

AI readiness vs. AI maturity: which one predicts whether the agent will work?

The two terms get used interchangeably, and the difference matters.

	AI maturity assessment	AI readiness assessment
Question	How advanced is our overall AI capability?	Can this specific agent succeed here?
Scope	Company-wide	One deployment
Output	A maturity level or score	Go, fix first, or not yet, with a list of what to fix
Timing	Periodic strategy exercise	Before you deploy capital on a project
Best for	Long-term planning	Preventing a stalled rollout

A company can score low on maturity and still be perfectly ready for one narrow, well-scoped agent. It can also score high on maturity and still fail a specific deployment because the exact data that agent needs was never connected. Maturity tells you where you are heading. Readiness tells you whether the next step will hold your weight.

The NIST AI Risk Management Framework makes the same move. Its MAP function starts by establishing the context and intended purpose of a specific system, and one of its categories (MAP 4) is explicitly about mapping risks and benefits “for all components of the AI system including third-party software and data.” That is a readiness check by another name, and it comes before anything gets measured or managed.

Why do most AI deployments fail? Readiness, not technology

Ged’s estimate on the show: more than 60% of AI implementations never get meaningful ROI. In his experience the cause is rarely the vendor’s model. The enterprise was never ready to feed the agent what it needed.

“Do these pre-flight checks first, before just blindly going to figure out gaps mid-flight. We see that more than 60% of AI implementations just don’t get any meaningful ROI. And the problem is, who can answer what went wrong? Enterprises will never say it failed because we were not ready. They would just say the vendors over-promised us. Because in the contract, they said they bring meaningful AI within six months, but there is no AI. But vendors, I know the other side.”

Ged Ossman, Interf, episode 99

We have heard the other side too. The vendor’s forward-deployed engineers show up to implement, the data is not where anyone said it was, the context is spread across five systems of record, and nobody can name the person who owns the inputs. Ged described what those engineers do next: “We raise tickets, we start exploring, we start basically the job of assessing the risk with their compliance and security teams, figuring out how to address that risk.” That is readiness work, performed at the most expensive possible moment, by the people with the least authority to fix anything.

The published research lines up with the anecdotes. RAND’s 2024 report on why AI projects fail, built on interviews with 65 practitioners, opens with “more than 80 percent of AI projects fail—twice the rate of failure for information technology projects that do not involve AI,” and two of its five root causes are readiness problems wearing other labels: organizations lacking the data the system needs, and underinvestment in the infrastructure for managing data and deploying models. MIT NANDA’s GenAI Divide report found 95% of organizations getting zero return on generative AI, yet purchased, customized tools reached deployment about twice as often as internal builds, so the model on offer is usually not the constraint. Gartner’s forecast that over 40% of agentic AI projects will be canceled by the end of 2027 blames “escalating costs, unclear business value or inadequate risk controls.” Two of those three are exactly what a pre-flight check settles before the contract.

This is a readiness failure in a technology costume, the same root cause we unpacked in why 95% of enterprise AI projects fail. The encouraging part: every cause on those lists is visible in advance, if someone goes looking on purpose.

What is a pre-flight check for an AI agent?

Ged borrows the term from aviation, and it fits. Pilots do not taxi out and hope; they walk the aircraft with a list, because the expensive surprises are the ones you cannot see from the cockpit. Before you deploy capital into an agent, you run the same kind of check. The parts that trip people up are rarely the obvious ones.

• Data quality and availability. Not “do we have data,” but is the specific data this agent needs present, accessible, and clean enough to use. Ged sees this fail before security is even in the room: “Usually we have this problem with data quality where the security is not even a concern. It’s just that the data is not there to make this specific agent deployment work.”
• Context dependencies. An agent that does scenario modeling might need ten different context entities connected before it works at all. These dependencies are usually hidden, and enterprises discover them only when the vendor’s engineers start asking.
• Hidden vendor requirements. The prerequisites that never make it into the security questionnaire. Ged’s longer-term answer is a machine-readable declaration that ships with every vendor agent; we cover that protocol in AI agent governance starts at onboarding, not runtime. Until your vendors ship one, you write the list yourself, in the same spirit as an AI data bill of materials.
• Risk and access. What the agent will read, what it can change, under which identity, and whether exposing that is acceptable to the people who own it.

Ged’s example of the hidden kind is worth hearing in full, because it is so ordinary:

“Everyone is already doing this with the forms and security questionnaires. That’s, I would say, stage one. But there are still a lot of requirements that are hidden, specifically around context dependencies. I have an AI agent that, for example, does scenario modeling for workplace strategy. For my AI agent to work, I need these ten answers, ten context entities we need to connect. And usually today this stuff is hidden, and enterprises start to figure out how to find those answers only when the vendor’s forward-deployed engineers tell them, during the onboarding process.”

Ged Ossman, Interf, episode 99

The expensive dependencies are the invisible ones. A readiness assessment is the act of making them visible, on purpose and early.

How to run an AI readiness assessment in five steps

You can apply this without a consulting engagement. The goal is to surface the work while it is still cheap to do, before the contract rather than during onboarding.

1. Define the specific job. Name the exact agent, the exact outcome it is supposed to produce, and the one person who owns that outcome. Readiness is always readiness for something. “Do something with AI” is not a job; “cut invoice-matching time in half for the AP team by Q2” is.
2. List the inputs it needs. Every data source, system, context entity, identity, and approval the agent depends on. Ask the vendor to declare these in advance rather than discover them live. Ged’s script for what a vendor should say is the one to borrow: “This is the risk that our solution introduced. So please, please share it with the industry, make this risk visible for everyone who plans to adopt this vendor solution.” A vendor that cannot produce the list is telling you something.
3. Check each input against reality. Is it present, connected, clean, and reachable today? Not on the roadmap, not in the warehouse someone is planning to build. Mark each one ready, fixable, or blocker (more on the scoring below).
4. Score the risk. What does the agent touch, and are you willing to expose it? Loop in security and compliance now, not after go-live; if you sell to enterprises, the agent will need to appear in your SOC 2 system description too. Unmanaged experimentation is its own exposure; see shadow AI. Start the agent on read access (write access is earned, not demoed) under its own scoped identity, because least privilege matters more for agents than it ever did for people. If the agent reaches tools through MCP servers, each server is an input on the list and each one is executable trust. Once the inputs are enumerated this step nearly writes itself: each data source carries a classification, the classification tells you what protection the agent needs, and the security review becomes a lookup instead of an argument.
5. Decide: go, fix first, or not yet. Write it down with names and dates. If it helps to have a standard behind you, the NIST AI RMF spells out this exact step. MANAGE 1.1 in the Playbook reads: “A determination is made as to whether the AI system achieves its intended purpose and stated objectives and whether its development or deployment should proceed.”

How to score readiness: ready, fixable, or blocker

Three states are enough. More than three and the exercise turns into a scorecard people argue about instead of a decision people act on.

Ready means the input exists today and someone has verified it, not that a diagram says it should. The table is reachable from the environment the agent will run in, it is fresh, it is classified, and anything that must never leave (payment data, health records, a customer’s secrets pasted into support tickets) has been excluded. If the agent’s inputs include data you would not want in a model’s context window, PII and secret scrubbing is a fixable with a known owner and price, not a blocker.

Fixable means the input exists in some form and a named person can make it ready by a date: scattered data, a shared credential where a scoped service account is pending, a review that is scheduled but not held. Fixables are the normal state of a healthy company. Each one gets an owner and a date, and the check is re-run before anything is signed.

Blocker means the deployment cannot succeed as scoped: the data does not exist or cannot legally be shared, the core system has no API and no owner, the agent needs write access nobody will sign for, or nobody will own the risk. A blocker is a verdict on this scope, not on the company, and the usual fix is a smaller job.

The roll-up is mechanical on purpose. All ready: go, and record who accepted the residual risk. Any fixable and no blocker: fix first, then re-check. Any blocker: not yet, said out loud before the contract. Ged’s priority order, learned building crypto custody long before agents, is the right one for the whole exercise: “It’s always about risk: understand it, remove it, then understand the business. Technology comes second. It should never come first.”

Who should run the AI readiness assessment (and why not the vendor alone)?

Every party in an agent deal has a reason to want the answer to be yes. The vendor has a quota, the sponsor has a board slide, and the consultancy selling readiness assessments has a follow-on engagement. Ged’s test for all of them is the most quotable line of the episode: “Trust is transparency divided by self-interest.” He was talking about vendor-intelligence platforms, and Jon pointed out on the show who pays Gartner. The formula travels.

So split the roles. The vendor declares what the agent needs, in writing, and is judged on the completeness of that list rather than the polish of the demo. Your data owners, security team and compliance lead verify each line and decide what you are willing to expose. The budget owner signs the decision and is explicitly allowed to sign “not yet.” If those people are not in-house, bring in someone whose fee does not depend on the outcome; the in-house, on-demand, or YOLO choice applies here as everywhere else in security.

The assessment should also protect room to experiment. A “not yet” on a production agent is not a ban on trying things. Ged put it well: “I want to create engineering sandbox environments that make experimentation friendly, so organizations incentivize people to experiment with AI, but in a secure environment. Not like you install something in your CLI, give it root access, and then you get hacked the next day and you didn’t think about it.” The sandbox is where fixables get proven, and it is the constructive answer to shadow AI: a safe place to say yes.

Where readiness meets compliance: SOC 2, ISO 42001 and the NIST AI RMF

A pre-flight check produces exactly the artifacts your compliance program wants anyway, which is a good reason to run it the same way every time.

The agent’s declared inputs and access become its entry in your SOC 2 system description, where enterprise buyers increasingly expect to see AI components named. The risk score and the recorded decision are the documented determinations the NIST AI RMF asks for across its MAP and MANAGE functions, and NIST’s Generative AI Profile (AI 600-1, July 2024) adds the generative-specific risks to look for. If you run more than one or two agents, ISO/IEC 42001, the AI management system standard published in December 2023, is the repeatable home for the routine: it specifies “requirements for establishing, implementing, maintaining, and continually improving an Artificial Intelligence Management System,” and a readiness check per deployment is exactly the kind of operating control an auditor wants to see running. We certify companies under it, and the ones already running pre-flight checks have most of the evidence before we arrive.

None of this is paperwork for its own sake. Jon’s line on the show was that innovation which is not easy to adopt is not really innovation; it might be a fun hobby, but it will not change anything. The readiness check is how an agent becomes easy to adopt: security says yes because they have seen the list, compliance says yes because the agent is in the description, and the budget owner says yes because the outcome has a name.

The checklist to take into your next vendor conversation

The short version, for the meeting where someone is about to say “let’s just sign and figure it out.” If you cannot answer yes to these, you are not ready yet, and knowing that today is the win.

• We can name the specific outcome this agent is supposed to deliver, and the person who owns it.
• The vendor has declared, in writing, every data source, system, context entity, identity and approval the agent needs.
• Each required input exists, is accessible from where the agent will run, and is clean enough to use.
• The context entities the agent relies on are actually connected, not theoretical.
• Security and compliance have reviewed what the agent will read and what it can change, and the agent has its own scoped identity.
• Every fixable has an owner and a date, and the check will be re-run before the contract is signed.
• We have an honest “not yet” option, and leadership will accept it.

Saying no to AI is rarely the smart move; saying yes without readiness is how you join Ged’s 60%. The middle path, running the check first, is how you get to yes and keep control of what you exposed to get there. His summary is the one to leave with: “For me, the structural fix, how to make this thing effective, is for enterprises not hoping vendors will solve all their problems, but first doing these pre-flight checks, answering: are they ready yet?”

If you have an agent deployment on the calendar and step four is the one nobody owns, our AI access control audit maps everything the agent can reach, and everything that can reach it, and cuts both down to least privilege before go-live. If you would rather hear the argument first, Ged and Jon make it in 43 minutes on episode 99.