AI Deployment Security: Build the Boundary Outside the Model

In June 2025, researchers at Aim Labs disclosed EchoLeak, a vulnerability in Microsoft 365 Copilot in which an email did not have to persuade a person. It had to persuade the assistant reading on that person’s behalf.

An attacker-controlled email, retrieved later by Copilot while it handled a user’s ordinary request, could redirect the assistant into a chain that exposed private information. The victim never had to click anything, because the attacker’s instructions arrived inside material the system was supposed to use as evidence, not obey as authority. Microsoft assigned the flaw CVE-2025-32711 with a CVSS score of 9.3, fixed it server-side and stated that no customers were affected. That is a vulnerability in a production product, subsequently remediated, and not a laboratory chatbot with every protection disabled. It is also a demonstrated exploit rather than a documented breach campaign, and the two should not be confused.

The boundary that failed was the one between something the organisation had received and something the organisation had authorised, not the one between the internet and the corporate network.

A model in production will at some point be fooled. The deployment has to be built so that a fooled model cannot exceed the consequences the organisation authorised.

AI security adds to application security without replacing it. The software running the model still needs patching. The identities still need protecting. Someone still has to apply access controls to the documents. An attacker can exploit any of those without understanding machine learning.

The distinctive challenge is that the model can also be manipulated while the surrounding software is functioning as intended. It may interpret an attacker’s material, produce a plausible answer, or propose a perfectly well-formed tool call that serves the wrong party. Whether that becomes an incident depends on the authority the system gives the output.

AI model security means protecting models, their data and supporting infrastructure, and the decisions and actions that depend on them, against unauthorised disclosure, manipulation, theft, misuse and disruption. In practice, securing the model is only one part of securing an AI deployment.

The companion article sets out what is different about securing a model and what is not. This one is about the deployment: what to build around a model that will, at some point, be fooled.

A useful programme therefore has two jobs: make the model and its environment harder to compromise, and ensure that a compromised or mistaken model cannot exceed the deployment’s permitted consequences. The first raises the attacker’s cost. The second limits what a successful attack buys.

Both jobs are required, and doing one well does not cover the other.

Model, system and agent are three different objects

A discussion about “the AI” becomes unmanageable unless it distinguishes the component being evaluated from the system being deployed.

A model is the learned mapping: its architecture, parameters and relevant configuration. It might classify a transaction, rank documents, recognise a face, generate text or predict a control variable. The parameters may be available to the organisation, or the model may be accessible only through a provider’s interface. A security assessment at this layer asks what information the model reveals and how an adversary can alter its behaviour. It also asks whether someone has already tampered with the model.

An AI system includes that model and the machinery that makes it useful: preprocessing, retrieval, prompts, application code, authentication, databases, serving infrastructure, output rendering and downstream integrations. At this layer the model’s prediction denies a transaction and its answer reaches a customer. Its generated code enters the build.

An agent is a system that uses a model to select and pursue actions towards a goal, with some degree of autonomy. Tools, persistent memory and delegation extend its reach. They are not a magical threshold beyond which security suddenly matters; they are mechanisms that increase the number of things a model-driven decision can affect. An ordinary fraud classifier connected to an automated approval process already has consequential authority without looking anything like a conversational agent.

These distinctions change the questions a buyer should ask. “Can an attacker make the model produce a prohibited sentence?” is not equivalent to “Can an attacker make this service disclose another customer’s records?” Neither answers “Can this agent change the payment destination?” A model-level result can inform all three, but it does not establish any of them without the deployment conditions. The site’s attack taxonomy sorts every attack family by which of the three objects it reaches, and that sorting is the first thing to check in any claim.

For an application owner, the best unit of analysis is the workflow: who requests the work, which data becomes available and which decisions are delegated. Then ask which actions can follow and where independent enforcement intervenes. The same model can be a modest risk in a public-document summariser and an unacceptable risk in an unrestricted production administrator.

The distinctions between secure, safe, responsible and trustworthy AI describe overlapping concerns here too, not interchangeable certifications. A system can refuse obviously harmful requests yet approve an unauthorised refund. It can protect customer confidentiality yet make inaccurate decisions. The security reviewer coordinates with the safety, privacy and reliability owners, then asks a separate set of questions about adversaries, authority and loss.

Start with this one: what can happen because the model produced that output?

Start with the deployment, not the model’s reputation

Before commissioning an AI red team or buying a guardrail, describe the deployment in operational terms. “We use an enterprise model” is not a threat model. Neither is “we host everything ourselves.”

The following is a starting allocation of responsibilities, not a claim that every service implements the same boundaries.

Deployment	Exposure to examine first	Responsibility the organisation retains
Hosted model API	Data sent to the service, account compromise, provider dependencies and API behaviour	Data selection, application authorisation, credentials, retention requirements, output handling and downstream actions
Self-hosted model	Model artifacts, loaders, inference servers, runtime dependencies and administrative access	The complete hosting stack, isolation, patching, provenance, capacity and recovery
Retrieval assistant	Who can read or modify indexed content, and how retrieved material influences the answer	Document-level permissions, ingestion controls, provenance, tenant separation and disclosure controls
Tool-using agent	The operations, identities, destinations and resources available to the agent	Delegation, action-level authorisation, execution isolation, approval design, budgets and shutdown
Training or fine-tuning programme	Dataset and label manipulation, training infrastructure, checkpoints and evaluation contamination	Data lineage, pipeline integrity, experiment access, release evidence and protection of sensitive artifacts
Predictive decision system	Inputs an adversary can change, decision thresholds and downstream consequences	Realistic evasion testing, independent constraints, monitoring, exception handling and recourse

These categories overlap. A self-hosted model can power a retrieval agent trained on proprietary data; responsibilities accumulate rather than cancel out.

Classify the consequences before choosing the controls. An error that wastes an analyst’s time is different from one that exposes a merger document, changes a beneficiary, deploys software or alters a physical process. Consider reversibility, scale, time to detection and the availability of an independent check. “Human reviewed” is not a consequence category: a person may review a draft carefully, rubber-stamp hundreds of proposed actions, or see the output only after the action has completed.

Provider selection should then follow the threat model. A managed service can remove infrastructure duties while introducing dependencies on the provider’s controls and change process. Self-hosting can improve control over data paths while making the organisation responsible for another security-sensitive platform. Open weights enable deeper inspection and local testing; they also give an attacker the same copy, and what that access buys is the first question to ask of any published attack. Each of these choices moves responsibilities between the provider and the operator. The organisation still owns the right-hand column of the table above.

The decision worth documenting is not whether a model is trusted in the abstract. It is whether this version, in this configuration, is permitted to perform this job under the constraints the organisation has stated.

The ordinary software still needs ordinary security

An AI deployment can fail before the model has interpreted a single token.

In June 2024, Wiz’s Probllama research described a path-traversal flaw in Ollama’s model-pull functionality, CVE-2024-37032, that gave an attacker an arbitrary file write and, on root-privileged Docker deployments, remote code execution. Nothing about the underlying failure required an adversarial example or a breakthrough in language-model manipulation. The companion article on what is and is not different walks that incident, the exposure counts around it and the malicious pickle files on model hubs; the point here is what an operator does with them.

That failure is part of AI security, not an embarrassing exception to it. The model registry, package repository, notebook server, inference endpoint, vector database and tool server are software services. Their authentication, dependencies, network reachability and administrative interfaces belong in the organisation’s normal security processes.

The same applies to model loading. A model file is not necessarily passive data. Pickle-backed formats can invoke executable behaviour when deserialised. PyTorch changed its default in version 2.6 so that torch.load runs in the restricted weights_only=True mode when no explicit pickle_module is supplied, and two CVEs since then have shown that the narrower default reduced the remote-code-execution surface without removing it.

The operational instruction is therefore more demanding than “upgrade PyTorch.” Use a supported, patched release; inspect the actual loading path; understand allowlisting; and identify wrappers or compatibility code that restore unrestricted loading. An application can undermine PyTorch’s safer default without downgrading it.

Prefer formats designed for tensor data, such as safetensors, over unrestricted object deserialisation. That removes pickle’s intentional execution mechanism from that part of the process without making an entire model repository safe. Optional Python code, installation scripts, parsers, custom operators and the learned behaviour of the weights remain separate questions; a backdoor implanted during training passes every integrity check a file format can offer. Converting an untrusted pickle into a safer format also requires care: the conversion process may execute the original payload before it writes anything safer.

For self-hosted deployments, an initial review should be able to identify every serving and management endpoint, its authenticated callers, its privilege level and its patch owner. Keep management operations separate from ordinary inference access. Restrict outbound downloads and execution. Keep registries and credentials out of unnecessarily privileged runtime environments. Put development experiments through a deliberate promotion process before treating them as production services.

NIST SP 800-218A, the July 2024 SSDF community profile for generative AI and dual-use foundation models, takes the sensible approach: extend secure software development to cover AI-specific work rather than discard the software lifecycle and start again.

Disclosed vulnerability counts describe the software layer’s exposure, not the number of incidents and not the size of the loss. A vulnerable GPU driver, an exposed inference server and a successful prompt-injection disclosure do not share a denominator. The software backlog and the model-level findings are tracked separately, and neither is used to postpone the other.

Prioritise the exposure you can demonstrate. Close an unauthenticated administrative endpoint immediately. Do not use the existence of that endpoint as an excuse to postpone reviewing the agent that can already reach your customer database through legitimate credentials.

The attack families, and what an attacker must control

An attack name is useful only when it brings its access assumptions with it. NIST AI 100-2 E2025, the March 2025 edition of NIST’s adversarial machine learning taxonomy, organises the field across objectives, lifecycle stages, knowledge and capabilities. The companion article maps the families, and the site taxonomy files each one against what contains it. Read each family below for what the adversary has to control, and check whether it exists in the system you operate.

Evasion changes the decision without touching the model

An evasion attack manipulates the input presented to a deployed model. Since Szegedy and colleagues’ 2013 paper, the research has established a real failure mode, and what has actually been demonstrated against deployed systems is thinner than the headlines. For a defender, the central issue is whether the attacker can make the required change while preserving the thing they want to accomplish. A malware sample that evades classification only after losing its malicious functionality is not a successful malware-evasion attack. A network record edited into an impossible packet sequence is not a demonstrated bypass of an intrusion-detection system. A digital image result does not automatically survive printing, distance, lighting or camera processing. This is the feature-space against problem-space gap, and it is the reason real attackers mostly do something cheaper than adversarial machine learning.

Test those constraints. In fraud detection, identify which transaction attributes a fraudster can actually influence. In intrusion detection, preserve the malicious operation and the protocol’s requirements. For perception systems, test the relevant physical and sensor conditions, not just an image file, and remember that semantic attacks which change the lighting or the phrasing fall outside every robustness guarantee written for a pixel budget.

White-box gradients are one capability, not the only one, and what the attacker’s access actually buys decides whether a published result applies to a hosted API. Surrogate models, transfer attacks and informative query responses create other routes. Conversely, a white-box result should not be reported as a demonstrated API exploit when the attacker lacks the necessary feedback. The label “academic” says nothing about whether the attack path exists in the system you operate.

Poisoning and backdoors shape what the model learns

The important practical distinction is between the ability to publish content and the ability to get content used in a specific training run. Those are not the same permission.

In October 2025, Souly and colleagues at the UK AI Security Institute, Anthropic and the Alan Turing Institute found that 250 poisoned documents could establish a trigger-driven gibberish response across models from 600 million to 13 billion parameters, and that the required count did not grow with training-data volume. The documents were inserted into the corpus directly. The target behaviour was narrow, and no frontier-scale model was tested. What 250 documents actually proved is that the arithmetic favours the attacker more than assumed; nobody has measured the probability that a planted page survives scraping, deduplication and filtering into a particular run.

For an enterprise, the more immediate exposure is usually a much smaller pipeline: a fine-tuning dataset, a feedback queue, a labelling contractor, a synthetic-data generator or a repository of accepted examples. A preference signal is a training label, and flipping 0.3% of preference labels was enough to defeat safety alignment in one 2024 study. Where the model updates continuously from an incoming stream, the order of the poison decides whether it takes hold, not the volume. Ask who can contribute, who approves inclusion, and whether the evaluation data can be influenced by the same party.

Backdoor detection is not limited to searching a list of already known triggers. Methods such as Neural Cleanse try to reconstruct candidate triggers; other approaches examine suspicious internal patterns. But a passing detector is not proof that a model is free of every conditional behaviour, and Goldwasser and colleagues showed in 2022 that backdoors can be made computationally undetectable under stated assumptions. Provenance, restricted ingestion and evaluation are complementary controls, not interchangeable ones.

Privacy attacks recover what the model should not disclose

Membership inference asks whether a particular record participated in training, which can be sensitive without recovering the record; Shokri and colleagues showed in 2017 that useful evidence can be available through black-box prediction interfaces under the tested conditions. Model inversion attempts to infer attributes or reconstruct input information from a model and its outputs, and what it actually recovers is class representatives more often than individuals. Training-data extraction targets memorised content directly; Carlini and colleagues extracted training sequences from GPT-2 in 2020. In each case access, memorisation and the attacker’s strategy determine the result.

A deployment must also distinguish these learned-information attacks from ordinary disclosure through context. An assistant that forwards a retrieved payroll document has not extracted anything from its training data. The sensitive information may have entered the prompt seconds earlier. Applying a training-time privacy technique would not repair that authorisation failure, and most of the incidents an enterprise deployment will see are of the second kind.

Model extraction and weight theft are different ways of losing the asset

“Model stealing” covers several objectives. An attacker may copy behaviour through queries, infer part of a model’s structure or parameters, or steal the actual weight files from the infrastructure. The last is an information-security compromise; the first two exploit what a service reveals by operating, and the three have split into different problems that share a name and little else. In Stealing Part of a Production Language Model, published in March 2024, Carlini and colleagues recovered one output-projection matrix, up to symmetries, from historical production interfaces for under $20 on two smaller OpenAI models. That is a result about one component and an informative interface, not recovery of a model. Where the model makes a security decision, query access buys reconnaissance for the evasion that follows, and that is usually the loss to plan for.

The controls follow the objective: protect valuable weights as sensitive artifacts, limit unnecessarily revealing API outputs, monitor sustained extraction-like use without assuming every heavy customer is an attacker, and assess whether functional imitation is an economic threat even when exact parameter recovery is not.

Prompt injection, jailbreaking and misalignment displace different instructions

Prompt injection attempts to make an AI system treat attacker-controlled material as an instruction that overrides the intended task or policy. In indirect injection, the material comes from something the system encounters: a document, search result, repository file, tool response or message. Greshake and colleagues’ 2023 paper showed why an attacker need not be the application’s direct user to influence an integrated language model.

A jailbreak targets the model’s own behavioural restrictions, usually through an adversarial user’s direct interaction; Wei, Haghtalab and Steinhardt’s 2023 analysis examines why safety training fails. The categories can overlap, but a model refusing to provide dangerous instructions still does not establish that an enterprise application enforces customer permissions.

The third mechanism has no adversary at all. Misalignment is the system pursuing the wrong objective competently, whether that comes from an objective that was specified badly or a behaviour that appeared at scale without being trained for. Three mechanisms, three defences, and a review that names the wrong one buys the wrong control. The same authorisation boundary can prevent the same harmful action in all three cases, which is why a good architecture should not need to diagnose the model’s motives before refusing an unauthorised operation.

An invoice arrives containing a request to update a bank account. Nothing in the sentence resembles prohibited content, and the harm is that the sender of an invoice is being allowed to alter the organisation’s payment authority. A safety classifier can accurately label the text as innocuous and the workflow can still be insecure. The classifier is a text classifier, and the 2018 literature on fooling text classifiers applies to it unchanged. The same principle extends beyond text: multimodal attacks put attacker-influenced material into images or other inputs through a channel a text filter cannot read, and moving an instruction from a web page into a screenshot does not create an independent source of trust.

Build the boundary outside the model

A source-code review can trace an explicit authorisation check. Reviewing the training code does not similarly reveal every behaviour encoded in a model’s weights, and the companion article sets out why that gap has no patch. But “there is no specification” is the wrong conclusion for the system around the model. A support assistant may read only the records of the tenant it is serving. A purchasing agent may prepare a payment against a beneficiary that already exists. A coding agent must get the required approval before deploying its own change. These are enforceable requirements. They do not depend on predicting every possible prompt.

The design task is to move consequential requirements out of the model’s discretionary behaviour and into independently tested enforcement. Use evaluation for the behaviour that remains statistical; use ordinary security engineering for the permissions and boundaries that can be explicit. Explainability tools can help investigate particular decisions; treat them as additional evidence, not as a replacement for an access-control check, because an explanation is computed from the model it describes and is both forgeable and informative to an attacker.

The most important architecture decision that follows is whether the model’s output is a proposal or an authority.

A proposal can be checked, restricted, rejected, recorded and approved through another mechanism. An authority can decide that its own proposal is permitted. Many unsafe designs collapse those two roles: the model chooses the action, explains why it is safe, and supplies the instruction that causes it to happen.

My default for consequential enterprise workflows is to separate interpretation from permission. Let the model extract, reason, draft and recommend. Give a different component responsibility for deciding whether the requested action is allowed, and make the executor incapable of bypassing that decision.

CaMeL, published by Debenedetti and colleagues at Google DeepMind and ETH Zurich in March 2025 and revised in June, is the research demonstration of this direction. It separates privileged planning from processing untrusted content and uses an interpreter with provenance-aware capability policies to constrain control and data flows. The revised paper reports 77% task completion with provable security on its AgentDojo evaluation, against 84% for the undefended system. Its guarantees are scoped: the design makes trust assumptions, including about the initial user request, and the paper discusses limitations and side channels. It does not prove that every generated answer is correct or that arbitrary agent workflows are secure.

The value is the architectural lesson, not a universal benchmark promise: some consequences can be constrained without requiring the model to recognise every attack.

Role markers do not do this job. Language-model interfaces mark system, developer, user and tool content differently, and the distinctions help, but a role marker does not enforce the organisation’s authorisation policy; the model is still interpreting content, including content written by an adversary. OWASP’s prompt-injection prevention cheat sheet consequently treats structured prompts, input checks, output validation and operational controls as layers. The comparison with SQL injection holds only up to a point. Parameterised queries separate query structure from values for the database operation, and the team building an AI application has to enforce the same separation, but its policy concerns a business action rather than a parser. “This field is a valid account number” and “this account is an approved beneficiary for this invoice” are different checks, and you do not have to solve every ambiguity inside the context window to enforce the second one outside it.

The following payment workflow is an illustrative design, not a description of CaMeL or a claim that one architecture fits every application.

An invoice is evidence, not payment authority

Suppose an accounts-payable assistant reads incoming invoices, matches them to purchase orders and prepares payments. A supplier document includes instructions to replace the supplier’s bank details and pay immediately. The model may read those instructions, understand them and even recommend following them. The system, not the model, decides what that recommendation can change.

In a weak design, the assistant has broad access to the accounting system and a tool that can update suppliers and release payments. Its prompt tells it to be careful. A second model reviews the proposed action. Both models receive the invoice, and both can be influenced by the same claim of urgency or authority. The workflow has duplicated interpretation without establishing independent authorisation.

In a stronger design, the task begins with an authenticated employee and a bounded delegation: prepare a payment proposal for a specified purchase order, within a specified amount and period. The employee’s instruction does not grant rights they do not already have, and it cannot override the organisation’s segregation-of-duties rules.

The document-processing component can extract an invoice number, supplier name, amount and claimed bank details. It cannot amend the supplier master record. It cannot issue a payment. Its output remains evidence with provenance: these values came from this document, supplied through this channel, at this time.

A policy service then checks the proposal against authoritative records. The permitted beneficiary comes from the approved supplier record, not from whichever account number the model found persuasive. The purchase order, receipt status, amount limits and duplicate-payment checks come from the relevant business systems. A requested change of bank details enters a separate verification process.

An authentic supplier email still does not automatically authorise that change. Authentication establishes who sent a message, within the limits of the authentication mechanism. The supplier’s authority to change payment instructions, and the process for verifying a potentially compromised supplier account, is a separate business control. The model classifies; it does not authenticate, and channel authenticity, measurement integrity and model correctness are three properties none of which supplies the other two.

The executor receives a narrow, approved operation. It has no general-purpose ability to edit suppliers, browse the filesystem or call arbitrary financial APIs. An audit record ties the resulting transaction to the employee, the agent workload, the source records, the policy decision and any required approval.

The model can still be wrong. It can misread an amount or invent a convincing explanation. The design is not relying on it to be incapable of error. It is preventing an untrusted invoice from manufacturing the authority needed to redirect money.

Typed output is plumbing, not an authorisation decision

A JSON schema can ensure that a proposed action has the expected fields and types. An enum can restrict the tool name. These checks are valuable because they remove ambiguity from the software interface. They do not establish that the action is appropriate.

A schema-valid send_document request can name the wrong recipient. A valid refund amount can be attached to an ineligible order. A valid SQL statement can ask for another tenant’s records. A tool allowlist can include exactly the tool an attacker needs.

The policy must therefore evaluate the arguments and their provenance, not only the operation name. Which object? Which tenant? Which recipient? Under whose delegation? For what task? With which permitted data? How much? How many times?

For the payment example, the decisive check is not whether the model used a payment tool correctly. It is whether this exact payment is authorised independently of the model’s claim that it is.

Approval must bind the action the person actually saw

“Human in the loop” describes an interaction, not its security properties.

A reviewer who sees only “Pay the approved supplier invoice” has not necessarily approved the actual recipient, amount or bank account. A reviewer shown a model-written reassurance may be relying on the same compromised interpretation that produced the request. A system that permits the agent to alter the arguments after approval has not preserved the approval’s meaning. The EU AI Act’s human-oversight obligation assumes a reviewable moment, and an agent workflow has to be designed to produce one.

OWASP’s transaction-authorisation cheat sheet, written for online banking, applies directly: users need visibility of significant transaction data, authorisation must be enforced server-side, and the transaction must remain bound to what was authorised.

For an AI workflow, show the canonical operation through a trusted interface: the actual recipient, object, amount, destination, material changes and supporting records. Tie approval to that exact operation and its relevant record versions. Make it expire. Prevent replay. Revalidate permissions and material state at execution time. A changed beneficiary or a changed payload should require a new decision, not inherit a stale approval.

The same design must handle retries. A network timeout does not authorise paying twice. Use transaction identifiers and idempotent execution where the underlying operation supports them, and reconcile uncertain outcomes before trying again.

This is more work than adding a confirmation button. It is also the difference between a genuine approval control and a record that someone clicked something.

Permissions follow the task, not the model’s ambition

Give an agent a distinguishable workload identity, but retain the identity and permissions of the person or process that delegated the task. Replacing user context with a broadly privileged service account can make access control worse while improving the appearance of auditability.

For a user-delegated task, effective permission should be constrained by the user’s entitlement, the agent’s role, the task’s scope and the organisation’s policy. An autonomous scheduled job needs its own explicitly approved service scope instead. Neither case should depend on the model asserting who it represents.

Use credentials that expire quickly and are scoped narrowly, or an execution broker that supplies credentials without exposing them to the model. Separate reading from writing, proposing from approving, and approving from executing. Restrict the environment as well as the API. A coding assistant with a narrow repository token but unrestricted access to the developer’s home directory may find more authority there.

Independence must be real. The constrained agent should not be able to edit its own policy bundle, modify the executor, approve its own deployment, disable audit logging or grant itself an exception. Protect those control-plane operations under separate administrative authority. Otherwise, the system has given the agent both a boundary and the means to move it.

The NCSC’s interim advice on managing the cyber risk of agentic AI, published on 20 August 2026 ahead of formal guidance, asks for threat modelling before deployment, proportionate autonomy, sandboxing, least-privilege credentials, monitoring, human oversight scaled to the risk and an independent ability to stop activity, and it warns against relying on the model vendor’s built-in safeguards as the boundary. Threat modelling, sandboxing, least privilege and monitoring are standard controls. The NCSC’s contribution is the insistence that useful autonomy must operate inside enforceable limits.

Restrict the data leaving, not only the domains contacted

Simon Willison’s lethal trifecta, named in June 2025, identifies a dangerous combination: private data, untrusted content and external communication. Meta’s Agents Rule of Two, published in October 2025, develops a related design constraint around untrusted inputs, sensitive access, and state change or external communication, with supervision when all three are required in one session. Both are useful architecture-review prompts. Neither is a universal safety proof or a substitute for least privilege.

Use the heuristic to identify a path, then examine the actual path. An allowed email service can still deliver to an attacker. An approved storage platform can host an attacker-controlled folder. A search query can contain confidential text. Rendering a generated link or image can create network activity outside an explicit “send” tool.

A meaningful egress policy needs to consider recipients, resources, payloads and the data’s permitted use, not just whether a hostname appears on an allowlist. Where possible, keep sensitive data out of components that can communicate freely, and keep arbitrary outbound communication out of components that process sensitive data.

Removing an exfiltration path also does not eliminate integrity or availability risk. An agent unable to contact the internet may still corrupt internal records. The objective is to break the relevant harmful chain, not to declare the entire system safe after satisfying a three-item checklist.

In retrieval, relevance must not become permission

Retrieval-augmented generation is often presented as a way to ground a model in approved knowledge. That description contains two separate claims. A document may be relevant without being accurate, and it may be accurate without being authorised for this user or this action.

The retrieval layer therefore needs two boundaries: one governing which material can enter the model’s context, and another governing what authority the system gives that material afterwards.

On access, enforce the requesting user’s and task’s permissions before sensitive content reaches the model. Asking the model to ignore documents the user should not see is too late. Apply the same discipline to chunks, metadata, embeddings, reranking inputs, cached answers and generated summaries where they can expose protected information; the leaking interface has moved to the embedding layer, and an embedding store that inherits no permissions is a plaintext store that inherits none. OWASP’s Vector and Embedding Weaknesses entry, LLM08 in the 2025 edition and LLM09 in the 2026 edition published in August 2026, addresses the risks introduced by shared retrieval infrastructure and insufficient separation.

Permissions also change. In an enterprise search system, removing a user from a project should remove their effective access to the project’s derived search material and cached answers according to the system’s revocation requirements. An access check that was correct at ingestion is not necessarily correct at retrieval. Design the lifecycle, not just the initial indexing job.

On integrity, control who can publish into authoritative collections. A shared drive, internal wiki or support ticket is not trustworthy merely because it is inside the organisation. Different authors have different rights, and some content legitimately originates outside it. Retrieval ranking should not silently promote a customer’s assertion into company policy.

A useful design labels the source, owner, trust class and version through metadata managed outside the model. It distinguishes an approved operating procedure from an unreviewed comment discussing that procedure. When the model combines them into a summary, that summary should not automatically inherit the stronger source’s authority.

OWASP’s RAG security cheat sheet, added in May 2026, treats ingestion, retrieval, access control and output as a connected security problem. For an operator, the practical test is a complete document lifecycle: can you identify where an untrusted document was indexed, which tasks retrieved it, which derived artifacts included it and what happens when it is withdrawn?

Call manipulation of that runtime corpus retrieval poisoning or knowledge-base manipulation where appropriate. Do not imply that the model’s weights were retrained. The responder has to know which one happened: removing a poisoned document and invalidating affected state is a different recovery process from replacing a poisoned checkpoint.

Finally, grounding is evidence support, not proof of truth. A model can cite an irrelevant passage, misunderstand a correct source or combine sources incorrectly. For a consequential decision, require the domain-specific check that makes the decision acceptable. A citation is not an approval signature.

Agents add persistence, delegation and speed

Tools expand what a model can affect. Memory expands how long an attacker’s influence may survive. Delegation expands how far it may travel. Automation expands how much can happen before anyone notices.

Those are separate dimensions of risk. An architecture review should examine each rather than treating “agentic” as a single product category.

Memory is a write path with its own permissions

An agent that saves a conclusion for future use can turn a transient input into persistent influence. That does not require changing model weights. A stored preference, summary, task plan or retrieved note may be enough to affect later behaviour, and the poisoning research on models that learn from a stream has the same shape once the stream is an agent’s memory. OWASP’s AI agent security cheat sheet includes memory and context among the surfaces that need protection.

Separate descriptive memory from authority. “The user prefers concise replies” is not in the same category as “the user has approved all future transfers.” The latter is a permission claim and should not become valid because a model wrote it into a memory store.

Decide which components may write which types of memory. Preserve the originating user, task and evidence. Give sensitive state an expiry and a deletion path. Prevent a low-trust document from silently rewriting a higher-trust operating instruction. Where a memory influences consequential actions, expose the relevant change for review rather than burying it in an opaque personalisation process.

Include memory in recovery. Restarting a conversation does not remove a malicious note that will be retrieved into the next one. Clearing the original input is insufficient if a summariser has already copied its instruction into a different store.

Tool protocols do not transfer the burden of authorisation

Model Context Protocol can standardise how clients discover and use tools. It does not make every connected server trustworthy or every requested operation appropriate. The security best practices that accompany the 28 July 2026 revision of the specification address familiar problems including confused-deputy behaviour, token handling, authorisation boundaries and unsafe access to network resources. The authorisation specification forbids token passthrough outright rather than treating any available credential as portable permission.

For an operator, onboarding a tool server is onboarding software and a permission surface. Identify its owner, version, executable dependencies, outbound access, credentials and update process. Review changes to tool descriptions and schemas as well as executable code: the descriptions influence the model’s choices, while the implementation determines the actual effects.

A local tool process can also inherit the power of its host environment. Do not confuse a local connection with a sandbox. The same review should cover repository instructions, agent skills and other files that alter how an assistant works. Content that influences execution belongs under change control even when its filename ends in Markdown rather than Python.

In multi-agent systems, require the receiving agent to validate the delegation instead of accepting a persuasive message from another model as proof of authority. Carry the requesting principal, permitted task, resource scope and expiry through the chain. A subordinate agent should not acquire broader permission simply because the first agent asked politely or described the task as urgent.

This is a distributed authorisation problem with model-generated messages in the middle. Treating every internal agent as mutually trusted recreates the flat-network problem at the application layer.

Budget the workflow, not only the request

A system can harm its operator without stealing data or issuing a forbidden command. It can consume paid inference, fill a queue, repeatedly call an expensive service or block other tenants from using shared capacity. OWASP’s Unbounded Consumption entry, LLM10 in the 2025 edition and promoted to LLM06 in the 2026 edition, captures the availability and economic dimension.

Set limits on the complete task: wall-clock time, token expenditure, tool calls, retries, concurrent work, spawned subtasks and downstream cost. Per-request limits alone do not stop a planner from creating thousands of individually permitted requests. In a delegated workflow, the children must share or draw down the parent’s budget rather than each receiving a fresh unlimited allowance.

Enforce those limits outside the model. An instruction to “stop after five attempts” is not a quota. A dashboard that reveals yesterday’s spend is not a preventive control.

Define degraded operation before an incident. A support service might fall back to search or a human queue. A consequential write operation might pause until its policy service is available. A physical process may require a separately engineered safe operating mode rather than an indiscriminate shutdown. The cyber-kinetic risks of autonomous systems make this last distinction the one to get right first: losing intelligent assistance must not mean losing the independent mechanisms that keep the process safe.

The supply chain is larger than the weight file

A clean hash answers a narrow question: are these the bytes associated with the expected digest? A verified signature can establish that a particular signer endorsed an artifact. Neither answers whether the artifact’s behaviour is acceptable.

A malicious publisher can sign malicious material. A compromised build process can produce a correctly packaged release. A model with an unwanted learned behaviour can be perfectly intact. Provenance is necessary evidence about origin and integrity; it is not a behavioural warranty.

The s1ngularity compromise of Nx on 26 August 2025 illustrates a different supply-chain route: a post-install script in a malicious npm package harvested credentials and published them through victims’ own GitHub accounts, and along the way drove locally installed AI coding assistants, with their permission checks disabled by command-line flags, to find more files worth taking. The companion article gets the mechanism right, because it has been overstated; the lesson for a release process is that a developer’s tools and ambient credentials are part of an attacker’s effective reach.

For models, maintain an approved release manifest covering the weights, adapters, tokenizer, configuration, prompt templates, relevant preprocessing and postprocessing, runtime image and executable extensions. For an agent application, add tool definitions, policy versions and the retrieval configuration. Record where the artifacts came from and what was evaluated together.

Pin identities precisely. A model family name or a floating repository tag is not a reproducible release. Restrict who can promote an artifact into production, and separate that authority from the ability to upload an experimental checkpoint. Verify integrity when artifacts enter the approved repository and when they are deployed.

A software bill of materials remains useful, but record the AI-specific relationships too. CycloneDX’s machine-learning bill of materials provides a way to represent models and related information alongside software components. The value is the inventory and traceability it supports, not a claim that a populated bill of materials proves the training data was trustworthy.

For training organisations, preserve dataset versions, contribution routes, transformations, filtering decisions and the relationship between data, runs and checkpoints. Protect evaluation sets against contamination and unauthorised modification. For consumers, request the evidence the supplier can actually provide and record the remaining visibility limits. A supplier’s failure to disclose a complete corpus does not automatically establish misconduct; it does mean the consumer cannot claim to have independently audited that corpus.

Weight confidentiality is a separate decision. A proprietary model or fine-tuning adapter may be a valuable asset; a publicly distributed base model is not made secret by downloading it into a private network, and a copied checkpoint cannot be uncopied by any measure taken against the building it came from. Weight exfiltration is an infrastructure and insider problem, not a machine-learning one. Protect the sensitive parts of the deployment accordingly, including training data, private adapters, evaluation material and credentials.

RAND’s Securing AI Model Weights, published in May 2024, develops threat models for frontier-weight theft across attacker capabilities. Its emphasis on limiting access, controlling copies and combining defensive measures is useful, and the EU AI Act’s Article 55 obligations on frontier providers have since turned that frame into a mandate, with the AI Office requesting evidence from August 2026. Its most demanding scenarios should not be turned into a universal compliance requirement for every organisation running a small public model. Scale the protection to the asset and adversary, not to the prestige of the phrase “frontier security.”

Finally, retire what you no longer operate. An obsolete checkpoint in a developer bucket, an old inference endpoint and a forgotten adapter are still assets or attack surfaces. Model sprawl is a lifecycle problem: approval, deployment and decommissioning must refer to the same inventory.

Privacy needs its own threat model

“Encrypted,” “private” and “not used for training” answer different questions. None should be accepted as a complete description of an AI service’s data exposure.

Map where sensitive information enters, where it is processed, who can access it and which copies persist. Include prompts, attachments, retrieved passages, outputs, caches, observability systems and human review. For a hosted service, distinguish the contractual treatment of customer data from the application’s own decisions about what to send. For a self-hosted service, do not mistake local processing for adequate access control.

Start with minimisation. A component that only needs a transaction category should not automatically receive the full customer record. A drafting task may not need historical messages unrelated to the request. Keeping unnecessary sensitive material out of the workflow reduces the information available to both an attacker and an accidental disclosure. Masking at the query layer protects the view, not the model: whoever builds the training set reads through the mask or under it.

Differential privacy addresses a more specific problem: the influence of an individual’s data on the output of a mechanism. NIST SP 800-226, the March 2025 guidelines for evaluating differential-privacy guarantees, explains why the protected unit, privacy parameters, implementation and accounting decide what a guarantee means. An organisation should ask what is protected, whether one record, one event or one person, and how repeated use is accounted for. The guarantee is not that the model can reveal nothing sensitive. It is also not a defence against an agent forwarding a confidential document placed directly into its current context. The same applies to the other privacy-enhancing technologies: federated learning keeps raw data on the device and leaves the question of what the aggregation server may see, and secure multi-party computation hides the inputs to a computation while revealing its output exactly as computed.

Unlearning requires equally precise language. In Do Unlearning Methods Remove Information from Language Model Weights?, first published in October 2024, Deeb and Roger found that fine-tuning on facts from the same distribution recovered 88% of pre-unlearning accuracy on information the tested methods were supposed to have removed. That involved further access to and training of the model, not merely an ordinary end-user prompt. It is evidence against equating the tested methods with irreversible removal, not proof that every possible unlearning approach must fail.

For a deletion requirement, inventory the actual objects: original records, transformed datasets, checkpoints, derived indexes, caches, backups and logs. Establish what has been deleted, what remains, and what evidence supports any claim about learned information. Suppressing a response is not the same operation as deleting a source record, and deleting a source record does not necessarily alter a model already trained on it.

This is also why encryption and authorisation should not be conflated. Protecting a communication channel does not decide whether its recipient should receive the plaintext. Protecting a stored model artifact does not decide whether the running application should answer a query. Homomorphic encryption lets an untrusted server compute on data it cannot read; it does not decide whether the computation should run. Privacy depends on the complete data path and its purpose, not on a single cryptographic or model-level feature.

Test whether the deployment survives a fooled model

The most useful AI security evaluation is not the one with the most dramatic screenshot. It is the one that tells an owner what can happen, under which conditions, despite the controls they intend to operate.

For a consequential deployment, I would require three complementary forms of evidence: conventional system testing, adversarial model-and-workflow testing, and direct tests of the containment boundaries. They answer different questions.

Separate realistic exploitation from defence-aware stress testing

A realistic test starts with the attacker’s actual foothold. Perhaps they can send an email but cannot modify the system prompt. Perhaps they can upload a customer document but cannot read the retrieval index. Perhaps they can query a classifier only through a rate-limited endpoint. Preserve those conditions, including authentication, preprocessing, permissions and network restrictions. Every deployment is grey-box from somewhere, and the test report has to say what the attacker held.

A defence-aware stress test deliberately gives the attacker more knowledge or feedback. It asks whether the defence depends on obscurity, whether it withstands adaptation and where it starts to fail. Such testing can reveal important weaknesses even when the test access is stronger than the ordinary external threat. The Attacker Moves Second, published in October 2025 by Nasr, Carlini and twelve co-authors from OpenAI, Anthropic and Google DeepMind, is the reference for what that finds: twelve published jailbreak and prompt-injection defences, most bypassed above 90% once the attacker adapted, most of which had reported near-zero. The companion article reads the result in full. For a test plan the lesson is narrower: a static regression suite checks that yesterday’s fixes still work while adaptive testing asks what an adversary can do after learning how those fixes work, and the adaptive test is the one ML detection never ran. Both belong in the programme, and neither should be presented as evidence it does not provide.

Do not blend the two into one percentage. A test requiring internal detector scores should say so. An attack that works only after removing the production gate should not be reported as a bypass of that gate. If the model refuses a malicious tool request, that result does not test whether the downstream authorisation control would have stopped it.

Test the paths by which untrusted material actually arrives: documents, tool results, source-code comments, attachments, rendered pages and persistent state. Include the transformations in between. A filter that examines extracted text may not see the same representation as a multimodal model. A document that is harmless before summarisation may be represented differently afterwards.

Use fresh attack development as well as historical cases. Keep the regression corpus, but do not let it become the entire definition of an adversary.

Measure the business outcome, not only the model’s sentence

For the invoice workflow, a sentence proposing the attacker’s account is evidence of model manipulation. An unauthorised payment instruction accepted by the executor is evidence of a boundary failure. Money actually moving is a further consequence. Record the distinctions.

Likewise, “the model attempted to exfiltrate” is not equivalent to “confidential data reached an unauthorised destination.” A policy gate may have stopped the operation. Conversely, a reassuring model response does not establish that no harmful tool call occurred earlier in the trace.

Use controlled environments, synthetic records and canary data to test complete paths without exposing real customer information or making real irreversible changes. Validate outcomes from system events and destination receipts where possible. Model-based judges can help triage test outputs, but manually validate serious findings and inspect tool arguments and effects rather than relying exclusively on a generated verdict.

Report task utility alongside security. A defence that prevents every payment by rejecting every invoice is not a functioning payment assistant. Measure legitimate completion, inappropriate refusals, human-review burden, latency and operating cost. The relevant trade is secure usefulness, not the highest isolated blocking rate.

Deliberately substitute a hostile model output

Here is a test I would add to almost every tool-using deployment: bypass the difficulty of persuading the model and deliberately supply an adversarial proposal at the model-to-tool boundary in a test environment.

Ask the executor to perform the wrong action, on the wrong object, for the wrong user, with the wrong recipient or after an approval has expired. Supply a schema-valid request with a malicious argument. Try a stale resource version. Attempt to replay an approved operation. Attempt to exceed the task budget. Ask a child agent to act outside the parent’s delegation.

The expected result is a policy rejection, regardless of how convincingly the model justifies the request.

Call this forced-compromise testing: assume the model has already failed and test what is left. It is a proposed practical testing pattern, not a claim that the deployment has been formally verified. It does not replace prompt-injection testing, because a real attack may find a different path through the application. It does test the central containment claim without confusing that claim with the model’s willingness to cooperate.

For the payment example, a malicious bank-account value should fail even when it comes from the most trusted model in the approved catalogue. The executor should have no reason to care which model proposed it. It is also the oldest test I know. In the early 2000s at CyberAgency, when defence clients asked my team to break the AI they intended to put into weapons systems, what they were paying to learn was what the system did once the model had been beaten.

Use numbers that keep their assumptions

An attack-success percentage is not a property of a model alone. It reflects the tasks, inputs, access, feedback, defence configuration and resources used in the test, which is why Article 15 of the EU AI Act requires a robustness nobody can measure: no threat-model-independent robustness metric exists.

An evaluation report should identify the release, tasks, number of trials, attack access, feedback, query or compute budget, success criterion and significant limitations. Report severe subsets separately. An average across harmless requests can conceal a dangerous failure rate on the few operations that count, in the same way that an average false-match rate conceals a demographic differential in exposure to impostors.

Repeated opportunity also counts. As a mathematical illustration, an independent 1% chance of success per attempt produces about a 63% chance of at least one success across 100 attempts: one minus 0.99 raised to the hundredth power. Real attacks are rarely independent or identically distributed, so this is not a deployment estimate. It shows why a per-attempt percentage without exposure and retry assumptions is an incomplete risk statement.

Equally, zero successes in a finite test does not demonstrate a zero failure probability. And a measured attack cost is not a guaranteed lower bound on the cost of a strategy nobody has tried.

Make the release decision against explicit criteria. Some findings should block deployment until a boundary is repaired. Others may be acceptable within a limited, monitored use case. Record which authority accepts the residual risk, which assumptions support that decision, and which changes require it to be revisited.

Operate the whole release, not only the model endpoint

Passing an evaluation is a release event, not a permanent property of the application. A different prompt, tool permission, data source or fallback route can invalidate the assumptions under which the test passed.

The NCSC’s guidelines for secure operation and maintenance of AI systems, part of the November 2023 secure AI system development guidelines, treat monitoring, updates and information sharing as continuing parts of AI security. For the operator, the challenge is to make that lifecycle refer to the deployment that actually exists.

Version the security-relevant configuration

A security review that records only the model name misses much of the deployed behaviour. The same weights can be deployed behind different system prompts, tokenizers, retrieval indexes, tool descriptions, generation settings and approval policies. A fine-tuning adapter changes behaviour without replacing the base model, and a quantised derivative of a model you scanned is a different artefact that you did not scan. Treat those elements as a release, not as miscellaneous configuration; OWASP’s secure AI model operations cheat sheet places model artifacts and their lifecycle inside operational security for precisely this reason.

Record the model version or provider identifier together with the prompt, tool schemas, retrieval configuration, policy bundle and runtime dependencies. Tie evaluation results to that manifest. Keep the ability to reconstruct what was deployed when an incident occurred.

Where a hosted provider exposes only a changing alias, record that limitation. Ask for change notifications and available version controls. Run representative canary tests to identify behaviour changes, and define a safe response when the provider changes something the organisation cannot pin. A fallback to another model must be an approved configuration, not an untested escape route activated precisely when the system is under stress.

Use risk-based regression. A typographical correction to a displayed label does not necessarily require a full adversarial campaign. A change that grants repository write access, adds a new tool, modifies instruction priority or expands the retrieval corpus needs the full pass. Make the triggers explicit so that “small configuration change” cannot become a route around security review.

A specific finding can be closed by a model update, a prompt change, a retrieval restriction or an application fix. Closing it is not eliminating the class. Keep both kinds of work: remediate specific findings, and maintain a residual-risk assessment for the broader class that a model change can reopen. Evaluate model changes for security regressions as well as improvements; a fix that blocks one attack while degrading legitimate decisions elsewhere still needs a release decision.

Retain rollback options, but distinguish code rollback from state recovery. Reverting a model does not undo a payment, retract a disclosed document or remove contaminated memory. A release plan must address the effects that persist after the release itself is withdrawn.

Log the chain that could explain the incident

A transcript containing only the user’s question and the assistant’s final answer may omit the attack itself. The injected instruction could have arrived in retrieved content, a tool response or a previous memory entry. The harmful operation might have occurred between two otherwise reassuring messages.

Build a correlated record of the user and workload identities, delegated task, retrieved object identifiers and versions, relevant inputs, model and configuration version, proposed tool calls, policy decisions, approvals and actual execution results. Include outbound destinations and the material state changes needed to determine impact.

The aim is reconstruction, not indiscriminate collection. Raw prompts and retrieved content can contain personal data, confidential business information or secrets. Decide what content is necessary, how long it is retained, who may access it and how it is protected. A restricted evidence store with references in ordinary operational logs may be more appropriate than copying everything into a broadly accessible monitoring platform.

A hash can identify a document version, but it cannot reconstruct a document you no longer possess. Conversely, storing every version forever creates a new sensitive-data repository. Resolve that trade before an incident, according to the workflow’s investigative needs and data-handling obligations.

Do not treat the model’s explanation as a definitive causal trace. Anthropic’s April 2025 research on reasoning faithfulness found that generated reasoning omitted influences on the answer in the tested settings. Explanations may be useful investigative material, but externally recorded inputs, policy decisions and system effects provide a different kind of evidence.

Contain the authority, then clean up the state

An AI incident playbook should identify the controls that stop actions without asking the agent to cooperate. Disable the executor, revoke or restrict the relevant credentials, block the necessary network paths and suspend affected jobs. Account for queued work, retries and delegated subtasks; stopping the conversational interface may leave those running.

Then scope what happened. Identify the entry point, the material the model consumed, the permissions available, the operations attempted and the operations completed. Distinguish a model failure caught by an enforcement layer from a completed disclosure or unauthorised change. Both can justify investigation, but they should not be reported as the same outcome. In Europe the same event can now fall under four reporting regimes with different clocks, with Cyber Resilience Act reporting starting on 11 September 2026.

Quarantine affected source documents, indexes, memories, cached summaries and artifacts. Preserve evidence before destructive cleanup. Restore approved configuration and state, retest the harmful path and verify that any downstream changes have been reconciled. Credential rotation does not repair a poisoned knowledge base; removing a poisoned document does not revoke a stolen token.

Exercise this with the teams that own the actual systems. The model provider may be able to help analyse behaviour while being unable to halt your payment queue. Your security operations team may be able to isolate a host while lacking permission to withdraw a generated customer communication. The playbook needs those owners and permissions before the incident creates urgency.

There is one additional trap: using a privileged security agent to investigate attacker-controlled evidence. Logs, malicious documents and extracted messages remain untrusted when they enter an incident-response assistant. Give that assistant a protected analysis role; do not automatically combine the evidence reader with unrestricted authority to disable accounts, execute commands or change containment policy.

What guardrails can legitimately promise

The evidence does not justify either extreme: that model-level defences solve the problem, or that they are useless because an adaptive attacker can sometimes defeat them.

Safety training, injection classifiers, output filters and model-based review can reduce risk. They can block common attacks, increase effort, produce useful signals and lower the number of dangerous proposals reaching an execution boundary. Their value depends on the attack model, operating threshold, false-positive cost and what happens after a miss. The best deployed defence is not a robust model; it is a model whose failures are bounded by something that is not a model.

There is continued empirical progress. In Constitutional Classifiers++, published on 8 January 2026, Anthropic reports a two-stage classifier cascade that cut compute cost 40 times against its own baseline exchange classifier while holding a 0.05% refusal rate on production traffic, and that survived more than 1,700 hours of red-teaming without a universal jailbreak against its eight target queries. That is a provider-reported result in a particular safety setting, against a particular threat, measured by the provider. It should neither be dismissed because it uses classifiers nor expanded into proof that an enterprise agent cannot be redirected through a malicious invoice.

Buy a classifier as a classifier. Buy an artifact scanner for the artifact threats it actually inspects. Buy a policy-enforcement layer for the operations and data flows it actually controls. A product can be valuable without being a complete AI security programme.

Ask the supplier to demonstrate failure handling, not only detection. When the classifier misses, what still prevents a forbidden action? When the policy service is unavailable, does execution pause or bypass it? When a tool changes its schema, is the change reviewed? When the model is replaced, which evaluations are rerun? Can you export the evidence needed to investigate an incident?

The strongest product demonstration is not always the most spectacular blocked prompt. It may be a mundane unauthorised request that the executor rejects, even after every model in the workflow has agreed to perform it.

The opposite error, treating the new failure paths as irrelevant because “it is just software”, costs more, and the companion article takes the five dismissals in turn. The short form: an authenticated session can carry an indirect injection, a syntactically valid request can ask for an unauthorised disclosure, an internal document can originate with an adversary. The attacker’s side has already industrialised, with one instruction sent a hundred thousand times and AI changing the economics for attackers faster than for defenders. Ordinary security remains necessary in every one of those cases. Each identifies where a familiar control must be applied to a less familiar path, and the correct response is to make application security and adversarial-machine-learning expertise examine the same workflow, with the same identities, inputs and consequences.

Who owns the deployment, and the decision to launch

The organisational risk is not that nobody owns anything. It is that each team owns a component while the harmful consequence crosses all of them: data science selects the model, platform engineering operates inference, application teams own prompts and tools, data owners control the source systems, identity teams manage entitlements, security evaluates threats, and an agent crosses every one of those boundaries in a single task. The case for a named AI security owner does not require a new executive title; it requires accountable authority over the deployment decision, and somebody with the authority to stop a deployment that does not meet its agreed conditions. The gap between stated AI principles and operating reality is widest here.

For a consequential workflow, I would require a short deployment dossier that an owner can actually read:

Evidence	The question it must answer
System and authority map	What can the system read, change, disclose, spend or delegate, and for whom?
Approved release manifest	Which model, code, prompts, tools, data configuration and policies were evaluated together?
Threat model and test results	What could the relevant attacker control, what happened under testing, and what remains uncertain?
Enforced boundaries	Which consequences are prevented independently of model behaviour, and how was that verified?
Operating and recovery plan	Who monitors, who can stop execution, what evidence survives, and how are affected systems restored?
Time-bounded risk decision	Who accepted the remaining exposure, under what limits, and what triggers review?

That dossier is more useful than a single green “model approved” label. Model approval can be an input to deployment approval; it cannot replace it.

Frameworks help organise the evidence. MITRE ATLAS provides an AI-focused adversarial knowledge base, while ATT&CK remains relevant to the surrounding intrusion paths. OWASP’s Top 10 for LLM Applications 2026, published on 4 August 2026, and its Top 10 for Agentic Applications, published on 9 December 2025 with entries ASI01 to ASI10, are complementary views of application and agent risk. None of them is evidence that a mapped control works.

Keep regulatory analysis in a controlled compliance register, tied to the organisation’s actual role, system and jurisdiction, rather than in a security article. The EU AI Act dates as they stood on 7 September 2026 are in the companion article and in what a security team has to build before December; a compliance file is not a containment boundary, and compliance evidence, governance evidence and adversarial-resilience evidence should support each other without being mistaken for one another.

The question at the release meeting is not merely whether the documents exist. It is whether those documents demonstrate that the deployment remains within the authority the organisation intended to grant.

What to do on Monday

Start with concrete exposure and consequential workflows, not a plan to classify every possible AI risk before fixing anything.

First, find the software and the authority. Inventory inference endpoints, model registries, retrieval stores, tool servers and autonomous jobs. Alongside versions and network exposure, record the identities and privileges they can use. An ordinary asset inventory will tell you where the service runs; the AI inventory must also tell you what decisions and actions it can initiate.

Second, close demonstrable boundary failures immediately. Remove unauthenticated administrative exposure. Patch vulnerable infrastructure. Eliminate unrestricted loading of untrusted model artifacts. Remove credentials and write permissions that a workflow does not need. Pause consequential autonomous actions that have no independent authorisation boundary. Do not wait for a specialist model evaluation to address a permission that is plainly excessive.

Third, take one high-consequence workflow end to end. Map its entry points, retrieval, memory, tool calls, approvals and downstream effects. Define the invariants. Implement the missing enforcement. Run both realistic adversarial tests and forced-compromise tests. Make that deployment, not a generic slide deck, the pattern other teams can reuse.

Fourth, make the evidence and the operating model real. Tie tests to release manifests. Capture the necessary trace. Assign incident actions to people who can perform them. Practise stopping execution and restoring state. Obtain a written, bounded acceptance of the residual risk before expanding autonomy.

Over the following release cycles, extend that pattern across the estate, prioritising sensitive data, irreversible actions, shared infrastructure and difficult recovery. Retire unsupported deployments. Add security tests when tool permissions or data reach expand, not only when model weights change.

Measure progress in terms a business owner can understand: consequential workflows with enforced action boundaries, known release configurations, tested revocation, usable evidence and rehearsed recovery. A growing catalogue of prompts that a guardrail blocks is useful supporting information. It is not the outcome the programme exists to deliver.

What is left when the model has been fooled

Since I first ran adversarial testing against AI systems in the early 2000s, the test has been the same: not whether the model looks impressive under normal conditions, but what happens when someone deliberately works against it. The field’s founding document asked something similar in 1956 and kept neither of its failure questions. The surrounding systems have changed, and language interfaces have made it easier to connect learned behaviour to organisational authority. A security programme still has to identify the adversary, understand the failure path and limit the consequences.

We should resist the temptation to demand a perfect model before deploying anything useful. We should resist just as strongly the temptation to treat usefulness as evidence that a model should be given broad, unattended authority.

A model may misunderstand a document, reveal memorised information, accept a malicious premise or propose an action it should not perform. Security work should make those failures harder. Architecture should decide what they are allowed to become. Operations should make them visible and recoverable.

The invoice can be convincing. The retrieved document can be relevant. The tool call can be valid. The model can explain, at length, why it believes the action is appropriate.

Not one of those facts should be enough to authorise the payment.