What AI Red Teaming Actually Is

The deployed model at a named version or snapshot, with its decoding parameters — temperature, top-p, maximum tokens — and whatever safety training and system-level filtering sit around it.

Whether the model can be induced to produce output its operator would not sanction: instructions it should refuse, content it should decline, claims it should not make. Also whether it behaves differently under the same input, which is the property that makes everything else on this page harder.

A benchmark score. How the model ranks on a public evaluation set is a statement about the model. It is not a statement about your deployment, your system prompt, your data or your tools.

The exact input, the raw output, the model identifier and snapshot date, the decoding parameters, and the number of trials the behaviour was observed in.

The system prompt, the instruction hierarchy, the conversation state carried between turns, and everything the application concatenates into the context window before the model sees it — user text, retrieved passages, tool results, metadata, prior turns.

Whether the boundary between instruction and data holds. The model sees one flat sequence of tokens, while the application believes it sent a trusted instruction followed by untrusted content. Every attack in the prompt-injection family lives in the gap between those two beliefs.

Input validation in the conventional sense. There is no grammar to validate against, because the interface accepts natural language by design. Escaping is not available, so the control has to be architectural.

The assembled context as the model received it, not as the application intended it — which usually means logging the final serialised prompt, and which is usually the first thing an engagement discovers nobody logs.

The corpus, the ingestion path, the chunking and embedding step, the index, and the retrieval policy that decides which passages enter the context for a given query. Ingestion is rarely text-only: where the system accepts images, audio or uploaded documents, a payload can arrive as rendered text inside an image, as speech, or as text that is present in a PDF and invisible to a human reader — white-on-white, zero-point, or behind an overlay.

Two things at once. Whether a document placed in the corpus by someone other than the operator can steer the system — indirect prompt injection — and whether retrieval respects the permissions of the person asking, or merely the permissions of the service account that built the index. A third, where the corpus is writable: whether a planted document can change what the system asserts as fact, which is a different failure from planting an instruction.

Document-level access control on the source system. A corpus indexed once with a privileged service account has already flattened the permission model, and the source system has no idea it happened.

The passage that was retrieved, the query that retrieved it, the identity the query ran as, and the path by which the passage entered the corpus.

Every function the system can call, the credentials it holds to call them, the parameters the model controls, and the approval steps — if any — between a generated intention and a real effect. Two things belong in this layer that are easy to miss. A tool description is text the model reads, so wherever a tool definition is supplied by a third party — a connector, a plugin, a server the application trusts — the description is attacker-influenceable content that enters the context. And where the system delegates to sub-agents, each delegation is a fresh trust boundary with its own tools and its own context.

What the system can do, as distinct from what it can say. The question is the blast radius of a single successful injection: which tools are reachable, with whose authority, against which data, and whether any of it is reversible. Where sub-agents exist, ask the same question of each one and then of the composition, because the parent agent rarely constrains what the child was given.

A permissions review of the service account in isolation. The interesting property is the composition — a read tool and a write tool that are individually reasonable become an exfiltration path when a model that can be steered holds both.

The tool schema exactly as the model received it, the tool-invocation log with arguments, the identity each call ran as, the side effects produced, and whether a human approval was requested, shown accurately, and acted on.

Wherever generated text goes next: a browser that renders it, a shell that executes it, a database that receives a generated query, a downstream service that parses it, a document a person will act on.

Whether the consumer treats model output as trusted. Generated content is attacker-influenceable content, and every classical injection class re-enters here through a channel the application team did not think of as an input.

A finding about the model. This class is a finding about the application around it, and it is usually fixed there — which makes it the most tractable material an engagement produces.

The generated string, the sink it reached, and a demonstration of what the sink did with it.

The operators, reviewers and approvers the design relies on: the person who clicks approve on a tool call, the analyst who reads a summary, the agent who forwards a generated reply to a customer.

Whether the oversight the design assumes actually exists in the workflow as performed. An approval step that presents twenty calls an hour with a truncated argument list is a rubber stamp with a log entry, and that is a finding about the control rather than about the person.

A social-engineering exercise against staff. Testing the oversight design is not the same as testing the people, and the two have different consent and reporting rules — see the note on §15.2.2(iii) below.

The approval interface as the approver sees it, the rate of requests, what is truncated or hidden, and what the log records when approval is granted.

The person talking to the system supplies text that the model treats as instruction rather than as content: an override of the system prompt, a role reassignment, an instruction to ignore a constraint, a request laundered through a framing the safety training did not anticipate.

A corpus of override attempts, run against every entry point the system exposes — the chat surface, but also any field whose contents reach the context window: a ticket title, a filename, a profile bio, a support form, a calendar invite.

The input, the output, the constraint that was bypassed, and the rate at which it reproduced. A single success is a lead. The reproduction rate is what makes it a finding a team can prioritise.

Rarely a prompt change on its own, because a prompt is a request rather than a control. The durable answers are architectural: privilege separation between the model and the actions, validation at the sink, and a design that assumes the instruction boundary will fail.

The instruction is not typed by the user at all. It sits in content the system ingests: a document in the retrieval corpus, a web page the system fetches, an email it summarises, a code comment it reads, the alt text of an image, a field in a record it was asked to process. The person asking the question is not the attacker, and may be the victim.

Place marked payloads into every ingestion path the system has, then exercise the ordinary workflows that would pull them into context. Cover the non-text channels the enumeration usually misses: text rendered inside an uploaded image, instructions in an audio clip where the system transcribes, and text present in a PDF but invisible to the human who approved it. The test is not whether the payload can be written — it is whether writing it anywhere in the pipeline changes what the system does for somebody else.

The payload, the path by which it entered the corpus or the fetch, the query that pulled it into context, the resulting behaviour, and the reproduction rate. Note who could plant it, because that decides whether the class is remote-unauthenticated or requires an insider.

Provenance and trust labelling on retrieved content, separation of retrieved data from instruction in the context assembly, and — the control that actually holds — refusing to let retrieved content influence tool invocation without a fresh authorisation decision.

Techniques that defeat the model’s own refusal behaviour or an external filter: multi-turn escalation that establishes a premise before asking, role-play framings, encoding and obfuscation, language switching, and payloads assembled across turns so that no single turn looks like a request.

Multi-turn sequences and mutation of a seed corpus, with the conversation state treated as part of the attack surface. Single-turn testing systematically under-reports this class, because the interesting failures need a setup turn.

The full transcript, not the final prompt. A jailbreak reported as a one-line payload is unreproducible by the team that has to fix it, and will be closed as not-reproduced.

Layered: refusal training, an independent output check that does not share the model’s context, and a limit on what a successful bypass can reach. Treat filter tuning as risk reduction rather than as closure.

Recovery of the instructions, examples, policies, keys or connection details the operator placed in the system prompt or the context window, on the assumption that the user cannot see them. The assumption is not sound, and it is a design error before it is an attack.

Direct requests, indirect elicitation, partial recovery across turns, and reconstruction from behaviour where verbatim recovery fails. Score partial recovery honestly: an attacker who learns the tool list and the refusal policy has gained most of the value.

What was recovered, how completely, in how many of how many trials, and what it would be worth to an attacker. A secret in a system prompt is a finding regardless of whether the extraction reproduced, because the exposure is structural.

Do not put secrets in the context window. Everything else in this row is mitigation. That sentence is the control.

Recovery of content the model was trained or fine-tuned on, or of data held in an attached store the model can reach — including, in multi-tenant deployments, content belonging to another tenant that entered a shared index, a shared cache or a shared memory.

Elicitation against known-marked records placed in the training or retrieval set where that is possible, cross-tenant probing where tenancy exists, and memory and session-boundary tests where the system retains state between conversations.

The recovered content, its classification, the identity it should have been confined to, and the boundary that failed. Cross-tenant recovery is the highest-consequence result in this class and is usually a retrieval or caching defect rather than a model defect.

Tenancy enforced at retrieval time against the asking identity, not at index time against a service account. Scrubbing and classification before fine-tuning. A hard rule that memory features are scoped to the identity that created them.

Content written into the retrieval corpus, the fine-tuning set or the feedback loop in order to change what the system asserts rather than to instruct it. The payload is not an instruction and will not read like one: it is a plausible, retrievable false fact — a superseded policy, an invented product limit, a wrong contact address, a fabricated exception to a rule — that the system will later ground an answer on and cite.

Write marked but innocuous false facts through every corpus write path, then ask the ordinary questions that would retrieve them and record whether the system repeats them, cites them, and prefers them over the correct source. Test precedence explicitly: when the corpus contains both the true record and the planted one, which wins, and does recency or chunk position decide it.

The planted content, the write path, the query that retrieved it, the assertion the system made, whether it was cited as authoritative, and the rate. Note who could plant it, because a corpus writable from outside the organisation makes this a remote unauthenticated integrity attack on the answers your staff act on.

Provenance on every ingested item with the writer recorded. Review or quarantine for externally-writable ingestion paths before indexing. Retrieval that prefers a source of record over a general corpus for facts that matter. A periodic integrity check of the corpus against its sources. Removal must be verified by retrieval, because deleting a source document does not always remove it from an index.

A write into persistent memory — a user profile, a long-term memory feature, a shared summary, a cached preference — that takes effect later, in a different session, possibly for a different person. An injection that succeeds once and is written down does not need to succeed again.

Cause a write to memory through an ordinary interaction, then start a fresh session and see whether the effect survives. Where memory is shared across a tenant or a team, test whether one identity can write something another identity later reads. Where the system summarises a conversation into durable state, test whether the summary can be steered.

This is the one class on this page whose finding is not honestly expressed as a rate. A single successful write with a persistent effect is a finding at 1/1, and reporting it as 1/30 understates it by a factor nobody can compute. Record instead: the write, the store it landed in, how long it persists, who can read it afterwards, and what removes it. Say in the report that the rate convention does not apply to this class and why.

Scope every memory to the identity that created it and never to a tenant by default. Treat a write to durable state as a privileged action rather than a side effect of a conversation. Make memory contents inspectable and deletable by the person they belong to. Apply the same provenance labelling to remembered content that retrieved content gets.

The system is able to take actions disproportionate to what the task requires: too many tools, too broad a credential, parameters fully under model control, no approval step, or an approval step that cannot be meaningfully exercised. This is the class that turns a conversation problem into an incident.

Enumerate every tool, the identity it runs as and the scope of that identity, then attempt to reach each one through the model, including through indirect injection. Test composition explicitly — chain a read tool into a write tool and see whether anything stops it. Where tool definitions come from a third party, read the descriptions as untrusted text. Where the system delegates to sub-agents, enumerate what each child was granted rather than assuming it inherited a subset.

The tool reached, the authority it carried, the effect produced or demonstrably reachable, whether it was reversible, and whether any log records who caused it. The severity of this class comes from the effect, not from the elegance of the prompt.

Least privilege per tool with a scoped credential. Human approval for irreversible or high-value actions, with the approval showing the actual arguments. Rate and value limits. An audit trail that ties every effect back to the conversation and the identity that started it.

Model output is passed to a consumer that trusts it: rendered as HTML or Markdown in a browser, executed as code, interpolated into a query, written to a file path, or handed to a downstream parser. Cross-site scripting, command injection and SQL injection all re-enter the system through this door.

For every sink, ask the model to produce the payload the sink is vulnerable to, then observe what the sink does. This is the most classical work in an AI engagement and it overlaps cleanly with ordinary application testing.

Indistinguishable in form from a conventional injection finding, and it should be written as one: the payload, the sink, the effect, the fix. Score it the way the rest of your register is scored.

Treat model output as untrusted input at every consumer. Contextual encoding, allow-lists, and no path from generated text to an interpreter.

Output the operator is answerable for: unsafe instructions, discriminatory treatment across protected characteristics, confidently false statements of fact in a domain where reliance is foreseeable, advice that is regulated in the jurisdiction the system serves, or disclosure that breaches a commitment the operator has made to its own customers. For an Indian operator the answerability is concrete rather than abstract. Where the output discloses personal data, section 8(5) of the Digital Personal Data Protection Act, 2023 requires a Data Fiduciary to protect personal data in its possession or under its control by taking reasonable security safeguards to prevent a personal data breach, and section 8(6) attaches an intimation duty when one happens. Where the domain is regulated, the operator is answerable to its own regulator for what its system told a customer — this page names no regulator and no sectoral rule, because no instrument it has read says one applies to AI output by name. And where the operator has made a contractual commitment to its customers about confidentiality or accuracy, the commitment binds the system too.

Scenario sets drawn from the system’s actual use, not from a generic harm taxonomy. The useful question is what this system is relied on for, and what the worst defensible output in that context would be. Draft the scenarios with the person who owns the obligation — legal, compliance, or the product owner who signed the customer commitment — rather than with the testing team alone.

The scenario, the output, the obligation or commitment it cuts against, and the reproduction rate. This class needs a reviewer who understands the obligation, which is usually not the person running the harness — so name that reviewer in the finding, by role, next to their judgement. A harmful-generation finding graded only by the tester is a technical result about a legal question, and it will be disputed by the first lawyer who reads it.

Scope restriction, refusal policy, retrieval grounding with citation, and — where the stakes justify it — human review before the output reaches the person who will rely on it. Where personal data is in play, the control is upstream: do not put it in the context window in the first place.

Inputs that cause the system to spend disproportionately — very long contexts, recursive tool chains, retrieval that fans out, an agent that loops, or a prompt that induces maximum-length generation on every turn. On a metered endpoint the consequence is a bill, and on a shared deployment it is availability for everyone else. It is the one class on this page where the attacker needs no secret and no privilege, only patience.

Measure cost and latency per request under ordinary use to establish a baseline, then attempt the amplification patterns: maximal context, induced recursion, a tool chain that calls itself, an input that defeats caching. Run it inside the agreed spend cap — this class is tested by finding the multiplier, not by exhausting the budget to prove a point.

Tokens, calls, wall time and cost for the ordinary request and for the amplified one, expressed as a multiplier, with the identity needed to send it. A 40× multiplier reachable unauthenticated is a different finding from a 3× multiplier that needs a logged-in account.

Per-identity rate and token budgets enforced server-side rather than in the client. A hard ceiling on context length, generation length and tool-chain depth. A loop detector on agent runs. Billing alerts that fire on rate of change rather than on a monthly total, because a monthly total arrives after the money has gone.

Where the model, its adapters, its embeddings, its plugins and its datasets came from, whether their integrity can be demonstrated, and whether the version running in production is the version that was tested.

Largely documentary and configuration work rather than prompting: provenance records, integrity verification, pinning, and a comparison between the manifest and what is actually deployed.

An unpinned or unverifiable component, or a production version that differs from the tested one. The second is the one that quietly invalidates every other finding on the report.

An inventory of AI components with provenance — the ground CERT-In’s §6(xxvi) AIBOM class and its BOM technical guidelines occupy — plus pinning, integrity checks and a change process that treats a model version bump as a change.

The report contains no tool-invocation log, and the limitations paragraph does not name the tools that were stubbed.

Read the limitations paragraph before the findings. If it does not list which tools were live, which were stubbed and what the test tenancy contained, the excessive-agency class is unexamined regardless of how many prompts were run.

The words “intermittent”, “occasionally” or “sometimes” appear in a severity-bearing sentence.

Search the register for a finding without a successes-over-trials figure. Every one you find is an anecdote that has been formatted as a finding, and it will be the first thing disputed.

Negative statements in the report carry no trial count.

Put the trial count next to every negative finding, then read the arithmetic table on this page and decide whether the count supports the sentence. Usually the honest fix is to change the sentence rather than to run more trials.

The report names the model but not a version, snapshot or date.

Capture the manifest at the start and at the end of every engagement and compare the two. It costs an hour and it is the difference between a result and a rumour.

The deliverable is a score. There is no transcript archive and no tool log.

Ask which of the six layers were touched. A deliverable that cannot answer that question is a model evaluation, which is a useful procurement input and not an assurance artefact.

The scope names a chat interface and the ingestion paths are not enumerated anywhere.

List every path by which text can enter the context window and mark who can write to each one. Any path writable by a person outside your organisation is a remote unauthenticated attack surface, and it is usually a surprise.

Every remediation in the closure column is an edit to the same file.

Classify each fix as architectural or behavioural, and re-test the behavioural ones at a real trial count. Then ask the question that settles it: does this fix survive a model version change? If the answer is no, it is mitigation and belongs in the residual-risk paragraph.

Nobody can produce a diff between the staging and production guardrail profiles.

Diff the two manifests before testing starts. The differences you find are either corrected or written into the limitations paragraph. There is no third option that leaves the report honest.

The control is evidenced by a screenshot of the dialog rather than by a rate and a field list.

Sit with the approver for an hour. Count the requests, note what the dialog hides, and write what you find as a control finding with an owner.

The contract is silent on ownership of the test corpus.

Put corpus handover in the statement of work, in a runnable form. It is the artefact with the longest useful life, and it is the one that turns an annual exercise into a control.

There is no removal list, and no verification pass after cleanup.

Maintain a written inventory of every payload planted, with its location, and verify removal by retrieval rather than by deletion receipt. Deleting a source document does not always remove it from an index.

Remediation dates slip uniformly and no single person can describe the end-to-end system.

Name one accountable owner for the deployed system before the engagement starts. §9.2 of the 2025 guidelines is a useful lever here by analogy: risk treatment and exceptions are authorised and accepted by the head of the auditee organisation, not distributed across the teams that own the parts.

The transcript archive shows one session identifier across the whole trial set, or the method note is silent on caching, or the observed rate is suspiciously stable across classes that should behave differently.

Require a fresh session per trial, disable prompt caching and provider-side response caching for the run, and record in the finding which of those you did. Where the platform will not let you disable caching, say so in the limitations paragraph and treat the rate as a lower bound on variability rather than a measurement of it.

The report contains reproduction rates and no grading criterion, or the criterion appears in the report and not in the test plan, or two findings in the same class are graded inconsistently.

Ask for the success criterion for any class whose rate you intend to act on, and ask who graded, at what sample a person checked the machine grading, and what the agreement rate between two graders was. A criterion two people apply differently is not a criterion, and the rate built on it is a count of one person’s impressions.