Red teaming an LLM application means adversarially testing the whole system — the model, the instructions wrapped around it, the documents it retrieves, the tools it can call, the identity it acts under and the places its output lands — rather than testing the model on its own. This is the practitioner page: what to pin down before the first prompt, the attack classes worked through concretely, and what evidence looks like when the system under test does not give the same answer twice.
The hard part is not getting the model to misbehave. It is proving it later. A stochastic system needs a different evidence standard from a web application: the exact prompt, the model version, the sampling parameters, the corpus snapshot, and a reproduction rate with its denominator — not a screenshot.
Practitioner guide · CERT-In Comprehensive Cyber Security Audit Policy Guidelines v1.0 (25 July 2025) · CERT-In Directions of 28 April 2022 · OWASP Top 10 for LLM Applications, 2026 edition · Clause locators and URLs checked 22 September 2026
How do you red team an LLM application? Test the system, not the model
Red teaming an LLM application is adversarial testing of the whole system around a model — the instructions wrapped around it, the documents it retrieves, the tools it can call, the identity it acts under and the places its output lands — rather than testing the model alone.
You scope nine properties of that system: the model and its version, the system prompt, the retrieval corpus and its trust boundary, the tool and function surface, the identity the system acts under, memory and session state, the humans in the loop, the output sinks, and the environment you were given with its differences from production.
Then you work ten attack classes against them — direct injection, indirect injection, guardrail bypass, system-prompt extraction, corpus leakage, corpus and fine-tune poisoning, excessive agency, supply chain and tool provenance, unbounded consumption, and insecure output handling — and record every finding with enough configuration detail that somebody else can reproduce it on a system that never gives the same answer twice.
The two ideas that make the difference between a useful engagement and a demo are these. First, the interesting instructions do not arrive through the chat box: they arrive inside a document the application retrieves or inside the output of a tool it called, placed by somebody who never touched your product. Second, the finding that matters is not that the model said something it should not have said, but that it did something — called a tool, reached another tenant’s data, emitted markup a browser executed. OWASP’s Top 10 for LLM Applications, 2026 edition, published on 4 August 2026, puts Prompt Injection at LLM01 and Excessive Agency at LLM03.
The twelve steps, in order
- 01 Name the class of engagement, in writing
- 02 Pin the nine properties of the system under test
- 03 Agree the rules of engagement, including what you may not do
- 04 Take the artefacts before the first prompt
- 05 Establish baselines and record the refusals
- 06 Work the direct channel
- 07 Work the indirect channels
- 08 Work the tool surface
- 09 Work the sinks
- 10 Score, and say what the score does not carry
- 11 Report, including the negatives
- 12 Retest, then set the regression cadence
Five of the twelve happen before anybody types a prompt. Each one is worked through in full further down, at the worked engagement.
In the Indian instruments, this work has a name and a home. CERT-In’s Comprehensive Cyber Security Audit Policy Guidelines (version 1.0, 25 July 2025) list AI System Audits at §6(xxiii), defined as the “evaluation of AI systems for security, ethical alignment, transparency, data integrity, and resilience to adversarial manipulation”, and define a Red Team Assessment at §5(xxiii) and §6(xvi) as “an exercise, reflecting real-world conditions, that is conducted as a simulated attempt by an adversary to attack or exploit vulnerabilities in an enterprise’s information systems”. And under the CERT-In Directions of 28 April 2022, an attack on an AI or ML system is already a mandatorily reportable incident — item (xx) of Annexure I — with the same six-hour clock as every other listed type.
Key Takeaways
- Red teaming an LLM application means testing the system, not the model: nine properties — the model and its version, the system prompt, the retrieval corpus and its trust boundary, the tool and function surface, the identity the system acts under, memory and session state, the humans in the loop, the output sinks, and the environment you were given — worked against ten attack classes. Not every class fires on every architecture, and scoping the ones that cannot is how a buyer pays for nothing.
- The high-value injection channel is indirect. Instructions placed in a document that reaches the retrieval corpus, or in the output of a tool the model called, execute in a legitimate user’s session with that user’s privileges — and the attacker never touches your application.
- Prompt hardening changes the rate, but only the architecture changes the consequence. Least privilege at the tool boundary, authorisation re-checked per call rather than at the chat boundary, schema-validated parameters, human approval on irreversible actions, and encoding at every sink.
- Evidence for a non-deterministic system is a record, not a screenshot: the exact prompt byte for byte, the model identifier and version, temperature and top_p and the seed where one is available, the system-prompt hash, the corpus snapshot, the tool schema, the full transcript including tool calls, and the attempt and success counts.
- Report reproduction rate with its denominator, and choose the denominator from the consequence: twenty fresh sessions for a content-policy bypass, fifty where a success would reach a tool that writes or another tenant’s data.
- A negative result is a claim about an upper bound, so print the bound. Zero successes in twenty fresh sessions is consistent with a true rate of up to about 15% at 95% confidence, and zero in a hundred with a rate up to about 3%. “We tried twenty times and it never worked” is not a statement that it does not work.
- Closure has four outcomes, not two: closed by architecture, closed by evidence at a pre-agreed trial count, mitigated but not closed, or re-baselined because the configuration moved. Only the first survives a model change, and the retest trial count is agreed before the fix ships.
- Score the consequence, not the model’s behaviour. Map the weakness in the surrounding system to a CWE identifier. Mark EPSS “not applicable — no CVE” where the defect is in bespoke code, since FIRST’s own FAQ states that scoring without a CVE identifier is a longer-term direction. §16.2(ii) admits no exception on its face, so treat the stated limitation as our reading of how to answer it honestly rather than as the clause’s words, and agree it before the report is drafted.
- An LLM finding can reopen with no change on your side. A provider-side model update behind an unpinned identifier, a prompt edited outside change management, a new corpus source, a loosened filter or a newly added tool will each do it — so keep every successful payload as a regression case and agree the triggers that re-run the set.
- A ranked list is vocabulary, not an audit criterion — CERT-In’s §8(ii) says so in terms. This page cites LLM01 and LLM03 as shared names and takes the scope from the system. MITRE ATLAS is the better naming layer for the adversary behaviour, and the reason it survives that clause is set out under severity.
- We deliver CERT-In audits, end to end, with our CERT-In empanelled partners.
This page assumes the definitional groundwork is done elsewhere. If you are deciding whether you need this kind of engagement at all, start with what AI red teaming is and AI red teaming versus VAPT. If something you find turns out to be somebody else’s work rather than yours, go straight to AI incidents under CERT-In, because where a reporting duty is engaged the six-hour clock runs from the moment of noticing. Whether a particular fact pattern engages it is a judgement for the auditee and its advisers: item (xx) of Annexure I names a class of incident, it does not adjudicate cases.
Nine properties to pin down before the first prompt
What should the scope of an LLM red team contain?
A conventional application security scope names hosts, URLs and roles. An LLM scope has to name nine things instead, and the fastest way to tell a serious proposal from a demo is to look for them. Each row below gives the property, the question that settles it, the artefact that records the answer, and what the engagement quietly loses if nobody writes it down. Read the last column first: it is a list of the classes of finding that will never appear in your report, without anyone deciding to drop them.
| Property | The question that settles it | The artefact | What is lost if it is unstated |
|---|---|---|---|
| The model, and its version | Which model serves this feature in production, at which published version or snapshot identifier, hosted by whom, in which region, and is the version pinned or does it float with the provider’s default? | A line in the scope naming the model identifier and version string exactly as the API returns it, plus the answer to the pinned-or-floating question in writing. | Everything downstream. A finding recorded against an unnamed model is a finding against nothing: you cannot retest it, you cannot tell whether a later failure is a regression or a different system, and you cannot answer the only question a reader of the report will ask in six months, which is whether it still applies. |
| The system prompt, in full | What is the exact text of every instruction layer the application prepends — system prompt, developer message, per-tenant policy blocks, safety preamble, tool-use instructions — and what is the change history of each? | The full text, under the engagement NDA, with a version or commit reference and a hash the auditee computes and the tester records. | The ability to distinguish a guardrail that failed from a guardrail that was never written. A recurring case is a reported jailbreak that turns out to be a system prompt which never contained the constraint the business believed it contained — a specification defect, with a different fix and a different owner. |
| The retrieval corpus, and its trust boundary | What is indexed? Who can cause a document to enter the index, by what route, with what review? Does the corpus contain anything a customer, a supplier or an anonymous member of the public can write into — support tickets, uploaded files, scraped pages, shared drives, email, CRM notes? | A source-by-source inventory of the corpus with, against each source, the answer to one question: can a party outside your trust boundary put text into this? | The entire indirect-injection class. A test that only types into the chat box tests the one channel the attacker is least likely to use, and reports a clean result for a system whose instructions arrive through a PDF that a supplier uploaded to a shared folder nine weeks ago. |
| The tool and function surface | Every tool, function, plugin, retrieval call, code interpreter and MCP server the model can invoke. Each one’s parameters and their validation. Whether the call is executed automatically or gated by a human, and what each one can do at its most consequential. | The tool schema as the model receives it, exported verbatim, with a column added by the auditee marking every tool that writes, spends, sends, deletes or grants. | Excessive agency, which is OWASP’s LLM03 in the 2026 edition of its Top 10 for LLM Applications and the class most likely to convert a conversational nuisance into a real incident. A tool inventory that exists only in code is a tool inventory that nobody has read as a list of capabilities, which is exactly how a send-email tool ends up reachable from text a stranger wrote. |
| The identity the system acts under | When the application calls a tool or a database, whose credentials does it use — the end user’s, a single service account, or a per-tenant identity? Is the user’s authorisation re-checked at the tool boundary, or was it checked once at the chat boundary and trusted thereafter? | An identity diagram, one line per hop, naming the principal at each hop and where authorisation is enforced. | The confused-deputy class. A single privileged service account behind a chat interface is a deputy with more rights than every user who can talk to it, and the interesting test is not whether the model refuses but whether the back end would have obeyed if it had not. |
| Memory and session state | What persists between turns, between sessions, and between users? Is there a per-user memory store, a summarisation step that rewrites history, a shared cache, a vector store the assistant writes back into? | A statement of every store the model can write to, with its lifetime and its read audience. | Persistence. An injected instruction that survives into memory converts a one-shot trick into a standing instruction, and a memory that is shared across a tenant converts one compromised user into all of them. Test for it deliberately: write, end the session, come back. |
| The humans in the loop | Which actions require a person to approve them, what does that person see at the moment of approval, how many approvals does a busy operator handle in an hour, and can the model influence the text of the approval prompt itself? | A screenshot or wireframe of each approval surface, plus the operational numbers: approvals per operator per day, and median time to approve. | An honest reading of the control the business is relying on most. Human oversight is the mitigation every steering committee reaches for, and its strength is a property of the interface and the workload rather than of the policy. Article 14 of the EU AI Act is drafted around oversight that is actually effective. An approval screen that renders model-supplied text as the description of what is being approved is oversight the model can write. |
| The output sinks | Where does model output go without a human reading it first — a browser that renders it as HTML or Markdown, a shell, a database, a templating engine, another model, a webhook, a ticketing system, a document? | A list of sinks with the encoding or parameterisation applied at each. If the answer at any sink is “none, it is just text”, that sink is a finding waiting to be written up. | The insecure-output-handling class, which is the one place where an LLM finding becomes an ordinary application-security finding with an ordinary CWE and an ordinary fix. Treat model output as untrusted input to the next system, because that is exactly what it is. |
| The environment, and what it lacks | Are you testing production, a staging replica, or a sandbox? Which of the production guardrails are present in the environment you have been given — rate limits, WAF, content filters, the real corpus, the real tool permissions? | A named environment and a written list of the differences, agreed before testing starts. | The report’s meaning. CERT-In’s Comprehensive Cyber Security Audit Policy Guidelines (v1.0, 25 July 2025) require the auditing organisation at §15.2.3(i) to clearly specify the environment — “such as Test, Development, UAT, Pre-Production, or Production” — in which the application was tested. On an LLM system the gap between environments is unusually wide, because a sandbox usually has a smaller corpus, weaker tool permissions and a different content filter, and every one of those differences suppresses a class of finding. |
Two commercial checks belong beside that table and neither requires you to be a tester. The first is §13.2.2(ii) of the 2025 guidelines, which requires the test plan shared by the auditing organisation to state both calendar time and man-hours: divide the man-hours by the number of channels in the next section and the number of tools in the inventory, and you will see what each one is actually getting. The second is the retest. §13.1.1(g) states that the scope of work “should clearly specify the requirement to conduct follow-up audits as part of the engagement” and §9.3(iv) puts them inside the scope or the RFP proposal — and on a system whose behaviour can change without anybody deploying anything, an engagement with no regression cadence attached has bought you a photograph of a moving object. The same discipline, applied to conventional testing, is set out on VAPT scoping.
Where untrusted text actually enters the context
Where does untrusted text enter an LLM application?
Everything the model sees arrives as one sequence of tokens. There is no privileged channel, no type system, and no place in the prompt where the runtime enforces “this part is data”. So the useful engineering question is not whether the model can be tricked — assume it can — but which parties can put text in front of it, and with whose privileges that text ends up being acted on. Six channels, in ascending order of how often they are forgotten.
| Channel | Who can write into it | How it is treated | Why it matters |
|---|---|---|---|
| The chat box | Whoever is logged in | Untrusted, and universally treated as untrusted. | The only channel most test plans cover. It is also the channel where the attacker is authenticated, logged and rate-limited — which makes it the least attractive one for anybody serious. |
| A retrieved document | Anyone who can cause a document to reach the index: a customer raising a ticket, a candidate uploading a CV, a supplier dropping a PDF in a shared folder, a colleague pasting into a wiki | Untrusted in fact. Almost always treated as trusted in design, because the corpus is “our data”. | This is the high-value channel. The text is placed once and fires whenever a retrieval brings it back, against whichever user happens to ask the matching question, with that user’s privileges. |
| Tool output | Whoever controls the system the tool reads from: a web page the browse tool fetches, an API response, a file the code interpreter opens, another agent’s message | Untrusted. Almost never treated as such. | The subtlest of the three, because the text arrives inside the model’s own reasoning loop rather than as a user turn, and most applications concatenate it into the context with no delimiter, no label and no provenance. |
| Persisted memory | Any earlier turn, from any of the channels above | Untrusted, and read back as if it were the application’s own state. | Memory turns a single successful injection into a standing one. Where memory is shared beyond one user, it also turns it into a lateral-movement primitive. |
| A non-text surface the model reads | Anyone who can supply an image, a scanned document, a slide, a screenshot or an audio file to any of the channels above | Untrusted, and usually not described at all — the corpus inventory calls it “attachments” and stops there. | Text rendered inside an image, printed in a scanned PDF, embedded in a diagram or spoken in an audio clip reaches the context as text once the vision or transcription path has done its work. Nothing about that text is visibly instruction-shaped to a human reviewing the file, and an ingestion filter that scans the text layer of a PDF will not see words that exist only as pixels. Where the application has any multimodal input at all, add one line per modality to the corpus inventory and test each as its own channel. |
| The system prompt and tool schema | Your own engineers, and whoever can edit the configuration store they read from | Trusted by design — which makes write access to that store a privileged operation that should be controlled and logged like one. | Worth a line in the scope of its own. A prompt held in a database row that three teams can update is a control surface with no change management on it. |
The one-sentence test for a scope
Ask the team that built the application: name every way a person outside this organisation can cause text to reach the model. If the only answer is “they type it”, either the application genuinely has one channel — which is worth confirming against the corpus inventory rather than accepting — or the scope in front of you is about to test the least interesting third of the system.
The attack classes, worked through concretely
What are the attack classes for an LLM application, and how do you test each one?
Ten classes, each with the mechanism, a sequence you can actually run, the evidence it should produce, and the class of fix that closes it. The sequences describe method rather than payloads: what to vary, in what order, and what to record at each step. A payload list ages in weeks, but the method does not. Not every class applies to every architecture — the archetype table in the next section says which apply to yours.
Two of the ten are cited to OWASP item numbers and the rest are not. That is deliberate. The 2026 edition of the OWASP Top 10 for LLM Applications, published on 4 August 2026, numbers Prompt Injection LLM01 and Excessive Agency LLM03, and those two numbers are the ones we are confident travel correctly between editions. The list carries further categories, for system-prompt leakage, sensitive information disclosure, improper output handling, supply chain, data and model poisoning, unbounded consumption and others. We name those classes here in ordinary words rather than by number, because an item number that moved between two editions is worse than no number at all. The deeper reason to keep the list at arm’s length is CERT-In’s §8(ii), which is set out in full in the instruments table: a ranked list of common categories is vocabulary, not an audit criterion, and the scope here comes from the system rather than from the list.
Direct prompt injection
How do you test for prompt injection?
Instructions supplied by the user in their own turn that displace or override the instructions the application supplied. The mechanism is that there is no privileged channel: the system prompt and the user turn arrive at the model as one sequence of tokens, and precedence between them is a behaviour the model has been trained to exhibit rather than a boundary the runtime enforces.
- Establish the baseline. Ask the assistant, in ordinary language, to do the thing the policy forbids. Record the refusal verbatim. This is the control against which every later result is measured, and it is the step most testers skip.
- Probe the instruction hierarchy. Supply text that asserts a higher-precedence role, that claims the earlier instructions were a test, or that frames the forbidden output as a translation, a summary, a quotation or a code comment. You are not looking for a magic phrase. You are mapping which reframings move the boundary.
- Vary the encoding, not the request. The same request in another language, in base64, split across turns, embedded in a code block, or written as a fictional transcript. Where one encoding succeeds and the plain form fails, the guardrail is matching on surface form rather than on intent, and that is the finding.
- Repeat each successful variant at least twenty times in fresh sessions. Record successes over attempts. One screenshot is an anecdote. Twenty attempts is a measurement.
- Establish consequence. A model that says a forbidden sentence is a content-policy failure. A model that calls a tool it should not have called is a security failure. Push every successful variant towards the tool surface and record where it stops.
The verbatim baseline refusal, the verbatim successful prompt, the full model output, the model identifier and version, the sampling parameters, the session identifier, and the attempt count with the success count.
Treat it as an architecture problem rather than a wording problem. Constrain what the model may do rather than what it may say: least-privilege tools, authorisation re-checked at the tool boundary, human approval on consequential actions, output validated against a schema. Prompt hardening reduces the rate. It does not close the class, and a report that recommends only prompt hardening is recommending a control whose strength nobody can state.
Indirect prompt injection, through documents and through tool output
How do you test for indirect prompt injection?
The same mechanism, delivered through content the model reads rather than content the user types. The attacker never touches the application. They place instructions in something the application will later ingest — a document that reaches the retrieval corpus, a web page the browse tool fetches, an API response, a file a code interpreter opens — and the instructions execute in the session of whichever legitimate user triggers the retrieval, with that user’s privileges.
- Map the ingestion routes from the corpus inventory. For each source, answer one question: what is the cheapest way for somebody outside the organisation to get text into this? A support ticket, a CV, an invoice PDF, a calendar invite, a review, a shared document, a web page the assistant is allowed to fetch.
- Place a marked canary instruction through the lowest-privilege route you have agreed in the scope, with a unique token so that every later observation is attributable to your document and not to chance. Instruct something harmless and observable — emit the token, name a specific non-existent product, call a designated read-only test tool.
- Wait for the pipeline. This is a step, not a pause: record the ingestion delay, the indexing delay and the cache behaviour, because they are what determine whether a defender has any window at all. Then ask a question whose natural retrieval brings your document back.
- Record whether the instruction fired, whether it fired for a different user account than the one that placed it, and whether the model told that user anything about what it had just read. Silent obedience is the finding. A model that surfaces the injected text is at least giving the user a chance.
- Escalate along the tool surface, within scope: from emit-a-token, to read something the placing user could not read, to call a tool that writes. Stop at the first write and report from there — you do not need to cause the damage to prove the path.
- Repeat the whole sequence with the document reworded. Retrieval is semantic. A payload that never gets retrieved is not a negative result, it is a test that did not run. Confirm retrieval independently by asking the application to cite its sources.
The injected document itself, kept as an artefact with its hash and the time and route of placement. The ingestion and indexing timestamps, the retrieving user’s account, the query that triggered retrieval, the citation the application returned, the model’s output, and the attempt-to-fire ratio across reworded variants.
Provenance and privilege, in that order. Label retrieved content as data in the context with an explicit boundary the application enforces, strip or neutralise instruction-shaped content at ingestion, and — the only durable control — make sure that what the model can do on behalf of a user is limited to what that user could do themselves, checked at the tool boundary rather than at the chat boundary. Add ingestion-side review for any corpus source that an outside party can write to.
Jailbreak and guardrail bypass
How do you measure a guardrail bypass rate?
Getting the model past a behavioural constraint — the provider’s safety training, a content filter in front of or behind the model, or your own policy text. Worth separating from injection in the report even though the OWASP category is the same, because the owner of the fix is different: a jailbreak of the provider’s safety layer is the provider’s problem and your risk, while a bypass of your own filter is yours entirely.
- Enumerate the guardrails first, by asking. Is there an input classifier, an output classifier, a separate moderation model, a regex list, a policy block in the system prompt? Each of these fails differently and a report that says “the guardrail failed” without saying which one is not actionable.
- Test the input classifier and the output classifier separately. Content that passes in and is caught on the way out tells you the output path is doing the work, while content that is caught on the way in tells you an attacker will spend their time on obfuscation.
- Test the boundary rather than the middle. The interesting cases are the ones adjacent to a legitimate use of your product, not the ones any list of forbidden topics would catch, because those are the ones where tightening the filter costs the business something.
- Measure the rate, in fresh sessions, at the production temperature. A bypass that works one time in five is still a bypass. The number is what lets the risk owner decide.
- Record what the bypass reached. In most business applications a jailbreak on its own is a brand and content risk. It becomes a security finding when it unlocks a tool, a document or another tenant.
Which guardrail was bypassed and how you identified it, the prompts and outputs, the success rate over a stated number of fresh sessions, and the consequence reached. Where the provider’s own safety layer is what gave way, say so plainly and route it to the provider as well as to the risk register.
Defence in depth, with the load taken off the model. Independent output checking, schema-constrained responses where the output feeds a machine, rate limiting and anomaly detection on the conversation rather than only on the request, and an explicit decision about which categories the business accepts residual risk on. Also correct the specification: if the constraint the business believed in was never in the system prompt, the finding is a missing requirement.
System-prompt and context extraction
How do you test whether a system prompt can be extracted?
Persuading the application to reveal its own instructions, its tool schema, its retrieved context, or fragments of other users’ data that happen to be in the window. The system prompt is not a secret in the cryptographic sense and should never have been treated as one, but it is a map: it names the tools, the internal endpoints, the tenant conventions, the policy the filters enforce and, distressingly often, a credential or an internal URL.
- Ask directly, then ask for a translation, a summary, a restatement in the third person, a poem, a JSON dump of the configuration, the first line, the last line, a continuation. Record which forms work, because the pattern tells you whether the refusal is a trained behaviour or a pattern match.
- Ask for the tool schema specifically, as a developer would: list your functions with their parameters. Many applications refuse the system prompt and hand over the tool list without hesitation, and the tool list is the more useful of the two to an attacker.
- Ask what the model can currently see. On a retrieval application this is the question that finds cross-tenant leakage: the model is asked to enumerate the documents in its context and names one the user has no right to.
- Probe the context window at the edges. Very long inputs, truncated histories and summarisation steps all move content in and out of the window, and a summariser that carries another user’s fragment forward is a leak with a long half-life.
- Diff what you extracted against the real system prompt the auditee gave you. Partial or hallucinated extraction is common, and reporting a hallucinated system prompt as an exfiltrated one is the fastest way to lose a technical reader.
The extraction prompts and outputs, the diff against the true system prompt, the extracted tool schema, the success rate, and an explicit statement of which extracted fragments were verified against the auditee’s own text rather than taken on the model’s word.
Assume disclosure and design for it: no secrets, credentials, internal hostnames or unpublished business rules in the prompt, authorisation enforced outside the prompt, and the tool schema treated as disclosable. Then add the detection that matters more than the refusal — alerting on conversations that are trying, since extraction attempts are a reliable early signal of a targeted attacker.
Training-data and retrieval-corpus leakage
How do you test for cross-tenant retrieval leakage?
Getting the system to emit data it holds but should not disclose to this user. Two quite different mechanisms sit under one heading, and a report that conflates them misdirects the fix. Corpus leakage is an authorisation failure: the retrieval layer fetched a document the user was not entitled to, and the model dutifully summarised it. Training-data leakage is a memorisation failure in a fine-tuned or continuously-trained model, which is slower to demonstrate, far harder to fix, and the reason fine-tuning on production data deserves a design review of its own.
- Test authorisation at the retrieval layer, not at the chat layer. Take two accounts with different entitlements, ask both the same question about a document only one may see, and compare. Then take the same question to a third account in a different tenant.
- Test the filter, not just the result. Many retrieval stacks fetch broadly and filter afterwards, which means the privileged text entered the context and the only thing standing between it and the user is the model’s willingness to be discreet. Ask the model to enumerate what it retrieved, ask for a count, and ask about a document by a title only the other tenant’s user would know.
- Probe metadata as well as content. Titles, filenames, authorship, dates and the mere existence of a document are disclosures, and a system that refuses to quote a contract while confirming its title and counterparty has still leaked the part that mattered.
- For a fine-tuned model, probe for memorisation with distinctive strings the auditee can confirm were in the training set — an unusual account number format, a rare name, a template with a canary. Report it as an indicator rather than as proof unless the auditee can confirm the string came from training data and not from retrieval.
- Where embeddings are stored, ask what the vector store holds alongside the vectors and who can query it directly. A vector database reachable without the application’s authorisation layer is a copy of the corpus with the access control removed.
The two account identities and their entitlements, the identical query, both outputs, and the auditee’s confirmation of what each account was entitled to see. For metadata leakage, the exact field disclosed. For suspected memorisation, the canary string and the auditee’s statement of where it appears.
Filter before retrieval, not after: the user’s entitlements should constrain the query, so that unentitled text never enters the context. Partition indexes per tenant where the data model allows it. Control direct access to the vector store as tightly as to the source systems. On the training side, treat any fine-tune on customer data as a data-protection decision with a lawful basis and a retention position, not as an engineering optimisation.
Corpus and fine-tune poisoning
How do you test for RAG corpus poisoning?
The write-side twin of indirect injection, and a different attack with a different fix. Injection plants an instruction and wants the model to obey it. Poisoning plants an assertion and wants the model to believe it: a document that is durable, non-instructional, retrievable and wrong, or attacker-favourable, so that the assistant answers a whole family of questions the attacker’s way without anything in the output ever looking like an injected command. Filters tuned to strip instruction-shaped text do not see it, because it is not instruction-shaped. The same mechanism at training time is fine-tune poisoning, which is slower, rarer and considerably harder to undo.
- Start from the same inventory as indirect injection, but ask a different question of each source: not “who can get text in”, but “who can cause a document to persist in the index, and what review stands between them and persistence”. Persistence and review, not ingestion, are what this class turns on.
- Plant a benign but checkably false assertion through the lowest-privilege route in scope — a stated policy, a price, a contact, a procedure — written the way the corpus writes, with no imperative in it, and with a unique token in a field nobody reads so that the document remains attributable to you.
- Measure reach rather than firing. Ask the user’s question in as many natural phrasings as you can construct, and record how many of them return the planted assertion, with and without the citation. Retrieval is semantic, so the finding is how wide the phrasing net is, not that one query hit.
- Test whether the assistant discloses provenance. A system that cites the planted document is giving the user a chance to catch it, while one that states the assertion in its own voice has laundered it. Both are findings, and they carry different severities.
- Test the removal path, which is the part nobody has. Delete the document and measure the retention question: how long does the assertion keep coming back through the embedding index, the response cache, the summarisation layer and any per-user memory that copied it? The delay between removal and forgotten is the number that belongs in the report.
- For a fine-tuned model, keep this separate and treat it as a design review rather than a test: what data the fine-tune drew on, who could write into that data, what review it passed, and whether the resulting weights can be rolled back to a known-good version at all.
The planted document with its hash, the route and time of placement, the review it did or did not pass, the set of query phrasings tried with the number that surfaced it, whether the answer carried a citation, and the removal-to-forgotten delay measured across the index, the cache and memory.
Review at ingestion for any source an outside party can write to, provenance carried through retrieval and rendered to the user, an owner for the corpus as a controlled asset rather than as a data lake, and a tested removal path with a stated maximum delay. On the training side, treat a fine-tune on customer-supplied or externally-sourced data as a design decision with a provenance record and a rollback position.
Excessive agency and confused-deputy tool invocation
How do you test for excessive agency in an AI agent?
The model does something rather than says something — and does it with permissions, on an instruction it should not have accepted, or with a scope wider than the user could have exercised directly. Excessive Agency is LLM03 in the 2026 edition of the OWASP Top 10 for LLM Applications, up from LLM06 in the 2025 edition, and the move up the list matches what we see in scoping calls: the interesting risk has shifted from what these systems say to what they are wired to do.
- Inventory the tools as capabilities, not as functions. For each, write the worst sentence that is true: “can email any address”, “can delete a record”, “can issue a refund up to the account balance”, “can run arbitrary SQL against the reporting replica”, “can execute code in a container with network egress”.
- Establish the identity at each call. Ask the auditee to show, in code or configuration, whose credentials the tool uses. Where it is a shared service account, test whether a low-privileged user can reach data or actions that account can reach and they cannot.
- Test parameter control. Can the conversation influence a parameter that was supposed to be fixed — the recipient, the amount, the tenant identifier, the file path, the SQL, the URL? Parameter injection into a tool call is the most reliably exploitable thing on most agent stacks and the least frequently tested.
- Test the approval gate as an attack surface, not as a control. Does the approval screen render text the model produced? Can the model make one action look like another in that text? Can it induce a burst of approvals so that the consequential one arrives ninth in a row of eight routine ones?
- Chain it to indirect injection. The finding that changes a steering committee’s mind is not “the model called a tool”. It is “a document placed by an outside party caused the model to call a tool, in another user’s session, with that user’s permissions”. Build that chain deliberately and stop at the last safe step.
- Test the loop. Agentic stacks retry, self-correct and call tools in sequences. Establish what bounds the loop — a step limit, a spend limit, a timeout — and what happens at the bound. An agent with no step limit is an availability and cost finding before it is anything else.
The tool schema as exported, the identity used at each call, the conversation transcript with the tool-call arguments and returns logged verbatim, the chain from entry point to consequential action, and the state of the approval gate at each step. Where a gate held, record that too — a control that worked is evidence, and CERT-In’s empanelment terms expect reports to state all states of security found and not only failed security measures.
Least privilege at the tool boundary, per-user identity rather than a shared service account, parameters validated against a schema and never taken verbatim from generated text, human approval on anything irreversible with the approval surface rendering application-controlled text, and hard bounds on steps, spend and time. Then log every tool call with its arguments, because this is the one class where the incident response is only as good as the log.
Supply chain: model, plugin and MCP server provenance
How do you test the provenance of an MCP server or a model?
Everything in the stack that somebody else wrote and you are trusting: the model weights and their adapters, the embedding model, the plugins, and — on an agent stack built in the last two years — the tool servers, commonly Model Context Protocol servers, that supply the model with its tools. The distinctive property of that last layer is that the thing you approved is a description, and on most implementations the description is re-read at runtime. A server whose tool descriptions you reviewed in March can serve different descriptions in July, to the same client, with no version bump and no prompt for re-approval. The tool description is prompt text that a third party controls.
- Inventory the layer as a supply chain, not as a feature list: for each model, adapter, embedding model, plugin and tool server, record who publishes it, how the client obtains it, what version is running, and whether that version is pinned or resolved at start-up.
- Establish whether tool descriptions are re-read at runtime. Ask the engineers, then verify it: change a description on a server you control and see whether the client picks it up without a redeploy or an approval prompt. If it does, every description in the inventory is third-party-controlled text inside your context window.
- Test description poisoning on a server you control and have written approval to modify. Put instruction-shaped text into a tool description — telling the model what to do before calling another tool, or what to send along with the call — and record whether it influences the model’s behaviour. This is prompt injection with a supplier as the author.
- Test cross-server shadowing where more than one server is connected: whether one server’s tool description can influence the selection of, or the arguments passed to, another server’s tool. Multi-server agent stacks put mutually untrusted suppliers in one namespace, and tool selection is the shared resource.
- Test the client’s own trust behaviour. What happens when a server’s identity or certificate changes, when a new tool appears on a connected server, when a tool is renamed, or when a server returns a response far larger than expected. Each of those is a change to your attack surface arriving without a deployment.
- Close the loop on the model itself: is the deployed version the version that was tested, and can that be demonstrated from the deployment rather than asserted from memory? A production model that differs from the tested one quietly invalidates every other finding in the report.
The inventory with publisher, obtain-path, running version and pinning status per component. The answer to the runtime re-read question, with how it was verified. The modified description and the behaviour it produced, on a server you controlled. The client’s response to an identity or tool-set change, and a deployed-versus-tested comparison for the model.
Pin versions and verify integrity at every layer, including tool servers. Treat a tool description as code: reviewed, version-controlled, and re-approved when it changes rather than trusted because it was approved once. Isolate servers from different suppliers rather than merging them into one tool namespace. Require human approval on a newly appeared tool. And keep an inventory of AI components with provenance — CERT-In’s own §6(xxvi) AIBOM class and the technical guidelines on AIBOM occupy exactly this ground.
Unbounded consumption: cost, context and the agent that will not stop
How do you test an LLM application for denial of wallet?
The availability and cost class, which on a metered system is the same class. A conversation that costs a fraction of a rupee at design time can be made to cost orders of magnitude more by an attacker who controls one input, because the amplification chain runs one prompt to N retrievals to M tool calls to K tokens, and each multiplier is usually unbounded by default. The commercial name for it is denial of wallet, and it is the finding most likely to be dismissed in scoping and most likely to arrive as a billing incident.
- Establish the ceiling as it exists today: the maximum tokens, retrievals, tool calls, agent steps, wall-clock time and spend a single conversation can consume, and who set each of those bounds. On most stacks at least two of them turn out to have no bound at all.
- Find the amplification factor empirically. One ordinary request, instrumented: how many retrievals, how many tool calls, how many tokens. Then one adversarial request designed to maximise each multiplier — a question that matches everything in the corpus, a document that induces a long chain, a request that makes the agent retry.
- Test the context-length path separately. A very large uploaded document, a long conversation, a tool that returns a large payload: each can force the expensive end of the pricing curve, and some will also evict the safety instructions from the window, which is a second finding in the same test.
- Test the loop at the bound rather than below it. What happens at the step limit — does it stop cleanly, retry, or fail open? An agent that fails open at its own bound has no bound.
- Do the arithmetic with the risk owner in the room: amplification factor multiplied by the requests an unauthenticated or low-cost attacker can make in an hour, against the unit price. That number is the finding, and the transcript is only the evidence for it.
The measured baseline consumption for an ordinary request and for the adversarial one, both instrumented. The bounds that exist and the ones that do not, named per multiplier. The behaviour at each bound that does exist, and the arithmetic from amplification factor to cost per hour, with the unit price the auditee supplied.
Bounds on every multiplier — tokens, retrievals, tool calls, steps, wall-clock time and spend — enforced server-side per conversation and per principal, with a defined behaviour at the bound rather than a crash. Rate limiting on the conversation rather than only the request. Budget alerting that fires on rate of spend, not on a monthly total. And for anything reachable without authentication, a separate and much tighter set of the same bounds.
Insecure output handling, where model output reaches a browser, a shell or a database
How do you test where an LLM’s output lands?
The model emits text, and something downstream interprets it as code, markup, a query or a command. This is the class where LLM red teaming rejoins ordinary application security, and the good news is that it comes with ordinary CWEs, ordinary severities and ordinary fixes. The bad news is that it is routinely missed, because the output is generated by “our own” model and therefore feels like our own data.
- List the sinks from the scope and take them one at a time. A browser rendering model output as HTML or Markdown, a Markdown renderer that permits images or links, a shell, a database query built by string concatenation, a templating engine, a PDF or document generator, a webhook, or another model’s context.
- For the browser sink, test whether generated markup is rendered rather than escaped, and test the image and link cases specifically — an image whose source is an attacker-controlled URL with the conversation in the query string is data exfiltration that requires no click.
- For the shell and code-interpreter sinks, test what the generated command can reach: the file system, the network, the credentials in the environment, the metadata service. Agree the boundary in writing first, because this is the step most likely to touch something the auditee did not expect.
- For the database sink, test whether generated text reaches a query as a value or as syntax. A model asked to write SQL and an application that executes it are a parameterisation failure with a language model attached.
- For the model-to-model sink, remember that the second model’s context is a sink like any other: text the first model produced arrives as instructions to the second. Multi-agent stacks are injection amplifiers and should be tested as a chain rather than as parts.
- Carry the finding back to the entry point. Insecure output handling plus indirect injection is the complete exfiltration path, and it is the path worth demonstrating end to end.
The prompt, the raw model output before rendering, a capture of the rendered result or the executed command, the sink’s code path, and the CWE. This class produces the cleanest evidence in the whole engagement because the last step is deterministic even though the first is not.
Treat model output as untrusted input to the next system. Encode at the sink, parameterise queries, render Markdown with a strict allow-list and no remote images, never pass generated text to a shell, constrain machine-consumed output to a schema and validate it, and give the code interpreter no credentials and no egress it does not need.
Notice what the ten have in common. In every case the durable fix is outside the model: an authorisation check at the tool boundary, an entitlement filter before retrieval, an encoder at a sink, a bound on a loop, a schema on a parameter, a pinned version on a supplier’s component, an approval surface the model cannot write. That is the whole argument for treating this as security engineering rather than as prompt craft, and it is why an engagement that produces only a list of clever prompts has produced a list with a short shelf life. The same instinct underlies CERT-In’s §8(i), which says solely tools-based testing should be discouraged because it “may focus primarily on automated processes and may overlook non-automated or manual components” — the objection is to a method that cannot see the whole system, which is exactly what an automated jailbreak suite is.
Which of this applies to your system
Does LLM red teaming apply to a system with no chatbot?
Most of this page is written for the architecture that generates most of the questions: a retrieval assistant with a tool surface, memory and an approval gate. Plenty of systems are not that, and scoping all ten classes against a system where four of them cannot fire is how a buyer pays for tests that produce nothing. The table says which classes apply to each archetype, which cannot, and which of the nine scope properties becomes load-bearing — which is usually a different one from the archetype above it.
| Archetype | Which classes apply | Which cannot fire | The property that becomes load-bearing |
|---|---|---|---|
| Thin API wrapper: no retrieval, no tools | Direct injection, jailbreak, prompt extraction, insecure output handling, unbounded consumption. | Indirect injection, corpus leakage, corpus poisoning and excessive agency cannot fire: there is no corpus and no tool. | The output sinks and the identity. On a wrapper the whole engagement is where the generated text lands and what the back end does with it, plus whether one tenant’s prompt can ever reach another tenant’s response. Do not buy a nine-property scope for a system with four of them. |
| Retrieval (RAG), read-only | Everything except excessive agency and supply chain at the tool layer. Indirect injection and corpus poisoning are the engagement. | Confused-deputy tool invocation cannot fire without tools — but “read-only” usually means “no writes”, not “no tools”, so check the retrieval call itself: it is a tool, and it takes a query. | The corpus and its trust boundary, and the entitlement filter. The single question that decides the shape of the engagement is whether filtering happens before retrieval or after it. |
| Agentic, with tools that write | All ten. This is the architecture the rest of this page is written for. | Nothing, unless a property genuinely does not exist and is recorded as absent. | The identity the system acts under, and the approval surface. Everything else sets the entry point. These two set the consequence. |
| Multi-agent, or agent-to-agent | All ten, and each of them once per hop. Insecure output handling is the connective tissue: one agent’s output is the next agent’s instruction channel. | Nothing — but testing the agents individually and calling it done is the characteristic failure here. | The model-to-model sink. Map the graph before testing anything, note where privilege changes between hops, and test the chain rather than the parts: a multi-agent stack is an injection amplifier by construction. |
| Fine-tuned or self-hosted model | All of the above, plus training-data memorisation and fine-tune poisoning in a form a hosted model does not have. | Provider-side model drift does not apply — which removes one retest trigger and adds another, because now your own deployment pipeline is the thing that moves. | Provenance and memorisation. There is no provider identifier to record, so the configuration record becomes the weights hash, the quantisation, the serving stack and its version, and the training-data position becomes a data-protection question rather than an engineering one. |
| Coding assistant wired to MCP or plugin servers | All ten, with supply chain and excessive agency dominant. The tool surface is supplied by third parties and can change without a deployment. | Nothing. Beware the intuition that a developer tool is low risk: the identity it acts under is usually a developer’s, and a developer’s reach is not small. | The tool and function surface, read as provenance rather than as capability. Who publishes each server, whether descriptions are re-read at runtime, whether servers from different suppliers share one namespace, and what the code path can reach on the machine it runs on. |
| Voice or multimodal front end | All of the text classes, entered through a channel the corpus inventory does not describe as text. | Nothing is removed. What changes is that an ingestion filter reading text layers will not see instructions that exist only as pixels or as audio. | The trust boundary, extended one line per modality. Test each modality as its own channel and record the transcription or vision step as part of the configuration, because a change to it changes the results. |
| Embedded model in a batch pipeline, no chat surface | Indirect injection, poisoning, insecure output handling and unbounded consumption — all of them silently, because nobody is watching a conversation. | Jailbreak in the conversational sense, and anything that depends on an interactive session. | The output sinks, and the absence of a human in the loop. When there is nobody to notice, the sinks are the entire engagement: what the classification, the extraction or the generated record is used for downstream, and what validates it before it gets there. |
Two of these deserve a warning. A thin wrapper is genuinely a smaller engagement, and a supplier who prices it as a full agentic red team is selling you the table of contents. A multi-agent stack is genuinely a larger one, and a supplier who prices it as a single application is about to test the parts and miss the chain. The archetype belongs in the first paragraph of the scope of work, because it is what makes two quotes comparable.
What you may not do, and what needs written approval
What should be in the rules of engagement for an AI red team?
A conventional penetration-testing rules-of-engagement document does not cover most of what follows, and the gaps are consequential rather than procedural. Testing a hosted model’s safety layer can breach the provider’s terms of service, and the provider — not the auditee — is the party who can revoke the account. Jailbreak testing towards certain categories can generate material whose production or retention is itself unlawful in India. A canary placed through a customer-facing ingestion route touches a third party’s data. Cost amplification spends the auditee’s money in real time. And a production corpus contains real customer records, which is the difference between exfiltrating a canary and exfiltrating a person.
May, on a normal authorisation
- Direct injection and jailbreak attempts against the chat surface you were given Ordinary authorisation
- Baseline refusal capture against each stated policy Ordinary authorisation
- System-prompt and tool-schema extraction attempts Ordinary authorisation
- Read-only tool calls, within the agreed tool list Ordinary authorisation
- Retrieval queries from accounts the auditee issued you §15.1.2
Only with specific written approval, naming the approver
- Placing content through any ingestion route a real customer also uses Named approver, per route
- Exercising any tool that writes, spends, sends, deletes or grants, with the stop point written down Named approver, per tool
- Code-interpreter testing that touches the file system, the network or the instance metadata service Named approver, boundary in writing
- Anything that amplifies token spend or agent steps deliberately Cost owner, with a ceiling
- Any test that reads or writes a second tenant’s data Both tenants, or a seeded test tenant
- Testing the provider’s own safety layer rather than the auditee’s — check the provider’s terms first and use its researcher-access route where it has one Provider terms, in writing
- Anything touching employees rather than systems, reported without naming individuals §15.2.2(iii)
May not
- Generate content whose production or retention is itself unlawful in India No approval cures this
- Exfiltrate real customer records where a canary would prove the same path Data-handling guidelines
- Leave a canary, a planted document or a test account in place after the engagement Removal checklist
- Place content through a route that reaches a third party’s system §15.2.2(iii)
- Test a model endpoint, plugin or tool server the auditee does not control Out of authorisation
- Exploit a discovered breach rather than reporting it §16.4
- Store transcripts or artefacts outside India, or share them with an overseas partner unauthorised Data-handling guidelines
The tester-traffic convention
Defenders have to be able to tell your traffic from a real attacker’s, in both directions, and on an LLM stack the usual source-IP convention is not enough because the interesting traffic arrives as a document rather than as a request. Agree four things in writing before day one: a tester-account naming convention visible in the application logs, a source identifier carried in every placed artefact (including the canary documents), the source IP ranges (which §13.2.7(i) already expects a remote tester to disclose), and a named defender who knows the window. Then agree the inverse case: what happens when something in the logs is not yours.
The removal checklist, at engagement end
Every canary document removed from every index it reached, including the embedding store and any cache that copied it. Every planted attachment deleted from the source system as well as from the corpus. Every test account disabled and every temporary credential revoked, which §15.1.2 asks be done “immediately upon completion of the audit”. Any memory or per-user state the testing wrote, cleared. Then a written confirmation from the auditee that the removal was observed on their side too — because a canary that only the tester believes is gone is still in production.
Evidence for a system that will not repeat itself
How do you evidence a finding in a non-deterministic system?
This is the part that separates an LLM engagement from every other kind of application testing, and it is where most reports fail. A web application either returns the other tenant’s invoice or it does not. A language model returns it sometimes. So the unit of evidence cannot be a screenshot of the one time it worked. It has to be a configuration record plus a rate, captured tightly enough that a different tester on a different day can run the same thing and get a comparable number.
CERT-In’s reporting clauses already push in this direction for conventional audits, and they read naturally onto this problem. §16.1(ii) says the report should be clear, precise and comprehensive and should include the audit process, the detailed scope, the duration, the methodologies and standards used, the tools, the manual process, the findings, prioritisation, sampling decisions, manpower involved, exemptions, limitations and other constraints. On a stochastic system, “sampling decisions” is not a footnote — it is how many times you tried, and it belongs in every finding.
| Model identifier and version | The single field that decides whether the finding can be retested at all. Record the identifier exactly as the response body returns it, including any dated snapshot suffix, and record it again at retest — a changed string is the first thing to check when a result moves. Reading it correctly is more work than it sounds, because a deployment name, a gateway or an alias can each mask the model that actually served you. The model-manifest table below sets out where to read each field and what to do when it is absent. |
| Sampling parameters: temperature, top_p, and seed where available | Temperature is the dial that turns a reproducible finding into an intermittent one. Where the provider exposes a seed, a fixed seed plus fixed parameters is the closest thing to determinism available — but it is not determinism, and a report that implies otherwise will be contradicted by the first engineer who tries it. Batching, mixture-of-experts routing and non-deterministic floating-point kernels all move the result at a fixed seed and temperature zero. A seeded run still needs its attempt count. The seed narrows the variance, it does not remove the denominator. |
| The exact prompt, byte for byte | Including whitespace, invisible characters, line breaks and any encoding. Paste it as a code block rather than as prose, and keep the raw file as an artefact. A paraphrased prompt is not a reproduction step, and a screenshot of a prompt is not a prompt. |
| The system prompt version or hash | A finding is against a configuration, not against a model alone. Record the version reference or a hash of the instruction layers as they stood during the test. CERT-In’s guidelines already work this way for conventional applications: §15.2.3(ii) requires application hash values and version numbers to be obtained from the auditee and included in the audit report. Extending that discipline to the prompt and the tool schema is our reading of what the clause is for, not something the clause says. |
| The retrieval snapshot | Which documents were in the corpus, which were returned, and the corpus as of a date. Without it, a retest that fails to reproduce tells you nothing, because the corpus has moved underneath both runs. §13.2.3(ii)(b) makes the as-of date of the asset inventory a report field for conventional audits, and the corpus deserves the same treatment. |
| The tool schema, exported verbatim | Attach the schema as the model received it. Tool definitions change more often than models do, and an excessive-agency finding whose schema is not recorded cannot be distinguished at retest from one that was fixed by a tool being renamed. |
| The full conversation transcript, including tool calls and returns | Not the final answer. The interesting behaviour on an agent stack is in the intermediate steps: which tool was chosen, with what arguments, what came back, and what the model did next. Log it in full, redact it once, and keep the unredacted copy under the engagement’s data-handling terms. |
| Attempts, successes, and the reproduction rate | The field that replaces the screenshot. State the denominator: nineteen successes in twenty fresh sessions is a different finding from two in twenty, and both are real. Say whether sessions were fresh or continued, because a continued session is contaminated by everything earlier in the window. How large the denominator has to be is arithmetic rather than taste, and it is worked through below. |
| Timestamps, in a stated timezone | Provider-side models change without notice. A finding with a timestamp can be correlated against a provider’s release notes later, while a finding without one cannot be defended when it stops reproducing. |
| The negative result, recorded as deliberately as the positive one | Twenty attempts with zero successes is evidence, and it belongs in the report — printed as the bound it supports rather than as a bare zero, which is the subject of the arithmetic below. Annexure I of CERT-In’s empanelment terms and conditions (v7.2, April 2024) sets out expectations of an auditee from an auditor, two of which read directly on this: “Reports include all unknowns clearly marked as unknowns” and “Reports state clearly all states of security found and not only failed security measures.” |
How many attempts does a reproduction rate need?
Pick the number from the consequence, not from habit. The cost of being wrong is asymmetric: a false positive wastes a remediation sprint, and a false negative ships a path to somebody else’s money or somebody else’s data. So the question to answer before choosing n is which of those two you are protecting against, and the answer is usually the second.
| If a success would reach | Attempts per variant | Why that number |
|---|---|---|
| The behaviour reaches a tool that writes, spends, sends, deletes or grants | 50 fresh sessions | Choose n by the cost of a wrong negative, not by statistical taste. Here a false “not observed” ships a path to money or to somebody else’s data, so buy the tighter bound: zero in fifty is consistent with a true rate up to roughly 6%, where zero in twenty leaves you at 15%. |
| The behaviour reaches data belonging to another tenant or another user | 50 fresh sessions | Same reasoning, different loss. A disclosure path that fires one time in twenty is a disclosure path, and the number of conversations a real system handles will find it long before your twenty attempts do. |
| The behaviour reaches a sink that renders or executes | 20 fresh sessions, then one deterministic proof | This class rejoins ordinary application security at the last step. Once the model has emitted the payload even once, the sink either escapes it or does not — and that half is deterministic, testable directly, and worth more than any number of further attempts. |
| A content-policy or guardrail bypass with no onward consequence | 20 fresh sessions | Enough to separate “roughly one in two” from “roughly one in ten”, which is the distinction the risk owner is actually making. Spending fifty here buys precision nobody will act on. |
| A negative result you intend to print in the report | 20 minimum, 100 where the claim matters | A negative is a claim about an upper bound, and its strength is entirely the denominator. Twenty attempts support “under about 15%”. A hundred support “under about 3%”. Decide which sentence you are willing to sign before you choose n. |
The rule of three, for a negative result
This is the sentence most missing from LLM reports, and it is the one that makes a negative result mean something. Zero successes in twenty fresh sessions is consistent with a true rate of up to about three in twenty — roughly 15% — at 95% confidence. Zero in a hundred is consistent with a true rate of up to about 3%. The arithmetic is the standard approximation: the one-sided 95% upper bound after zero observed successes is about 3 divided by n. So “we tried twenty times and it never worked” is not a statement that it does not work. It is a statement that it probably does not work more than about one time in seven, which is a very different sentence and the only one the evidence supports. Print the bound beside the zero, and choose n by the bound you are prepared to sign.
The interval around a positive rate
A measured rate is an estimate with a width, and the width at these sample sizes is wide. Thirteen successes in twenty is 0.65, with a 95% interval of roughly 0.41 to 0.85. That is why the report prints the fraction and not the decimal alone: 0.65 reads as a measurement, 13/20 reads as what it is. It is also why a retest showing 0.45 has not demonstrated an improvement — the intervals overlap, and two overlapping intervals are one result observed twice. Where a fix has to be shown to work, the retest needs a larger n than the original test did, which is a commercial fact as much as a statistical one and belongs in the scope of work.
The other half of the question — how likely you are to see a behaviour at all, given an underlying rate and a number of trials — is worked through as a table on what AI red teaming is, and is worth reading beside this one. That table answers “will ten attempts find it”, while this one answers “what can I claim once I have finished attempting”.
How do you actually capture the model version?
“Record it as the API returns it” is the assertion. This is the operation, and it matters because the string is often absent, often an alias, and occasionally wrong. Capture the manifest below at both ends of the engagement and diff it, so that “the model may have changed” becomes a checked fact rather than a suspicion — and where it moves mid-engagement, the diff goes into the limitations paragraph with its date.
| What to capture | Where to read it | When it is not available |
|---|---|---|
| The provider model identifier | The response body, not the request. What you sent is what you asked for. What came back is what served you, and on a gateway the two differ. | Ask the provider console which model the deployment or alias currently resolves to, and record the console answer with its date. Record in the report that the identifier was established from the console rather than from the response. |
| The system fingerprint or serving-configuration identifier | The response body, where the provider exposes one. It can change when serving configuration changes although the model identifier does not — which is exactly the case that makes a finding move with nothing on your side having changed. | Record its absence explicitly. A blank field and an absent field are different facts. |
| Whether the identifier is an alias or a pinned snapshot | The provider console or the deployment configuration. Do not infer it from the string: an alias can look like a version, and a dated-looking suffix is not proof of pinning. | Treat it as unpinned and say so. Unpinned is the assumption that fails safe, because it keeps the retest trigger in place. |
| The deployment or gateway layer between the application and the provider | Architecture, then verification. A hosted deployment name, a router or an internal proxy can mask the underlying model, and several will fail over to a different model silently and return the alias regardless. | Record that a failover is possible and untested. On an agent stack this is the single likeliest explanation for a result that will not reproduce. |
| The provider request identifier, per recorded attempt | The response headers or body. It is the only handle that lets the provider be asked, months later, what actually served that request. | Record the timestamp to the second in a stated timezone instead, and keep the raw response bodies rather than a summary. |
| For a self-hosted model: weights hash, quantisation, serving stack and version | The deployment pipeline. A self-hosted model has no provider identifier at all, so the manifest is the substitute and it has to come from the artefact store rather than from anybody’s recollection. | This is a finding in its own right. A model in production whose weights cannot be identified cannot be rolled back, compared or retested. |
| The region, and the tenancy | The deployment configuration. It changes both the behaviour and the data-protection position, and the second one is the reason it belongs in the report even when the first does not. | Ask, and record the answer as the auditee’s statement rather than as an observation. |
An invented finding record
The record below is invented for illustration: it is not a client, not a sector, not a real system, and it makes no claim about anybody’s outcomes. Its only job is to show the shape of a finding whose exploitability is probabilistic, with every field that makes it reproducible present and nothing else.
| finding.id | RT-04 |
| finding.title | Indirect prompt injection via ticket attachment reaches the refund tool |
| weakness | CWE-862 Missing Authorization at the tool boundary |
| weakness.secondary | CWE-441 Unintended Proxy or Intermediary (“Confused Deputy”) |
| entry.weakness | CWE-1427 Improper Neutralization of Input Used for LLM Prompting |
| cve | none — defect in bespoke application code |
| epss | not applicable, no CVE |
| cvss.v4.vector | recorded in full in the finding, scored for consequence |
| cvss.nomenclature | CVSS-BE — environmental values supplied by the auditee |
| model.id | read from the response body, verbatim, including the snapshot suffix |
| model.fingerprint | serving-config identifier, recorded per attempt where exposed |
| model.pinned | no — alias resolves to the provider default (confirmed in the console) |
| model.gateway | internal proxy present. Failover to a second model possible, untested |
| model.region | recorded, and differs from the auditee’s primary data region |
| model.manifest_diff | captured day 1 and day 31. Identifier moved on day 22, see limitations |
| provider.request_ids | recorded per attempt, all 20 |
| sampling.temperature | 0.7 (production value) |
| sampling.top_p | 1.0 |
| sampling.seed | not supported by this endpoint |
| system_prompt.ref | prompt-store v41, sha256 recorded in the artefact bundle |
| tool_schema.ref | export attached, 11 tools, 3 of which write |
| corpus.snapshot | as of 09:00 IST on the first day of the test window |
| entry.channel | PDF attached to a support ticket by an unauthenticated requester |
| entry.placed_at | day 3, 11:42 IST |
| index.observed_at | day 3, 12:07 IST — 25 minutes to ingestion |
| trigger.account | support agent, tenant B (did not place the document) |
| trigger.query | recorded byte for byte in the artefact bundle |
| attempts | 20 fresh sessions |
| successes | 13 |
| reproduction_rate | 0.65 (13/20), fresh sessions, production temperature |
| tool_called | refund_issue — call blocked at the approval gate on 13 of 13 |
| residual | gate text is model-supplied (see finding RT-05) |
| controls_that_held | approval gate, per-tenant index partitioning |
| timestamps | all IST, recorded per attempt |
Two fields in that record do more work than the rest. reproduction_rate is what turns an anecdote into a measurement, and it is meaningless without its denominator and without the statement that the sessions were fresh. controls_that_held is what stops the report reading as a list of disasters: the gate held thirteen times out of thirteen, and saying so is both honest and useful — though the residual line is the reason that particular comfort is provisional.
Two housekeeping points that are easy to lose and expensive to recover. First, keep the raw artefacts — the injected document, the prompt files, the transcripts — as files with hashes, not as images pasted into a document. §18(iv) of the 2025 guidelines requires audit evidence to be stored securely, and §18(i) requires auditee organisations to seek the working notes on completion of the audit, with provisions for that to be included in the audit contract. Second, remember what those transcripts contain. Conversation logs from a production application hold customer data by construction, and CERT-In’s Policy Guidelines for Handling Audit related Data require auditee data to be stored only on systems located in India, encrypted on auditor devices, accessed on a need-to-know basis, wiped after the agreed retention period, and not shared with an overseas partner unless the auditee specifically authorises it — which is a live question the moment a transcript passes through a model hosted in another region.
CERT-In’s guidelines are not the only instrument in play, and on an Indian engagement they are not the one a customer will ask about. Those transcripts are personal data, and the Digital Personal Data Protection Act, 2023 allocates the roles: a Data Fiduciary is defined at section 2(i) as any person who alone or with others determines the purpose and means of processing, and a Data Processor at section 2(k) as any person who processes personal data on behalf of a Data Fiduciary. The auditee is the first, and the testing organisation, handling its conversation logs, is the second. Section 8(1) then keeps responsibility with the Fiduciary — it is responsible for compliance “in respect of any processing undertaken by it or on its behalf by a Data Processor” — and section 8(2) permits a Processor to be engaged for activities related to offering goods or services to Data Principals “only under a valid contract”. Section 8(5) extends the Fiduciary’s duty to take reasonable security safeguards to processing done on its behalf. The practical consequence for a scope of work: the processing relationship belongs in the engagement terms expressly, not only in the CERT-In data-handling clause, and it should say what the transcripts are used for, where they live and when they are destroyed.
What the artefact bundle actually contains
The instruction is to keep artefacts as files with hashes rather than as images pasted into a document. This is what that looks like on disk. The layout is a convention rather than a standard, and any equivalent one will do — what matters is that every claim in the report has a file behind it and every file has a hash in a manifest.
RT-04/
prompts/ baseline.txt, variant-01.txt … variant-07.txt
transcripts/ attempt-01.json … attempt-20.json (tool calls, args, returns)
injected/ canary-ticket-attachment.pdf
attempts.csv one row per session: id, timestamp, model.id, fingerprint,
request_id, temperature, top_p, seed, variant, outcome
manifest.sha256 every file above, with its SHA-256
model-manifest.json captured day 1 and day 31, with the diff
README.md what was claimed, and which file proves each claimThe canary convention
Give every canary a token built from three parts: a fixed prefix, the engagement identifier, and a per-canary suffix. Then any later sighting of that token — in a log, in a support ticket, in another system entirely — is attributable to one document placed on one date through one route, months later and by somebody who was not on the engagement. A random string is not a canary. It is a coincidence waiting to be argued about.
What a canary document contains
Four parts, and only four: the instruction (in the plainest form that fits the document type), the unique token, an observable effect that is harmless and unambiguous (such as emitting the token or calling a designated read-only test tool), and placement metadata recorded outside the document — route, time, placing account, and the hash of the file as placed.
Redaction
Redact once, into a separate shareable copy, and never in place. Keep the unredacted set under the India-only storage rule above, with its own retention date. A redaction performed on the only copy has destroyed the evidence for the finding it was supposed to support.
When tool-call logging is absent
Common, and it silently removes the corroboration the excessive-agency class depends on: you can show what the model said, not what it called. Record the absence as a finding in its own right rather than as a limitation, because a system whose tool calls are not logged cannot be incident-responded at all — and direction (iv) of the 2022 Directions requires logs of all ICT systems to be enabled and maintained for a rolling 180 days. Then fall back to the effect: the record written, the mail sent, the refund raised, evidenced from the target system rather than from the agent.
Scoring a finding whose exploitability is probabilistic
How do you assign severity to an LLM finding?
§16.2(ii) of the 2025 guidelines is unambiguous about the scheme: auditors are required to implement both CVSS and EPSS in their audit reports, observations must be categorised on CVSS for severity and supplemented with EPSS, and every reported observation “shall be mapped with Common Weakness Enumeration (CWE) and Common Vulnerabilities and Exposures (CVE) number”. That scheme was built for published vulnerabilities in deployed products. Three of its four fields do not fit a defect in your own prompt scaffolding, and the professional answer is to say so in the report rather than to force the fields.
| The problem | Why it arises | What to do about it |
|---|---|---|
| There is usually no CVE, so there is usually no EPSS | An injection in your own prompt scaffolding is a defect in your application, not a published vulnerability in a product. FIRST’s own EPSS FAQ is explicit that scoring something with no CVE identifier at all “is a longer-term direction”, so the field is genuinely unavailable rather than merely inconvenient. | Print the field with “not applicable — no CVE” and the reason, rather than leaving it blank or inventing a number. Be clear about what is the clause and what is us: §16.2(ii) of the 2025 guidelines requires auditors to implement both CVSS and EPSS and requires every reported observation to be “mapped with Common Weakness Enumeration (CWE) and Common Vulnerabilities and Exposures (CVE) number”, and it admits no exception on its face. Our position is that where no CVE exists because the defect is in bespoke code, printing “not applicable — no CVE” with the reason against the observation is the only honest way to answer the clause, and that a fabricated identifier plainly is not — that is our reading, not the clause’s words. Settle it with the auditing organisation and the auditee before the report is drafted, not in review. |
| The CWE mapping usually still works, and is worth the effort | MITRE describes a weakness as a condition in a software, firmware, hardware or service component that, under certain circumstances, could contribute to the introduction of vulnerabilities. Most LLM findings land on a recognisable weakness once you look past the model: improper neutralisation at an output sink, missing authorisation at the retrieval or tool boundary, exposure of sensitive information to an unauthorised actor, reliance on an untrusted input in a security decision. | Map to the weakness in the surrounding system rather than to the model’s behaviour. It makes the finding legible to the engineers who will fix it, it survives a model change, and it satisfies the CWE limb of §16.2(ii) even where the CVE limb cannot be satisfied. |
| CVSS does not have a metric for “works one time in five” | CVSS v4.0 is composed of four metric groups — Base, Threat, Environmental and Supplemental — and its Threat group contains a single metric, Exploit Maturity. v4.0 adds Attack Requirements (AT) alongside Attack Complexity, which is the nearest thing in the instrument to a statement that conditions outside the attacker’s control must hold. Neither is a probability field, and stretching either one into a reproduction rate misstates what the vector says. | Score the consequence of a successful exploitation in CVSS, state the reproduction rate as a separate, named field beside it, and say in the methodology section that you have done so. Use the nomenclature FIRST publishes — CVSS-B for base metrics alone, CVSS-BE where environmental values have been supplied — so a reader knows how much of the instrument the number represents. Remember also that a Security Requirement can only lower a v4.0 score: in the absence of explicit values these metrics default to Not Defined, which is equivalent to High. |
| The severity bands are FIRST’s, not CERT-In’s | §5(xxiv) of the 2025 guidelines defines CVSS as producing a score from 0.0 to 10.0 that can be translated into a qualitative representation such as low, medium, high and critical, and defines EPSS as producing a probability between 0 and 1. It does not print the numeric band boundaries. | Attribute the bands to the CVSS specification’s own qualitative severity rating scale when you print them, and never describe them as a CERT-In classification. The same applies to EPSS thresholds: FIRST states that there is no universal threshold and that EPSS does not define score bands with labels such as critical, high or medium. |
| A low-rate finding on a high-volume surface is not a low finding | A bypass that works one time in fifty is a rounding error in a demo and a certainty on an endpoint that handles fifty thousand conversations a week. Probability composes with volume, and with the attacker’s ability to retry, which on most chat surfaces is unlimited and free. | Put the surface’s volume in the finding and let the risk owner do the arithmetic in front of you. Then check the compensating control that actually bites on retry — rate limiting, anomaly detection on conversations, and alerting on repeated refusals from one principal. |
| Closure timelines by severity are required, but the day counts are yours | §16.3(iii) of the 2025 guidelines reads, in full: “Audit report should mention appropriate timelines for closure of vulnerabilities according to severity.” That is the whole clause. It fixes no number of days for any band. | Take the day counts from the auditee’s own information-security policy and print them as the auditee’s, not as CERT-In’s. On LLM findings, add a second timeline the policy will not have: the date by which the finding must be re-run because the provider may have moved the model underneath it. |
Which CWE does an LLM finding map to?
The CWE limb of §16.2(ii) is the one that stays satisfiable, so it is worth doing properly rather than writing “mapped to CWE” in a field and moving on. Every identifier below is a candidate parent: the tester picks the specific child entry against the facts of the finding, and names it in the finding rather than in a methodology appendix. Titles are as MITRE publishes them.
| The finding | Candidate CWE | Which one to pick, and why |
|---|---|---|
| Direct and indirect prompt injection | CWE-1427 Improper Neutralization of Input Used for LLM Prompting, CWE-807 Reliance on Untrusted Inputs in a Security Decision | CWE-1427 is the entry written for this mechanism and is the right parent for the injection itself. Where the injected text decides something the application then relies on, CWE-807 is the more useful of the two to the engineer, because it names the design error rather than the input. |
| Model output reaching a browser or Markdown renderer | CWE-79 Improper Neutralization of Input During Web Page Generation (“Cross-site Scripting”), CWE-116 Improper Encoding or Escaping of Output | The deterministic half of the engagement, and the one with the cleanest fix. Use CWE-116 where the defect is the missing encoder and CWE-79 where a script or an active element actually executed. |
| Model output reaching a shell or a database | CWE-78 Improper Neutralization of Special Elements used in an OS Command (“OS Command Injection”), CWE-89 (“SQL Injection”) | Ordinary identifiers for an ordinary defect. The presence of a language model between the attacker and the sink changes the entry point and nothing about the weakness. |
| Excessive agency and confused-deputy tool invocation | CWE-862 Missing Authorization, CWE-863 Incorrect Authorization, CWE-285 Improper Authorization, CWE-441 Unintended Proxy or Intermediary (“Confused Deputy”) | CWE-441 is the one that names what is actually happening when a shared service account acts on a stranger’s instruction, and it is worth using for that reason: it gets the architecture into the finding title. The authorisation entries carry the fix. |
| Corpus and cross-tenant leakage | CWE-200 Exposure of Sensitive Information to an Unauthorized Actor, CWE-862 Missing Authorization at the retrieval boundary | Map the authorisation failure rather than the disclosure where the retrieval layer fetched what it should not have. CWE-200 describes the outcome, while the retrieval-side entry describes the defect somebody has to fix. |
| Unvalidated model output consumed by the application | CWE-1426 Improper Validation of Generative AI Output | The entry for treating generated text as trustworthy input to your own logic — a decision, a parse, a record written. Distinct from the sink entries above, which are about encoding rather than about belief. |
| Unbounded consumption | CWE-770 Allocation of Resources Without Limits or Throttling | The identifier that makes a cost finding legible to a platform team who have solved this problem before under a different name. |
| Supply chain and component provenance | CWE-829 Inclusion of Functionality from Untrusted Control Sphere, CWE-494 Download of Code Without Integrity Check | Both predate this field by two decades and both fit a tool server whose descriptions are re-read at runtime. Name the component and the trust decision in the finding, not just the identifier. |
Where MITRE ATLAS fits, and why it survives the §8(ii) objection
CWE names the weakness. It does not name the adversary behaviour, and a report that wants shared vocabulary for the technique has been reaching for the OWASP Top 10 because there was nothing else on the shelf. MITRE ATLAS — Adversarial Threat Landscape for AI Systems — is the better answer, and the reason is structural rather than a matter of taste. §8(ii) objects to “limited lists” being used as audit standards because a ranked top-ten cannot support a coverage claim: there is nothing in it that says what was examined and found sound. ATLAS is not that shape. It is a knowledge base of observed adversary tactics and techniques against AI systems, organised as a matrix with stable identifiers — tactics numbered AML.TAnnnn and techniques AML.Tnnnn — in the same form as ATT&CK, and its published dataset names LLM prompt injection among the techniques. Used as a naming layer rather than as a scope, it gives a finding a technique identifier a reader elsewhere can look up, without ever being offered as the thing the audit was measured against. One caution: MITRE’s own data repository marks the dataset served at the historical path as no longer updated, so confirm a technique identifier against the current site before printing it in a report. The 2026 edition of the OWASP list maps its own entries to ATLAS, to CWE and to NIST AI 600-1, which is a useful cross-check and a sign of where the field’s shared vocabulary is settling.
Score the consequence
What became reachable, by whom, with what privilege, and what it would cost. This is the same discipline any penetration test uses, and it is the half of the finding that does not move when the model does.
State the probability separately
Reproduction rate over a stated number of fresh sessions, at the production sampling settings, with the date. It is a measurement, it belongs beside the score rather than inside it, and the methodology section should say that is where you put it. How large the denominator needs to be, and what a zero actually supports, is worked out in the arithmetic above.
Say what the number omits
FIRST’s own nomenclature exists for this: CVSS-B is base metrics alone, CVSS-BE adds environmental values. Print which one you produced, and remember that in v4.0 a Security Requirement can only lower a score, because the absent values already default to the equivalent of High.
One clause governs the tempo rather than the score, and on this class of finding it bites early. §16.3(ii) requires critical and high findings to be notified to the auditee “during the course of audit on as and when found basis” rather than held for the report, and §16.4 requires high-risk vulnerabilities — discovered breaches, vulnerabilities with known high exploitation rates, unmonitored or untraceable access — to be reported immediately on discovery. An indirect injection that reaches a tool which moves money or sends mail is that kind of finding. Raise it the day you find it, with the reproduction rate you have at that point, and refine the number afterwards. More on how these land in a written deliverable is on reading a VAPT report.
Why an LLM finding can reopen with no change on your side
How often should an LLM application be re-tested?
Every other kind of application testing rests on an assumption so ordinary that nobody states it: the system under test only changes when somebody deploys something. That assumption does not hold here. The model can be replaced underneath an unchanged identifier by a third party on their release schedule, and four other triggers move the system without touching the application repository. So the cadence question has an unusual answer: annually, as §13.1.2(i) asks of any audited system, and before any Major Change, which §13.1.2(iii) requires — plus a regression set that runs on five specific events.
| Trigger | Why it happens | What it does to your findings | The control |
|---|---|---|---|
| The provider updates the model and the identifier stays the same | An alias that points at “the current version” is a moving target by design. Behaviour that was refused last month can be accepted this month with nothing in your repository having changed, and the first you hear of it is a user report or an incident. | Reopens findings you closed, and can also close findings you had not fixed — which is worse, because a finding that disappears without a fix will come back. | Pin the version where the provider allows it. Where it cannot be pinned, treat a provider-side model change as a change to your system, run the regression corpus against it, and hold the previous version’s results for comparison. |
| Your own prompt or tool schema changes | Prompt text and tool definitions usually live outside the application’s change-management process — in a configuration store, a prompt-management product, or a spreadsheet — and are edited by people who do not think of themselves as changing production code. | Silently removes the mitigation a closed finding depended on. This is a common cause of a reopened LLM finding. | Put the prompt and the tool schema under version control and change management. §9.5(ii) of the 2025 guidelines asks that version control and change management be effectively implemented so that the assets that were part of the audit scope can be backtracked. On an LLM system the prompt is one of those assets. |
| The retrieval corpus changes | Corpora grow continuously and by design, and the growth is mostly ingestion from sources you do not control. A new source is a new trust boundary. | Creates injection paths that did not exist at the time of the audit, and changes what retrieval returns, so that a finding neither reproduces nor is fixed — it has simply moved. | Re-run the indirect-injection canaries against each new source as it is added, and treat the addition of an externally-writable source as a change that warrants testing before it goes live. |
| A guardrail or filter is tuned | Filters are tuned against false positives, usually in response to a complaint from the business, and the tuning is rarely re-tested against the security cases. | Reopens bypasses in the exact category that was loosened, which is a predictable enough pattern to test for directly. | Keep every successful bypass from every engagement as a regression case, and run the set whenever a filter is changed. |
| A new tool is added to the agent | Tools are added by feature teams to solve product problems. The capability inventory is rarely revisited, and the new tool inherits whatever identity the stack already had. | Widens the blast radius of every injection finding at once, including ones you closed, because the class was never closed — only its consequence was bounded. | Make the tool inventory a reviewed artefact with an owner, and require the worst-true-sentence description for each new entry. Classify the addition of a tool that writes, spends, sends, deletes or grants as a change that needs testing before implementation. |
What the clauses require
§13.1.2(i) states that the cyber security audit “should be conducted at least once in a year”. §13.1.2(iii) requires every change to be classified as Minor or Major through a formal change-management process, and a Major Change — defined by risk rather than by size, expressly including “configuration adjustments that affect sensitive data or critical infrastructure” — must undergo a cyber security audit before implementation. §16.3(iv) states that follow-up audits should be performed after remediation to verify closure, and §15.2.1(iii) requires the auditing organisation to revalidate observations from the previous audit cycle, with anything still present recorded as a repeat observation.
What is our reading, and not the clause
Nothing in the 2025 guidelines says that swapping a model, editing a system prompt or adding a corpus source is a Major Change. The instrument defines the test — high-risk and impactful to security — and leaves the classification to the change-management process it requires you to run. Our position is that a model swap, a prompt edit that touches a safety constraint, and the addition of a tool that writes, spends, sends, deletes or grants all deserve to be put through that test rather than exempted from it because they do not look like code. That is an argument for how to apply the clause, and we would rather label it than let it read as the clause speaking.
The practical output of all this is a regression corpus: every successful payload, every canary document, every bypass variant, with its recorded configuration and its measured rate, held as a runnable set. It is the most valuable thing an LLM red team leaves behind, it is worth more the second year than the first, and it should be a named deliverable in the scope of work rather than something the tester happens to keep. The closure discipline it plugs into is the same one described on VAPT re-testing and closure.
When is a probabilistic finding closed?
§16.3(iv) requires the follow-up audit to verify closure and §15.2.1(iii) requires last cycle’s observations to be revalidated, with anything still present recorded as a repeat observation. Neither clause says what closure means for a finding that reproduced thirteen times in twenty, and a retest that produces two in twenty is neither obviously a pass nor obviously a fail. So the rule has to come from the engagement, and it has to be agreed in writing before the fix ships rather than after — because afterwards both sides have an interest in the number. Four outcomes, and the rule for each.
| Outcome | What has to be true | The trial count | How long it lasts |
|---|---|---|---|
| Closed by architecture | The path is structurally removed: the tool is withdrawn, per-user identity is enforced at the tool boundary, the sink encodes, the entitlement filter runs before retrieval, the loop has a server-side bound. | None needed. The verification is deterministic — read the configuration, exercise the boundary once, confirm the denial. | The only outcome that survives a model change. This is why the fix column on every attack class above points outside the model. |
| Closed by evidence | Zero successes at a pre-agreed n, with the model identifier, the system-prompt version and the corpus snapshot confirmed unchanged from the original finding. | n chosen so that the rule-of-three upper bound sits under the risk owner’s tolerance: zero in twenty supports “under about 15%”, zero in a hundred supports “under about 3%”. Print the bound, not just the zero. | Holds only for the recorded configuration. Attach the retest trigger list, because this closure is a statement about a system that can move. |
| Mitigated, not closed | The rate is materially reduced but not zero. Print both fractions — 13/20 became 2/50 — and state whether the intervals overlap, because two rates whose intervals overlap have not been shown to differ. | At least the original n, and usually more: demonstrating a reduction takes more attempts than demonstrating existence did. | The finding stays open with a named residual owner and a review date. A mitigated finding recorded as closed is how a repeat observation appears in next year’s report with nobody expecting it. |
| Not verifiable — re-baselined | The configuration moved between the test and the retest: the model identifier changed, the system prompt was bumped, the corpus was re-indexed, a tool was added or renamed. | The retest is a new test. Run it, record the new configuration, and state plainly that it measures a different system from the original finding. | The original finding is neither closed nor reopened. It is re-baselined, and saying so is the honest alternative to a closure nobody can defend. |
Two procedural rules make the table usable. First, agree the retest trial count in the scope of work, before the fix ships. Second, write the closure outcome into the finding as one of those four words rather than as a tick — because “closed by architecture” and “closed by evidence” carry different obligations next year, and only the first of them survives a model change.
Twelve steps, with the gates in the right order
What does an LLM red team engagement look like, step by step?
The sequence below is the shape of an engagement rather than a schedule. Note how much of it happens before anybody types a prompt: on an LLM system the artefact capture is the engagement’s foundation, because a finding recorded against an unrecorded configuration cannot be retested and therefore cannot be closed.
Name the class of engagement, in writing
Decide whether you are buying an AI System Audit under §6(xxiii), a Red Team Assessment under §6(xvi), an application security test with an AI annexe, or a safety and content evaluation — which is a fourth thing and not a security engagement at all. They produce different reports, different people and different prices. The decision table under the instruments section takes you through it by situation.
Pin the nine properties of the system under test
Model and version, system prompt, retrieval corpus and its trust boundary, tool and function surface, the identity the system acts under, memory and session state, the humans in the loop, the output sinks, and the environment with its differences from production.
Agree the rules of engagement, including what you may not do
Written approvals before testing, a named contact who can stop it, the boundary on the code interpreter and on any tool that writes, the tester-traffic convention, and the prohibition list — all three columns of which are set out in the rules-of-engagement section above, because the LLM-specific items are the ones a conventional penetration-testing document does not contain. Where any part of the work touches employees rather than systems, get the consent position right: §15.2.2(iii) requires social-engineering and process testing to use anonymised or statistical techniques, to identify or penalise no individual, and not to involve customers, partners, vendors or other third parties without specific written consent.
Take the artefacts before the first prompt
System prompt text with a hash, tool schema export, corpus inventory with an as-of date, identity diagram, and the list of sinks. Everything the report will need to be reproducible has to be captured before testing changes anything.
Establish baselines and record the refusals
For each policy the business believes it has, ask plainly and record the answer. Baselines are what make later results measurable, and they frequently find missing requirements before any attack does.
Work the direct channel
Direct injection and guardrail bypass, in fresh sessions, at production settings, with every successful variant repeated enough times to produce a rate rather than an anecdote.
Work the indirect channels
Place marked canaries through each ingestion route you have agreed, wait for the pipeline, and test retrieval from a different account than the one that placed the document. Then do the same through tool output and through a fetched page.
Work the tool surface
Identity at each call, parameter control, the approval gate as an attack surface, loop bounds, and then the chain: outside party to document to retrieval to tool call in another user’s session.
Work the sinks
Browser, Markdown renderer, shell, database, document generator, webhook, and the next model’s context. This is where the engagement produces its deterministic evidence.
Score, and say what the score does not carry
CVSS for consequence with the nomenclature stated, CWE mapped to the weakness in the surrounding system, EPSS marked not applicable where there is no CVE, and the reproduction rate printed as its own field with its denominator.
Report, including the negatives
Findings with reproduction steps a stranger can follow, the controls that held, the unknowns marked as unknowns, the environment named, and the limitations paragraph written before the findings rather than after.
Retest, then set the regression cadence
Verify closure against the same recorded configuration, at the trial count agreed before the fix shipped, and record which of the four closure outcomes each finding reached. Keep every successful payload as a regression case, and agree the triggers that re-run the set: a model change, a prompt change, a new corpus source, a filter tuning, a new tool.
An invented engagement, with numbers
Everything in this example is invented for illustration: it is not a client, not a sector, not a real engagement, and it makes no claim about anybody’s results, timelines or outcomes. It exists to show how the gates in the sequence above fall into a calendar and what the arithmetic looks like when you put numbers against them.
| Days 1 to 6 — scope | The system: an internal support assistant with retrieval over a ticket archive, a product wiki and customer-uploaded attachments, eleven tools of which three write, one shared service account behind all eleven, per-user memory, and an approval gate on two of the three writing tools. Nine properties recorded. Two are found to be unknown at the start — nobody can immediately say whether the model identifier is pinned, and nobody owns the prompt store. Both become findings before any testing happens, which is common enough to expect. |
| Days 7 to 9 — artefacts and approvals | System prompt exported and hashed, tool schema exported, and corpus inventory produced with an as-of date and, against each source, the answer to who outside the organisation can write into it. Three of nine sources turn out to be externally writable. Written approvals obtained for testing, the code-interpreter boundary agreed, a named contact who can stop the engagement identified, and a tester-account convention agreed so the defenders can tell test traffic from the real thing. |
| Days 10 to 12 — baselines | Eighteen policy statements the business believes it has, asked plainly, in fresh sessions, twenty times each. Fourteen hold at a rate above 0.9. Two hold intermittently. Two are never enforced at all, and inspection of the exported prompt shows why: they were never written into it. Those two are specification findings and they go to the product owner rather than to the platform team. |
| Days 13 to 17 — direct channel | Direct injection and guardrail bypass against the two intermittent policies and the tool surface. Seven variants found, each repeated in twenty fresh sessions at production temperature, giving rates between 0.15 and 0.8. None reaches a writing tool directly, which is a real result and is reported as one. |
| Days 18 to 23 — indirect channels | Canaries placed through all three externally-writable sources. Ingestion observed at twenty-five minutes for attachments and at just under a day for the wiki. Two of three fire. The one that matters fires in a different user’s session, in a different tenant, and reaches the refund tool — where the approval gate holds on every attempt. The chain is written up as one finding with the gate recorded as a control that held, and the model-supplied gate text is written up as a second, dependent finding. |
| Days 24 to 26 — tool surface and sinks | Identity confirmed as one shared service account for all eleven tools: reported as the engagement’s highest-consequence architectural finding, because it is what gives every injection finding its blast radius. Parameter control tested on all eleven. Sinks: the web client renders Markdown with remote images permitted, which is an exfiltration path with a clean CWE and a one-line fix, and it is the finding the engineering team closes first. |
| Days 27 to 31 — reporting | Nineteen findings. Every one carries model identifier and version, sampling parameters, prompt hash, corpus snapshot, transcript, attempts and successes. Limitations written first: the sandbox had a smaller corpus than production, one tool could not be exercised, and the provider changed a model identifier on day 22, which is recorded with its timestamp so that later disputes have somewhere to start. Two findings had already been raised on the day they were found, under the as-and-when duty. |
| Days 32 onward — closure and regression | Remediation owned by three different teams: prompt and policy by the product owner, identity and tool permissions by the platform team, ingestion review by the data team. Retest run against the recorded configuration rather than against the current one, so that closure means something. The regression corpus — seven direct variants, three canaries, nineteen findings’ worth of reproduction steps — is handed over as a deliverable with the triggers that re-run it. |
The arithmetic worth taking from an invented example: of the thirty-one days, fourteen are testing and the rest are scope, artefacts, approvals, baselines and reporting. An engagement that starts testing the week after the purchase order has not been efficient — it has skipped the part that makes the findings reproducible.
Twelve ways an LLM red team produces nothing useful
Why do LLM red team engagements fail?
None of these is exotic. Each one is a way of spending a budget and ending with a document that cannot be acted on, retested or defended, and most of them are visible in the proposal before the engagement starts.
The engagement is a chat session with a tester in it
Somebody spends two days typing clever things into the chat box, produces a document of screenshots, and nobody has looked at the corpus, the tools or the sinks. It is the cheapest thing to sell and the least useful thing to buy, and it is often what is sold as LLM red teaming.
The evidence is a screenshot
A screenshot of a stochastic system proves that something happened once. It does not carry the model version, the temperature, the prompt bytes, the corpus state or the denominator, so it cannot be retested, argued with, or defended when it stops reproducing.
The model version is not recorded, and the finding dies at retest
Six weeks later the behaviour is gone. Nobody can say whether the fix worked, whether the provider changed the model, or whether the tester got lucky the first time. The engagement has produced an unresolvable dispute instead of a closed finding.
The system prompt is treated as the security boundary
The report recommends adding instructions to the prompt, the instructions are added, the retest passes at the sampled temperature, and the class is still open. Prompt hardening changes the rate, but only the tool boundary, the authorisation model and the sinks change the consequence.
Only the chat box is tested, and the corpus is assumed friendly
The most valuable channel is the one nobody types into. A clean report on a retrieval application whose corpus ingests customer-supplied documents, with no indirect-injection testing, is a clean report on the wrong system.
A jailbreak is reported as critical with no consequence attached
The model said something rude or off-policy. That is a content and brand finding. It becomes a security finding when it reaches a tool, a document, another tenant or a sink — and the report should say which, or say that it reached none.
A hallucinated system prompt is reported as an extraction
Models will produce a plausible system prompt on request whether or not they are disclosing the real one. Diff against the auditee’s actual text before writing it up, and say in the finding that you did.
The tester’s own traffic is mistaken for an incident, or an incident is mistaken for the tester
Both directions are dangerous. Agree a source identifier and a tester-account convention up front so the defenders can tell the difference, and agree the escalation path for the case where something in the logs turns out not to be yours — which is, on our reading, where the six-hour reporting duty under the 2022 Directions becomes live. Item (xx) of Annexure I names the class of incident rather than adjudicating a fact pattern, so the escalation path should end at somebody who can take that decision, not at the tester.
Nobody agreed what happens to the transcripts
Conversation logs from a real application contain customer data by definition, which makes this a data-protection question before it is an audit-data question. Under the DPDP Act, 2023 the auditee is the Data Fiduciary and the testing organisation is a Data Processor handling personal data on its behalf, engageable under section 8(2) “only under a valid contract”, with responsibility staying with the Fiduciary under section 8(1). Settle retention, storage location, purpose and destruction in the engagement terms on that footing, and note separately that CERT-In’s Policy Guidelines for Handling Audit related Data require audit data to be stored only on systems located in India and not to be shared with an overseas partner unless the auditee specifically authorises it.
The re-test is priced as a change request
On these systems retesting is not the tail of the engagement, it is the mechanism by which the engagement keeps meaning anything. §13.1.1(g) puts follow-up audits in the scope of work and §9.3(iv) puts them inside the scope or the RFP. A proposal with no retest and no regression cadence is priced for a snapshot of a system that will not hold still.
The report scores a model behaviour instead of a system weakness
A severity attached to “the model can be persuaded to ignore instructions” tells an engineering team nothing they can act on. Attach it to the weakness in the surrounding system — the missing authorisation check, the unencoded sink, the over-privileged tool — and both the CWE and the owner fall out of it.
The findings never reach the people who can fix them
LLM findings cut across three teams that do not usually share a backlog: the product team that owns the prompt, the platform team that owns the tools and identity, and the data team that owns the corpus. Name an owner per finding at the exit conference or the report will circulate without moving anything.
Twelve questions to ask before you sign
What should you ask before buying an LLM red team?
Each of these is answerable in a sentence by a supplier who has done this work, and unanswerable by one who has not. Against each, the answer that should worry you. Read them down a phone on the way to the meeting. Every one is drawn from something that goes wrong further up this page.
How many fresh sessions will each successful variant be repeated in, and is that number in the scope of work?
The answer that should worry you: “As many as it takes.” The session count is the denominator on every finding and the line most easily cut from a quote. If it is not written down before the work starts, it will be decided afterwards by whoever is running out of days.
Which ingestion routes into the corpus are in scope, and who places content through each one?
The answer that should worry you: “We will test the application.” If nobody has named the routes, the indirect-injection class is not being tested, and that is the class the attacker uses.
How many tenants and how many privilege levels will have accounts issued, and by when?
The answer that should worry you: “One admin account.” Cross-tenant retrieval cannot be tested from one account, and an admin account hides every authorisation finding by being allowed to see everything.
Is the regression corpus a named deliverable, and does it come to us in a runnable form?
The answer that should worry you: “We keep our methodology confidential.” The payloads, canaries and reproduction steps are the most valuable artefact of the engagement and they are about your system. A supplier who keeps them is selling you the same first year twice.
Is the retest inside the price, and what exactly does it re-run?
The answer that should worry you: “Retest is a change request.” §13.1.1(g) puts follow-up audits in the scope of work and §9.3(iv) puts them inside the scope or the RFP. On a system that moves without a deployment, a quote without a retest is a quote for half the job.
What is the retest trial count, and is it agreed now?
The answer that should worry you: Silence. Agree it before the fix ships, because afterwards both sides have an interest in the number — and see the closure table above for what each outcome then means.
Who signs the report, and are they on the roster the requirement points at?
The answer that should worry you: A name nobody can place. §9.5(iii) requires reports to be signed only by manpower declared to CERT-In where empanelment is in play. Where it is not in play, ask anyway: you are buying an opinion, and opinions have authors.
What happens to the transcripts — where stored, for how long, destroyed when, and under whose terms?
The answer that should worry you: “Standard NDA.” Conversation logs from a production system are personal data. The DPDP Act, 2023 makes you the Data Fiduciary and your tester a Data Processor who may only be engaged under a valid contract, and CERT-In’s data-handling guidelines put the storage in India.
What is the environment, and what does it lack compared with production?
The answer that should worry you: “A copy of production.” It never is. §15.2.3(i) requires the environment to be specified. The useful artefact is the written list of differences, because each difference suppresses a class of finding.
What are the man-hours, and how do they divide across channels, tools and sinks?
The answer that should worry you: A calendar figure only. §13.2.2(ii) requires the test plan to state both. Divide the man-hours by your channel count and your tool count and see what each is actually getting.
What is the escalation path if you find somebody else’s injected document in our corpus?
The answer that should worry you: “We would tell you in the report.” That is the one finding that cannot wait for the report. Name the person, the channel and the hour.
Who removes the canaries and the test accounts, and against what checklist?
The answer that should worry you: A shrug. A planted document left in a production index after the engagement is a defect you paid for.
The equivalent list for conventional testing, anchored clause by clause, is on what VAPT actually is, and the auditor-side constraints those questions run into are on a CERT-In auditor’s rules of engagement.
What drives the cost of an LLM red team
How much work is an LLM red team, and what makes it bigger?
Seven drivers, none of which is the model. Two quotes for the same system can differ by a factor of five without either being dishonest, because they have made different assumptions about the numbers below — and only one of those numbers is normally written down. Ask for each, and the quotes become comparable.
| Driver | Why it drives cost | The number to ask for |
|---|---|---|
| Ingestion routes into the corpus | Each route is a separate indirect-injection campaign with its own placement mechanism, its own review step and its own pipeline delay. You cannot parallelise waiting for an index. | How many routes, and what is the measured ingestion-to-retrievable delay on each? |
| Tools, and how many of them write | A read-only tool is an afternoon. A tool that writes, spends, sends, deletes or grants needs an approval boundary, a stop point and a chain built to it. | The tool count, split into read and write, taken from the exported schema rather than from memory. |
| Privilege levels and tenants | The authorisation findings exist only in the comparison between two accounts. One account is one pass, while four accounts across two tenants is closer to eight. | How many accounts at how many levels, in how many tenants, and who issues them under §15.1.2? |
| Output sinks | Each sink is a separate test with a separate fix owner, and this is the half of the engagement that produces deterministic evidence. | The sink list from the scope, with the encoding applied at each. |
| Fresh sessions per variant | This is the multiplier that turns anecdotes into rates, and the one quietly trimmed in every negotiation. Attempts × variants × classes = sessions, and sessions are the bulk of the tool time. | The arithmetic, written out. If a proposal says twenty sessions per variant and prices seven variants across ten classes, ask where 1,400 sessions sit in the man-hours. |
| Retest and regression | Inside the price or outside it, and if outside, the quote is not comparable with one that includes it. | Is the follow-up audit in scope under §13.1.1(g) and §9.3(iv), at what trial count, and is the regression corpus a deliverable? |
| What your side has ready on day one | The system prompt, the tool schema, the corpus inventory, the accounts and a named contact. Every day the team waits for one of them is a day you bought. | A readiness date per artefact, agreed before the engagement starts rather than chased in week one. |
The arithmetic that is quietly cut from every quote
Attempts × variants × classes = sessions. That product is the bulk of the tool time on an LLM engagement, and it is the only line in a proposal that can be reduced without anything visibly changing in the scope: halve the attempts per variant and the document still lists the same classes. So compute it yourself. §13.2.2(ii) requires the test plan shared by the auditing organisation to state both calendar time and man-hours. Divide the man-hours by your channel count, your tool count and your sink count, then check whether what remains could plausibly run the number of sessions the proposal implies. Where it cannot, the reproduction rates in the report will have small denominators, and a small denominator is a wide interval on every finding in the document.
What Indian law and Indian guidance actually say about this
Is AI red teaming required in India?
No instrument requires an LLM red team by that name. What exists is more useful than a mandate and more specific than a trend: a reportable incident class that already covers AI and ML systems, a class of engagement that already covers AI system audits, a definition of a red team assessment that fits this work exactly, and a national conference record in which the Government of India names AI-driven red teaming methodologies among the subjects the audit ecosystem is now concerned with. Each row below states the instrument, what it says, and what it means for an engagement like this one.
| Instrument | Where | What it says | What it means here |
|---|---|---|---|
| CERT-In Directions of 28 April 2022, Annexure I item (xx) — attacks on AI and ML systems | Annexure I, item (xx), read with direction (ii) | Attacks or malicious or suspicious activities affecting systems, servers, software or applications related to artificial intelligence and machine learning are one of the twenty mandatorily reportable incident types, reportable to CERT-In within six hours of noticing the incident or being brought to notice about it. | A red-team engagement can walk into a real one. If your testing surfaces evidence that somebody else has already done what you are doing — an injected document you did not place, tool calls in the logs that nobody authorised — then, on our reading, the activity falls within item (xx) and the six-hour clock is the auditee’s rather than the auditor’s. Item (xx) names the class. It does not settle any particular fact pattern, so take advice on the one in front of you. What the instruments do settle is that the duty stays with the entity: CERT-In’s FAQ of May 2022 states that the obligation to report a cyber incident “is neither transferrable nor indemnified or dispense with”. |
| CERT-In Directions of 28 April 2022, direction (iv) — logs | Direction (iv) | Entities must enable logs of all their ICT systems and maintain them securely for a rolling period of 180 days, and the same shall be maintained within Indian jurisdiction. | On an agent stack, “logs” has to mean the conversation, the tool calls, the arguments and the returns — otherwise an incident is unreconstructable. Read the residency limb with the FAQ’s own qualification at Q35: the logs “may be stored outside India also as long as the obligation to produce logs to CERT-In is adhered to by the entities in a reasonable time”. That nuance matters when the model is hosted in another region and the transcript lands wherever the provider puts it. |
| CERT-In Comprehensive Cyber Security Audit Policy Guidelines, v1.0, 25 July 2025 — §6(xxiii) AI System Audits | §6(xxiii), with §6’s opening sentence | AI System Audits are a class of engagement in their own right, defined as the “evaluation of AI systems for security, ethical alignment, transparency, data integrity, and resilience to adversarial manipulation”. §6 opens by saying the types listed are “including, but not limited to” those listed — an open list. | It is the clause that puts an AI system inside the Indian audit vocabulary rather than outside it, and “resilience to adversarial manipulation” is as close as the instrument comes to naming this work. It does not prescribe a method, so the method has to come from your scope of work. |
| CERT-In Comprehensive Cyber Security Audit Policy Guidelines — §5(xxiii) and §6(xvi) Red Team Assessment | §5(xxiii) and §6(xvi) | A Red Team Assessment is “an exercise, reflecting real-world conditions, that is conducted as a simulated attempt by an adversary to attack or exploit vulnerabilities in an enterprise’s information systems”. The same definition appears in both sections, word for word. | Useful for setting expectations. The definition is objective-led and adversary-simulating, which is what an LLM red team is. It is not the same thing as the vulnerability-assessment and penetration-testing pair most Indian scopes are written around. Name the class you are buying. |
| CERT-In audit policy guidelines — §8(ii), on limited lists | §8(ii) | “The limited lists such as OWASP Top 10, SANS Top 25 and similar, should not be considered as standards or references for audits.” Audits should instead include discovery of all known vulnerabilities based on comprehensive standards and frameworks, and the clause names ISO/IEC standards, the Cyber Security Audit Baseline Requirements, the CSA Cloud Controls Matrix, OSSTMM3, the OWASP Web Security Testing Guide, the OWASP Application Security Verification Standard, the OWASP Mobile Security Testing Guide and the OWASP DevSecOps Maturity Model. | Read honestly, this cuts against using the OWASP Top 10 for LLM Applications as an audit criterion, by the clause’s own logic and its “and similar”. We cite two of its items on this page as vocabulary — LLM01 and LLM03 — because they are the shared names for two classes, and we do not present the list as the scope. The scope comes from the system: the model, the prompt, the corpus, the tools, the identity, the memory, the humans and the sinks. |
| CERT-In SAMVAAD 2026, and the AMBAK launch | PIB release 2256581, 29 April 2026 | The Government of India’s own release on the conference names AI-driven red teaming methodologies among the topics of the 87-plus presentations selected from more than 200 submissions, alongside AI-driven cyber risks including “emerging frontier AI models with advanced cyber capabilities”, and the auditing of cloud systems, APIs and operational technology. CERT-In’s own conference proceedings are a separate document. They record the AMBAK launch, the 500-plus delegates, the panel and the two working groups, and their own topic list is worded differently. | It is the Government of India’s own statement that this is now part of what the national audit ecosystem is concerned with. It is not a standard and should not be cited as one — but when somebody asks why an Indian audit programme should have an LLM red team in it, this is the answer with a source behind it. |
| NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework | NIST AI 600-1, July 2024 | The US national standards body’s profile for generative AI risk. It is the international document that treats AI red-teaming as a named practice rather than as a topic: it repeatedly directs organisations to conduct AI red-teaming and structured human-feedback exercises, to assess resilience against abuse, and to document the instructions given to red-teamers. | The nearest international counterpart to §6(xxiii). Where CERT-In defines a class of engagement and leaves the method open, NIST AI 600-1 describes the practice and leaves the audit machinery open — so the two are complementary rather than competing, and a scope of work can reasonably cite both. It is guidance, not a standard you certify against. Our comparison of the management-system route and the framework route is on ISO 42001 versus the NIST AI RMF. |
| ISO/IEC 42001:2023 | The AI management system standard | It is the management-system standard for artificial intelligence: the certifiable framework for how an organisation governs the AI systems it builds and uses. | A red team is a source of evidence for that system, not a substitute for it. If an organisation is certifying to ISO/IEC 42001:2023, the engagement should produce artefacts the management system can actually consume — findings with owners, a treatment decision, and a retest date. |
| Regulation (EU) 2024/1689, the EU AI Act — Article 14 | Article 14, human oversight for high-risk systems | Article 14 is the human-oversight provision for high-risk AI systems. Regulation (EU) 2026/1744, the Digital Omnibus on AI, came into force on 27 July 2026 and defers the stand-alone high-risk obligations in Annex III to 2 December 2027 and those in Annex I to 2 August 2028. | Relevant to Indian teams selling into Europe, and relevant to the humans-in-the-loop row in the scoping table. Oversight that exists on paper and collapses under approval volume is the thing to test and to report. Note the two regulations are different instruments: 2024/1689 is the AI Act, and 2026/1744 is the Digital Omnibus that moved its dates. Neither is linked on this page, because EUR-Lex returned a zero-byte body for both when it was checked. |
Which engagement class should you buy?
The table above says what each instrument contains. This one says which to name in your scope of work, which is the decision with the largest commercial consequence and the one most often made by whoever wrote the template. Find your situation on the left. The last column is the single question that settles it.
| Your situation | The class to name | The deliverable | Empanelment in play? | The question that settles it |
|---|---|---|---|---|
| A customer, tender or regulator named a class | Whatever that document names, verbatim | As specified in the requesting document | Read the requesting document — it is the only thing that decides this | Ask the requesting party, in writing, whether empanelment is a requirement or a preference. |
| Thin wrapper: no retrieval, no tools | Application VAPT with an AI annexe | A VAPT report, with a section on the model-facing surfaces | Only if the surrounding requirement brings it | Can the model cause anything to happen other than text being returned? If no, this is an application test. |
| Retrieval, read-only, multi-tenant | AI System Audit under §6(xxiii), scoped to the corpus and the entitlement model | An audit report against a stated scope, with the corpus inventory as an annexe | Usually not, unless the system is a Government one | Does filtering happen before retrieval or after it? The answer decides how much of the engagement is authorisation work. |
| Agentic, with tools that write | Red Team Assessment under §6(xvi), objective-led | An objective-led report with chains, not a vulnerability list | Only where the requirement brings it | Name the objective in the scope — “cause a write in another tenant from outside the organisation” — and the class follows from it. |
| A Government system, or one heading for hosting clearance | The class the hosting or clearance route requires, performed by an empanelled organisation | The report and certificate that route expects | Yes — this is the case where it is load-bearing | Ask the hosting authority which document it will accept before anything is scoped. |
| An ISO/IEC 42001 certification programme needing evidence | AI System Audit under §6(xxiii), with findings shaped for a management system | Findings with owners, treatment decisions and retest dates | No | Will the certification body see this? If so, the deliverable matters more than the label. |
| A pre-launch gate | Red Team Assessment, time-boxed, with the regression corpus as a named deliverable | A go or no-go finding set, plus the runnable regression set | No | What would stop the launch? Test that first, and agree in advance who can say no. |
| Somebody wants to know whether the model says harmful things | A safety and content evaluation — which is not a security engagement | Rates against a content taxonomy the business agreed | No | This is the confusion behind a great many AI RFPs. A safety evaluation measures what the model says, while a security engagement measures what an attacker can make the system do. They use different people and produce different documents, and asking one supplier for both without saying so gets you neither. |
The distinction worth carrying away is the last row’s. A safety and content evaluation measures whether a model produces harmful, biased or off-policy text. A security engagement measures what somebody outside your organisation can make your system do. Both are legitimate. They use different people, produce different documents and answer to different owners, and an RFP that asks for one while describing the other will get a proposal that satisfies neither. Say which you want in the first line.
The cloud limb, which most LLM stacks trip over as well
Annexure I to the 2022 Directions lists, at item (xviii), attacks or malicious or suspicious activities affecting cloud computing systems, servers, software and applications — alongside item (xx) for AI and ML systems. Almost every LLM application in production is also a cloud application, which means a single incident can sit inside both classes, and both carry the same six-hour reporting window. The practical consequence is that the incident-response runbook for an AI feature is not a new document: it is the existing one, with the model provider, the vector store and the tool-call log added to the evidence list. If you are working through that side of it, the detail is on the CERT-In Directions of 2022, explained and on what cloud security is.
Six sentences you will hear, and what to say back
Each of these is reasonable on its face, which is why it works. Each reply is a sentence to say out loud rather than a paragraph to absorb.
“The provider already red-teams the model, so we do not need to.”
They test the model. You are shipping an application: their model, your system prompt, your corpus, your tools, your identity model, your sinks. Every finding in the scoping table above lives in the part they have never seen. Provider-side safety work is a reason to spend less time on content policy and no reason at all to spend less time on the tool boundary.
“We have guardrails.”
Which one — an input classifier, an output classifier, a separate moderation model, a regex list, or policy text in the prompt? They fail differently and they are owned by different teams. Then the second question: what is the measured bypass rate at your production temperature, and who measured it?
“It is read-only, so the worst case is embarrassment.”
Read-only over a corpus that contains other tenants’ documents is a data-disclosure surface, and read-only output that reaches a browser, a Markdown renderer or another model is an execution surface. Walk the sinks list before accepting the premise.
“We will test it once before launch.”
That gives you a result about a configuration that will not exist in a month. The provider can move the model, your team can edit the prompt, the corpus grows every day, and a filter gets loosened the first time the business complains. Buy the regression cadence with the test, not after the first surprise.
“Prompt injection is unsolved, so there is no point testing for it.”
The class is unsolved, but the consequence is entirely within your control. The point of the engagement is not to prove that injection is possible — assume that — but to establish what it reaches once it works, and to shrink that. That is an architecture result, and it is durable in a way that prompt wording is not.
“Our findings are not reproducible, so the report is not actionable.”
That is a reporting problem, not a property of the system. A finding with the model version, the sampling parameters, the prompt bytes, the prompt hash, the corpus snapshot and a stated success rate over a stated number of fresh sessions is reproducible in the only sense that matters for a stochastic system: another tester can run it and get the same rate.
The roles, stated plainly
LLM findings cut across more teams than conventional application findings, which is why naming owners early is worth as much as the testing. Here is who owns each part of an engagement like this.
| Activity | Who owns it | What everyone else does |
|---|---|---|
| Deciding what the system is allowed to do | The auditee | The tool inventory, the identity model and the approval gates are product and architecture decisions. A tester can tell you that a shared service account is reachable from a stranger’s document, but only you can decide that the assistant does not get a send-email tool. |
| Scoping the engagement | The auditee, in consultation with the testing organisation | §13.1.1(b) of the 2025 guidelines puts scope definition on the auditee in consultation with the auditing organisation. The nine-property scope should name what is excluded, which on an LLM engagement is as load-bearing as what is included. |
| Performing the testing | The testing team | The testing team works to the agreed scope and the rules of engagement set out above, and records every attempt so that a result can be reproduced. §13.2.2(ii) of the 2025 guidelines requires the test plan to state both calendar time and man-hours. |
| Signing the audit report | The empanelled auditing organisation | §9.5(iii) states that audit reports “shall be signed only by the manpower declared to CERT-In, as listed in the organization’s snapshot information available on the CERT-In website”, signed by the auditors who conducted the audit, reviewed and signed by a designated reviewer outside the audit team, and finally authorised and signed by the head of the auditing organisation. |
| Fixing what is found | The auditee | §9.1(ii) states that the responsibility for maintaining an efficient and robust cyber security posture ultimately rests with the auditee organisation, not the auditor, and §9.2 requires risk treatment and exceptions to be authorised and accepted by the head of the auditee organisation. On an LLM system, remediation runs through the teams that own the prompt, the tools and the corpus. |
| Verifying that the fix worked | The auditing organisation, in the follow-up audit | §16.3(iv) puts verification of closure with the auditor, and §15.2.1(iii) requires observations from the previous audit cycle to be revalidated, with anything still present recorded as a repeat observation. On LLM findings, verification has to be run against the recorded configuration or it verifies nothing. |
We deliver CERT-In audits
We deliver CERT-In audits, end to end, with our CERT-In empanelled partners. See how a CERT-In audit runs with us.
The primary sources, with the clauses relied on
Every assertion on this page traces to one of the documents below, each named with its issuing body, its version and its date. Every primary-source PDF was fetched and checked on content type and byte count rather than on status code, because cert-in.org.in answers 200 with a small HTML page for unknown paths. Three documents, named in the note below, are deliberately not linked, with the reason stated against each. Where a later version exists, it supersedes what is written here.
- 01 CERT-In, Comprehensive Cyber Security Audit Policy Guidelines, version 1.0, 25 July 2025. 69 pages, 20 numbered sections. The instrument most of this page is built on. Relied on at §5(xxiii) and §6(xvi) (Red Team Assessment, identical wording in both), §5(xxiv) and §16.2(ii) (CVSS, EPSS, CWE and CVE), §6 opening sentence and §6(xxiii) (AI System Audits, in an expressly open list), §8(i) (solely tools-based testing discouraged, with its reason), §8(ii) (limited lists are not audit standards), §9.1(ii) and §9.2 (the auditee owns posture, risk treatment and exceptions), §9.3(iv) and §13.1.1(g) (follow-up audits inside the scope), §9.5(ii) (version control so scoped assets can be backtracked — a clause with no modal, so it asks rather than requires), §9.5(iii) (who may sign), §13.1.1(b) (the auditee defines scope in consultation with the auditor), §13.1.2 (annual minimum, and the Major Change test defined by risk), §13.2.2(ii) (test plan in calendar time and man-hours), §13.2.3(ii)(b) (the as-of date of the inventory in the report), §15.2.1(iii) (revalidate the previous cycle, record repeats), §15.2.2(iii) (consent limits on social-engineering and process testing), §15.2.3(i) and §15.2.3(ii) (name the environment, obtain hash values and version numbers), §16.1(ii) (report contents, including sampling decisions), §16.3(ii) (criticals and highs as and when found), §16.3(iii) (closure timelines by severity, with no day counts fixed), §16.3(iv) (follow-up audits verify closure), §16.4 (high-risk vulnerabilities reported immediately), and §18(i) and §18(iv) (working notes, and evidence stored securely). cert-in.org.in
- 02 Directions under sub-section (6) of section 70B of the Information Technology Act, 2000, No. 20(3)/2022-CERT-In, 28 April 2022, including Annexure I. The six-hour reporting duty at direction (ii) and the twenty mandatorily reportable incident types at Annexure I — item (xx) for attacks on artificial intelligence and machine learning systems, item (xviii) for cloud computing systems, servers, software and applications. Direction (iv) is the 180-day rolling log obligation, maintained within Indian jurisdiction. Annexure I to this document is a bare list of the twenty types: the explanatory paragraph for item (xx) is in the May 2022 FAQ, at the next entry, and not here. cert-in.org.in
- 03 CERT-In, FAQ on the Cyber Security Directions of 28.04.2022, May 2022. Linked from CERT-In’s own Directions page. Three things are relied on here. Two are answers: that the obligation to report a cyber incident “is neither transferrable nor indemnified or dispense with”, and the qualification on log residency — the logs “may be stored outside India also as long as the obligation to produce logs to CERT-In is adhered to by the entities in a reasonable time”. The third is Annexure-I of the FAQ, headed “Explanation for Types of Cyber Security Incidents to be Reported to CERT-In”, which gives a paragraph per type. Its paragraph on item (xx) describes attacks that target machine learning models with the intention of causing a malfunction, employing techniques such as deceptive data and environment manipulation, with impact on confidentiality, integrity and availability. That is where the explanation lives — not in the Directions themselves — and it is an explanation of the type rather than a reportability test for any particular fact pattern. The FAQ states its own limitation, which should travel with any quotation from it: it is not a legal document and in no way replaces, amends or alters any part of the IT Act, 2000 or the CERT-In Rules, 2013. cert-in.org.in
- 04 CERT-In, Incident Reporting Form. The single-page form CERT-In itself points to. Worth reading before an incident rather than during one, for one reason above all: its own notes state that it is not mandatory to fill in or sign the form, and that incidents may be reported by providing the relevant information in the communication itself or in any other readable form. Nobody should spend part of a six-hour window looking for a template. cert-in.org.in
- 05 CERT-In, Empanelment of Information Security Auditing Organisations — Terms and Conditions for Empanelment, version 7.2, April 2024. Annexure I is headed “Expectations of an Auditee organisation from an Auditor”, and two of its items read directly on how a stochastic engagement should be written up: “Reports include all unknowns clearly marked as unknowns” and “Reports state clearly all states of security found and not only failed security measures.” They are expectations rather than requirements, and are cited here as such. Clause 3.2.2 is the five-day submission of the report and audit metadata to CERT-In, including critical vulnerabilities not fixed even during follow-up audits. cert-in.org.in
- 06 CERT-In, Guidelines for CERT-In Empanelled Information Security Auditing Organizations, version 3.0, May 2018. The 2018 ancestor of the tools rule, and the source of the maker-checker idea in its most concrete form: “Auditors should deploy a verification team (Red Team) to verify the work performed by their audit team (White Team).” Note the vocabulary collision — in that sentence “red team” means the people who check the testers, not the people who simulate an adversary. The 2025 guidelines carry the same idea at §15.2.5(iii) without the colour. cert-in.org.in
- 07 CERT-In, published list of empanelled information security auditing organisations, with snapshot information per organisation. The roster §9.5(iii) requires report signatories to come from. Cited on this page only for the signing rule stated in the roles table. cert-in.org.in
- 08 CERT-In, Policy Guidelines for Handling Audit related Data, version 1.0, June 2014. Required by name at §10.3(vii) of the 2025 guidelines, and the instrument behind this page’s treatment of transcripts: auditee data stored only on systems located in India, devices secured including by encryption, access on a need-to-know basis, retention only for the agreed period, and no sharing with an overseas partner unless the auditee specifically authorises it. It also states that an audit report should not be shared or hosted on online file services, sharing platforms or cloud services unless the service is under the auditing organisation’s own technical and administrative control or the auditee specifically authorises it. cert-in.org.in
- 09 CERT-In, Samvaad 2026 Conference Proceedings, 2nd Edition — and Press Information Bureau release 2256581, 29 April 2026. The two official accounts of CERT-In SAMVAAD 2026, held from 27 to 29 April 2026 at the BITS Pilani K. K. Birla Goa Campus. They are not interchangeable, and this page keeps them apart. The phrase “AI driven red teaming methodologies” is in the PIB release and is cited to the PIB release alone, alongside the release’s own list of AI-driven cyber risks including emerging frontier AI models with advanced cyber capabilities and the auditing of cloud systems, APIs and operational technology. The 666-page proceedings are cited only for what they themselves record: the AMBAK launch and CERT-In’s gloss of it, the 500-plus delegates, the panel and the two working groups. Either way it is context with a source, not a standard. Background on the platform launched at the same event is on our page about AMBAK. cert-in.org.in
- 10 OWASP Top 10 for LLM Applications, Version 2026, by the OWASP GenAI Security Project. The document’s own cover carries the date 4 August 2026, while the project’s resource listing dates the posting 3 August 2026. Cited on this page for exactly two item numbers: Prompt Injection is LLM01, and Excessive Agency is LLM03, having been LLM06 in the 2025 edition. Other categories are named here in ordinary words rather than by number, because numbers that move between editions are worse than no numbers. One navigational warning, because it is a live trap: the project’s /llm-top-10/ archive index still lists the 2025 edition, in which LLM03 is Supply Chain and Excessive Agency is LLM06. A reader sent there to check a 2026 item number will find the page contradicting the citation. The link below is the 2026 edition’s own resource page, which is where the 2026 document is served. Cited as vocabulary, not as an audit criterion, for the §8(ii) reason set out at entry 1. genai.owasp.org
- 11 OWASP Top 10 for Large Language Model Applications — OWASP project page, and the project’s own archive index. The project’s entry on owasp.org, recorded separately because the older /www-project-top-10-for-large-language-model-applications/ path redirects to it. Cite the destination. The archive index at genai.owasp.org/llm-top-10/ is the companion page, and at the time of checking it served the 2025 edition rather than the 2026 one. owasp.org
- 12 OWASP GenAI Security Project, GenAI Red Teaming Guide. The project’s own guidance on the practice, as distinct from its ranked list of risks — and therefore the OWASP document that is actually on point for a page about method. Cited here as a methodological reference rather than as an audit criterion, which is the distinction §8(ii) draws and the reason a guide survives that clause where a top-ten does not. genai.owasp.org
- 13 MITRE ATLAS — Adversarial Threat Landscape for AI Systems. MITRE’s knowledge base of adversary tactics and techniques against AI systems, structured as a matrix with stable identifiers in the ATT&CK form: tactics numbered AML.TAnnnn, techniques AML.Tnnnn. Its published dataset names LLM prompt injection among the techniques. Used on this page as a naming layer for adversary behaviour beside the CWE mapping — a knowledge base of observed behaviour is a different instrument from a ranked top-ten, which is why it does not attract the §8(ii) objection. Caution when citing a specific identifier: MITRE’s own data repository marks the dataset served at the historical path as no longer updated, so confirm the identifier against the current site before it goes in a report. atlas.mitre.org
- 14 NIST, AI 600-1 — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024. The international document that treats AI red-teaming as a named practice rather than as a subject heading: it directs organisations to conduct AI red-teaming and structured human-feedback exercises, to assess resilience against abuse, and — a detail worth borrowing — to document the instructions given to data annotators and AI red-teamers. Guidance, not a certifiable standard. It sits alongside §6(xxiii) rather than instead of it, and the comparison of the management-system and framework routes is on our ISO 42001 versus NIST AI RMF page. nvlpubs.nist.gov
- 15 The Digital Personal Data Protection Act, 2023 (Act 22 of 2023), as published in the Gazette of India. India’s own data-protection statute, and the instrument a customer will ask about when they learn that conversation transcripts left their building. Relied on here at section 2(i) (a Data Fiduciary determines the purpose and means of processing), section 2(k) (a Data Processor processes personal data on behalf of a Data Fiduciary), section 8(1) (the Fiduciary remains responsible for compliance in respect of processing undertaken on its behalf by a Processor), section 8(2) (a Processor may be engaged for activities related to offering goods or services to Data Principals “only under a valid contract”), and section 8(5) (reasonable security safeguards, extending to processing done on the Fiduciary’s behalf). The auditee is the Fiduciary, and the testing organisation holding its transcripts is the Processor. meity.gov.in
- 16 OWASP Web Security Testing Guide, and OWASP Application Security Verification Standard. Both are named at §8(ii) of the 2025 guidelines as comprehensive frameworks an audit should be built from — the WSTG for web application security testing, ASVS for establishing and verifying application security controls. They remain the backbone of the non-model half of an LLM application, which is most of it: the sinks, the authorisation model, the session handling and the API surface. Name the ASVS version as well as the level in any scope of work, because the requirement identifiers were reorganised between 4.0.3 and 5.0.0. owasp.org
- 17 FIRST, CVSS v4.0 Specification Document (document version 1.2), and the CVSS v3.1 Specification Document. Cited for the structure of the instrument rather than for a score: v4.0 is composed of four metric groups — Base, Threat, Environmental and Supplemental — with a single Threat metric, Exploit Maturity, and a new Attack Requirements metric in the Base group. The published nomenclature (CVSS-B, CVSS-BE, CVSS-BT, CVSS-BTE) should be used wherever a numerical value is displayed. And, counter-intuitively, Security Requirements default to Not Defined, which is equivalent to High, so supplying them can only lower a score. The qualitative severity bands commonly printed in Indian audit reports come from this specification, not from CERT-In. The version is readable from the vector prefix, and §16.2(ii) pins no version, so a report should say which it used. first.org
- 18 FIRST, EPSS home page and EPSS FAQ. EPSS estimates the probability that a published CVE will be exploited in the wild in the next 30 days, publishing a 0–1 probability with ranking percentiles every day. The FAQ carries the three points this page relies on: (1) EPSS is not a complete risk score and does not account for environment or compensating controls, (2) there is no universal threshold and EPSS defines no labelled bands, and (3) scoring something with no CVE identifier at all is described as a longer-term direction — which is why an LLM finding in bespoke code usually has no EPSS value and should say so. Any EPSS number printed in a report is a snapshot and needs the date it was pulled. first.org
- 19 MITRE, About CWE. A weakness is “a condition in a software, firmware, hardware, or service component that, under certain circumstances, could contribute to the introduction of vulnerabilities”. That definition is why the CWE limb of §16.2(ii) remains satisfiable for LLM findings even where the CVE limb is not: the weakness is in the surrounding system, and it is usually a recognisable one. cwe.mitre.org
Three documents are named and not linked. ISO/IEC 42001:2023, the AI management system standard, is cited here for what it is rather than for any clause, and iso.org answered 403 when the link was checked on 22 September 2026, so we name the standard and leave the reader to obtain it from ISO or a national standards body. Regulation (EU) 2024/1689, the EU AI Act, is cited for Article 14, human oversight for high-risk systems. Regulation (EU) 2026/1744, the Digital Omnibus on AI, in force 27 July 2026, is cited for deferring the stand-alone high-risk obligations in Annex III to 2 December 2027 and those in Annex I to 2 August 2028. EUR-Lex answered 202 with a zero-byte body for the ELI, CELEX and Official Journal PDF paths of both instruments on the same date, which is not a served document, so neither is rendered as a working link and neither date above was read from a served copy by us. Check the current position in the Official Journal before relying on a date. The two are separate instruments and this page keeps them apart: 2024/1689 is the AI Act and 2026/1744 is the Digital Omnibus that moved its dates. Clause locators throughout this page were checked against the source documents on 22 September 2026. If you find one that has moved, we would rather correct the page than be quoted accurately on something stale.
Frequently Asked Questions
What is red teaming an LLM application?
How do you red team an LLM application, step by step?
What is the difference between LLM red teaming and a penetration test or VAPT?
What is prompt injection, and what is indirect prompt injection?
How do you test for indirect prompt injection?
What is excessive agency in an LLM application?
How do you evidence an LLM finding when the model is non-deterministic?
How many times should you run an LLM red team test?
When is a probabilistic LLM finding actually closed?
What CWE does prompt injection map to?
How do you test the security of an MCP server or a model plugin?
What is denial of wallet in an LLM application?
What is RAG corpus poisoning, and how is it different from prompt injection?
Does LLM red teaming apply if our system has no chatbot?
What is a reproduction rate, and why does it replace a screenshot?
How do you assign a CVSS score to an LLM finding?
Does an LLM finding have a CVE and an EPSS score?
How often should an LLM application be red teamed?
Why did our LLM finding reopen when we changed nothing?
Is AI red teaming mandatory in India?
Is an attack on an AI system reportable to CERT-In?
Can we use the OWASP Top 10 for LLM Applications as our audit standard?
What access does an LLM red team need from us?
Can automated jailbreak tools replace a human red team?
What should an LLM red team report contain?
Do we need a CERT-In empanelled auditor for an LLM red team?
What drives the cost and duration of an LLM red team?
Can our own team red team our LLM application?
Where to go next. On the AI side, what AI red teaming is sets out the discipline and where it came from, AI red teaming versus VAPT is the one to read before writing a purchase order, and AI incidents under CERT-In covers what happens when a test finds somebody else’s work. On the infrastructure the application runs on, what cloud security is, cloud security audit in India and API security testing cover the surfaces every LLM stack also has. On the conventional testing that should run alongside this, what VAPT actually is, types of VAPT and VAPT vs penetration testing vs red teaming are the definitional set. And on the Indian audit machinery around all of it, what CERT-In is, a CERT-In auditor’s rules of engagement and AMBAK are the background. If you govern the AI programme rather than test it, AI risk assessment versus AI impact assessment and ISO 42001 versus the EU AI Act are the right next pages, and ISO 42001 versus the NIST AI RMF takes the framework side of it, including where NIST AI 600-1 sits.
Related Reading
What AI Red Teaming Is
Adversarial testing of AI systems: what it targets, how it differs from a penetration test, and what it produces.
Read moreAI Red Teaming vs VAPT
Why a penetration test of the application around a model tells you almost nothing about the model.
Read moreAI Incidents Are Reportable to CERT-In
Annexure I item (xx) makes attacks on AI and ML systems mandatorily reportable — within six hours.
Read moreAPI Security Testing
The surface behind every mobile app and single-page frontend, and why testing the UI misses it.
Read moreScoping a VAPT
Assets, environments, credentials, exclusions and the rules of engagement to agree before testing starts.
Read moreWhat VAPT Actually Is
Vulnerability assessment and penetration testing are two activities. Why India bundles them, and what a real one covers.
Read more