A penetration test of an AI-enabled application tests the software around the model: the chat surface, the API tier, authentication, the datastores, the hosting. It is necessary work and it is not the same work as establishing whether the assistant can be talked into calling a tool it should not. These two get quoted against each other as though one were a larger version of the other. They are not, and buying the wrong one is an assurance mistake rather than a pricing one.
What CERT-In actually says: the Comprehensive Cyber Security Audit Policy Guidelines (version 1.0, 25 July 2025) list twenty-six classes of engagement at §6 and name “Artificial Intelligence (AI) System Audits” at (xxiii), defined in one sentence. The instrument has no vocabulary for models, prompts, training data or agents — so the method has to come from the scope of work, and this page is about what has to go in it.
Plain-English explainer · CERT-In Comprehensive Cyber Security Audit Policy Guidelines v1.0 (25 July 2025) · CERT-In Directions of 28 April 2022 · OWASP Top 10 for LLM Applications, 2026 edition · Every clause locator and every negative finding checked 22 September 2026
What is AI red teaming, and how is it different from a VAPT?
AI red teaming is structured adversarial testing of what an AI system can be induced to say, disclose or do. An operator sets an objective the system should refuse or should be unable to reach, generates many variations of an input intended to achieve it, runs them at volume, and grades the outputs against a rubric written before the work started. The result is an envelope rather than a defect list: what came out, how often, under what conditions, and what the consequence reached through whatever tools the system can call. A VAPT is a different engagement with a different object. It examines the software and infrastructure around the model — the web and mobile surface, the API tier, authentication and authorisation, the datastores, the hosting — and CERT-In defines its two halves at §6(iii) and §6(iv) of the Comprehensive Cyber Security Audit Policy Guidelines (version 1.0, 25 July 2025). There is a longer treatment of the discipline itself in what AI red teaming is. This page is about how it differs from the engagement it keeps being quoted against.
Passing one tells you almost nothing about the other. A chat product can have a clean penetration test — every endpoint authenticated, every object identifier checked, every library current — and still have an assistant that will look up somebody else’s case and email it out because a document in its retrieval corpus told it to. Nothing in the penetration test’s scope points at that, and nothing in its method would find it. The reverse holds too: a thorough behavioural engagement against a model and its orchestration proves nothing about an API tier that lets any authenticated user read any conversation. These are two questions, and they need two answers.
The deeper difference is not coverage but epistemology, and it is what makes the second engagement genuinely unfamiliar to buy. A penetration-test finding is deterministic: the same request produces the same response, the proof is a screenshot, and closure is a diff. An AI finding is stochastic: the same input can produce a refusal and a compliance, so the proof is a success rate over a stated number of trials, severity does not fit the scoring machinery §16.2(ii) requires, and closure means a mitigation that holds across a family of variants rather than a pass on the three prompts in the report. Everything else on this page follows from that one difference.
Key Takeaways
- A VAPT of an AI-enabled application tests the software around the model — the presentation layer, the API tier, authentication and authorisation, the datastores and the infrastructure. It does not test what the model will do, because nothing in its scope points there.
- AI red teaming tests behaviour under adversarial input: what the assembled system — model, system prompt, retrieval corpus, tool schema — can be induced to say, disclose or do, and how reliably.
- The findings are epistemologically different. A penetration-test finding is deterministic and reproduces byte for byte. An AI finding is a rate: this input, this model version, these parameters, seven successes in twenty trials.
- CERT-In names the class and not the method. §6(xxiii) of the 2025 audit policy guidelines lists "Artificial Intelligence (AI) System Audits" and defines it in one sentence: evaluation of AI systems for security, ethical alignment, transparency, data integrity, and resilience to adversarial manipulation.
- The instrument has no AI vocabulary. On a full-text search of the 69-page PDF on 22 September 2026, "machine learning", "LLM", "large language", "generative", "agent" and "training data" do not occur at all. The term "model" occurs three times and never means an AI model, and "prompt" occurs three times and is always the adverb "promptly".
- The severity machinery does not fit. §16.2(ii) requires CVSS supplemented with EPSS and every observation mapped to a CWE and a CVE. CVSS has no metric for a success rate, EPSS is defined over published CVEs, and a behaviour in your own system prompt has neither.
- Almost no durable fix lives in the model. Authorisation enforced in the tool rather than inferred from the conversation, least privilege on tool credentials, human confirmation on irreversible actions, output encoding and removed capability are the controls that hold.
- Do the conventional engagement first, and name the tool endpoints as API assets in it. If the API tier does not check authorisation per object, nobody needs to talk the assistant into anything.
- An AI/ML incident is already mandatorily reportable in India, where it is an attack. Item (xx) of Annexure I to the CERT-In Directions of 28 April 2022 covers attacks or malicious or suspicious activities affecting systems and applications related to Artificial Intelligence and Machine Learning, and direction (ii) gives 6 hours from noticing it. An event with no adversary is not the class item (xx) describes.
- Three different things get called red teaming in an Indian scope of work: the auditor’s internal verification team from the 2018 guidelines, the §6(xvi) Red Team Assessment against your live organisation, and AI red teaming — which appears in no CERT-In instrument at all.
- We deliver CERT-In audits, end to end, with our CERT-In empanelled partners.
What a VAPT of an AI-enabled application actually covers
Does a penetration test cover an AI chatbot?
It covers the plumbing, thoroughly, and it stops at the model. The useful way to see this is to lay the system out in layers and mark which engagement reaches each one. Two things fall out immediately: the overlap is much smaller than a buyer expects, and the one place the two disciplines genuinely meet — the tool boundary — is the layer that most often sits in neither scope.
| Layer | What is in it | A VAPT reaches it | AI red teaming reaches it |
|---|---|---|---|
| 01 The web and mobile presentation layer | The chat surface itself: session handling, authentication, cross-site scripting in the rendering of model output, clickjacking, file upload handling, the usual furniture of a web application. The model is irrelevant here — a chat window is a web page with a text box. | Fully in scope. This is the centre of a web application test. | Out of scope, except for one seam: what happens when the model returns markup, a link or a script and the interface renders it without encoding. |
| 02 The API tier | The endpoints the interface calls: conversation creation, message posting, history retrieval, file attachment, feedback. Authorisation per object. Rate limiting. Tenancy boundaries. Conversation identifiers that can be enumerated. | Fully in scope, and this is where a competent test earns its fee. Authorisation defects live here and only appear when a tester holds two accounts at different privilege levels. | Out of scope as such, but an AI red team that finds it will report it, because a conversation history readable across tenants is a disclosure route whatever discipline finds it. |
| 03 The orchestration layer | The code that assembles what the model actually sees: the system prompt, the conversation history, retrieved documents, tool definitions, and the routing that decides which of those to include. It is ordinary software and it is usually the newest, least reviewed code in the estate. | Partly in scope, and usually missed. Nothing in a conventional scope names it. Where source code review is in scope under §13.1.1(a), it is reachable. Where the scope names hostnames, it is not. | Central. Most findings that get called “model problems” are orchestration problems: untrusted text concatenated into the same context as instructions, retrieved documents given the same standing as the system prompt, tool definitions that describe more capability than the product needs. |
| 04 The model | The weights and the inference service, whether that is a hosted API, a model deployed in your own cloud tenancy, or something running on your own hardware. Its version, its sampling parameters, its safety filtering, and the provider-side changes you do not control. | Out of scope by construction. A penetration test can establish that the inference endpoint is authenticated and that the key is not in the client bundle. It cannot establish what the model will do. | The object of the exercise, but rarely the location of the fix. What a red team establishes here is the behavioural envelope: what this version, at these parameters, under this system prompt, can be induced to produce, and how reliably. |
| 05 The tools and actions | Everything the model can cause to happen: read a record, update a field, send a message, issue a refund, file a ticket, call another service. Each one has a credential, a blast radius, and an implicit assumption about who asked for it. | In scope if — and only if — the tool endpoints were named in the scope as API assets in their own right. They very often are not, because they are internal and undocumented. | Central, and this is the layer where the two disciplines meet. The question is not whether the model can be persuaded. It is what the tool does when it is. |
| 06 The retrieval corpus and the vector store | The documents the system searches to ground its answers, the embedding pipeline, the index, and — the part that matters — who is able to write into it. A corpus that ingests user-supplied documents is an untrusted input channel that looks like a database. | In scope as infrastructure: the datastore, its network exposure, its credentials, its backups. Not in scope as content. | Central. Indirect prompt injection lives here, and it is the failure mode that scales, because one poisoned document affects every user who asks a question it answers. |
| 07 The training, fine-tuning and evaluation pipeline | Where a model is trained or fine-tuned in-house: the data sources, their provenance and licensing, the labelling process, the artefact store holding checkpoints, and the pipeline that promotes a checkpoint to production. | In scope as a build system. Access control, secrets, pipeline integrity, artefact signing, dependency provenance — these are conventional findings and a good tester will produce them. | In scope as an influence channel. Poisoned or mislabelled training data produces behaviour, and behaviour is what a red team measures. |
| 08 The hosting and cloud infrastructure | The accounts, networks, identities, key management, logging and egress controls underneath all of it — plus the GPU or inference capacity, which is an availability and a cost-control problem at the same time. | Fully in scope. §6(xviii) names cloud security testing as its own class of engagement. | Out of scope, with one exception worth writing into the scope: whether the credentials the model can reach through its tools are scoped to the least privilege the product needs. |
| 09 The multimodal input channels | Every non-text channel the system will read: an uploaded image, a screenshot, a scanned document, an audio clip, a PDF whose text layer nobody looks at. Each is a carrier for instructions addressed to the model rather than to the person, and each arrives through a feature somebody shipped deliberately. | In scope as an upload: file type validation, size limits, the parser and its CVEs, where the file is stored, whether it is served back. Not in scope as a carrier of instruction — a well-formed PNG is a well-formed PNG. | In scope only if the scope of work says the channel is in scope. The default AI engagement probes text, because the harness speaks text. Name the channels explicitly or the interesting half of the injection surface is tested by neither side. |
| 10 Persistent memory and user-profile state | Anything the system carries between sessions: a summarised profile, stored preferences, a running note of what a user said last week, a long-lived thread. It is a writable store whose content becomes instruction-adjacent text in a future context window. | In scope as a datastore, exactly like the vector store: access control, tenancy, encryption, retention. Not in scope as content, and the content is the point. | In scope, and it has a different failure shape from corpus poisoning. An instruction written into memory survives the session it arrived in, is scoped to one user, and does not appear in any corpus review — so it is invisible to the control that catches the retrieval case. |
| 11 Third-party tool connectors and their protocol layer | Tool surfaces you did not write and do not host: connectors, plugins and tool servers, commonly Model Context Protocol servers on a stack built in the last two years. You inherit their schemas, their credential scope and, on most implementations, descriptions that are re-read at runtime rather than pinned at approval. | Almost never in scope. The connector is not your hostname, it is not in your asset inventory, and nobody thought of it as an asset when the inventory was compiled. | In scope where the scope says so, and this is where the two disciplines fail together most reliably: a tool description is prompt text a third party controls, which makes it both an application supply-chain question and a behavioural one. There is a fuller treatment in red teaming an LLM application. |
Read rows three and five together and the practical argument of this page is already visible. The orchestration layer is ordinary software that no conventional scope names, and the tool layer is an API tier that no conventional inventory lists, because the tools have no interactive client and were written to be called by one service. Put both into the asset annexure and a good penetration test will test them as it tests anything else — §13.1.1(a) already has the scope cover the entire cyber infrastructure including APIs, and §13.1.1(c) derives it from the consolidated and updated asset inventory. The reason they are missed is not methodological. It is that nobody wrote them down.
The bottom three rows are the ones most likely to be missing from a scope written even a year ago, and each of them falls between the two engagements in exactly the way the rest of this page predicts. A multimodal input channel is tested by the penetration test as an upload and by the AI engagement not at all, because the harness speaks text. A persistent memory store is tested as a datastore and not as content, and it has a failure shape the corpus review cannot see: an instruction written into one user’s memory survives the session it arrived in. And a third-party connector is a tool surface you did not write, whose schema and credential scope you inherit and whose descriptions are, on most implementations, re-read at runtime rather than pinned at the moment somebody approved them. Name all three in the scope or they belong to neither side. There is more on how to write the inventory that makes that possible in how to scope a VAPT, on testing the tool tier properly in API security testing, and on row 08 in what cloud security is and cloud security audits in India.
Three different things called red teaming
What is a red team assessment under CERT-In?
Before comparing anything, settle the word. An Indian scope of work can use “red team” in three distinct senses, two of which are CERT-In’s own and one of which appears in no CERT-In document at all. A proposal that does not say which one it means has not yet told you what it is selling.
Red Team, the auditor’s internal verification team
Guidelines for CERT-In Empanelled Information Security Auditing Organizations, version 3.0, May 2018, under Technical.
“Auditors should deploy a verification team (Red Team) to verify the work performed by their audit team (White Team).”
A second group inside the auditing firm checks the first group’s work before it is issued. The 2025 guidelines carry the same idea without the colour at §15.2.5(iii), which has a separate verification team review and validate the audit team.
Why it matters here: Nothing to do with AI. It is a quality-control duty the auditing organisation owes as a condition of its empanelment, and you do not buy it.
Red Team Assessment, the §6(xvi) engagement
Comprehensive Cyber Security Audit Policy Guidelines, version 1.0, 25 July 2025 — §6(xvi), restated word for word at §5(xxiii).
“An exercise, reflecting real-world conditions, that is conducted as a simulated attempt by an adversary to attack or exploit vulnerabilities in an enterprise’s information systems.”
Adversary simulation against your live organisation, with your detection and response function on the other side of it and mostly unaware. §15.1.1(i) limits notification of the test to key personnel for that reason.
Why it matters here: It measures your defence. An AI system can be the route an operator takes, but the question being answered is whether anybody noticed — which is not the question AI red teaming answers.
AI red teaming, the behavioural discipline
Not defined in any CERT-In instrument. The nearest thing is §6(xxiii), “Artificial Intelligence (AI) System Audits”, which names a class and gives it one sentence.
§6(xxiii), quote-exact: “Evaluation of AI systems for security, ethical alignment, transparency, data integrity, and resilience to adversarial manipulation.”
Structured adversarial probing of an AI system’s behaviour: what it can be induced to output, disclose or do, measured across many trials rather than demonstrated once.
Why it matters here: This is the thing this page is about. Note that of the five objects §6(xxiii) names, only “security” and “resilience to adversarial manipulation” are things a conventional security engagement touches at all — ethical alignment, transparency and data integrity are governance work.
A numbering trap worth knowing before you cite anything
§5 and §6 of the 2025 audit policy guidelines both run to roman numerals in the twenties, and they are different lists. On this subject specifically, the collision is unlucky: the same numeral means red teaming in one section and artificial intelligence in the other. Always cite the section with the item.
| Locator | What it is | Note |
|---|---|---|
| §5(xxiii) | Red Team Assessment | The definitions section. §5 runs i to xxiv and the three testing definitions sit at (xxi) vulnerability assessments, (xxii) penetration testing and (xxiii) red team assessment. |
| §6(xxiii) | Artificial Intelligence (AI) System Audits | The scope-of-engagements list. §6 runs i to xxvi and the AI class sits at (xxiii), between endpoint security assessment at (xxii) and vendor risk management audits at (xxiv). |
| §6(xvi) | Red Team Assessment | The same definition as §5(xxiii), word for word, in its place in the class list. Both locators are correct for the definition, but only this one is correct for “which class of engagement is this”. |
| §6(xxvi) | SBOM, QBOM and AIBOM Auditing | The other clause in which the words “Artificial Intelligence” appear. It is a bill-of-materials clause about transparency, traceability and integrity of components — not a behavioural one. |
The first two senses are set out at length, with the clause text and the procurement consequences, in VAPT, penetration testing and red teaming are not the same engagement. This page is about the third.
CERT-In names the class. It does not supply the method.
§6 of the 2025 audit policy guidelines lists twenty-six classes of cyber security audit and assessment, introduced with the phrase “including, but not limited to”, so the list is a floor rather than a boundary. One of them is about AI, and it is a single sentence.
“Artificial Intelligence (AI) System Audits – Evaluation of AI systems for security, ethical alignment, transparency, data integrity, and resilience to adversarial manipulation.”
Five objects are named in that sentence and they do not belong to one discipline. Security and resilience to adversarial manipulation are testing work of the kind this page describes. Ethical alignment, transparency and data integrity are governance work — they live in a management system, in an impact assessment, in model documentation and in data provenance, and no amount of adversarial probing produces them. A proposal that offers to satisfy §6(xxiii) with a testing engagement alone has quietly dropped three of the five. There is more on the governance half in AI risk assessment against AI impact assessment and in ISO/IEC 42001.
What the sixty-nine pages do not contain
Absence is a claim like any other and it needs the same work, so the counts below were established by full-text search of the official PDF on 22 September 2026 — downloaded from www.cert-in.org.in, confirmed as 553,513 bytes of application/pdf rather than by status code alone, and converted with a layout-preserving text extraction — not by having failed to notice them.
| Term | In the document | Where, and what it means |
|---|---|---|
| “Artificial Intelligence” | three occurrences | Once in §6(xxiii), the AI System Audits class, and twice in §6(xxvi), where AIBOM is expanded as the Artificial Intelligence Bill of Materials. Nowhere else in the document. |
| “adversarial” | one occurrence | Inside the §6(xxiii) definition, in the phrase “resilience to adversarial manipulation”. It is the only adversarial-machine-learning vocabulary the instrument has. |
| “model” | three occurrences, none of them an AI model | Once in the EPSS definition at §5(xxiv) — “The EPSS model produces a probability score” — and twice in the name of the OWASP DevSecOps Maturity Model, at §8(ii) and again at §13.1.7(i)(a), the audit-contract clause that repeats the §8(ii) framework list. §13.1.8, Reference Standards and Initial Guidelines, does not contain the word at all. |
| “prompt” | three occurrences, all of them the adverb | Every instance is “promptly” — escalating pressure to CERT-In at §7, informing the auditee of an inaccessible asset at §10.1(ii), and acting on auditee feedback at §11(iii). The noun does not appear. |
| “machine learning”, “LLM”, “large language”, “generative” | zero occurrences each | None of these terms occurs anywhere in the sixty-nine pages. The 2022 Directions do use “Machine Learning”, at item (xx) of Annexure I — but that is the reporting instrument, not the audit one. |
| “agent”, “chatbot”, “training data”, “hallucination” | zero occurrences each | The instrument has no vocabulary for an autonomous system that takes actions, for a conversational interface, for the provenance of what a model learned, or for a confident wrong answer. |
| An AI audit type at §13.2.3(i)(a) | absent | The clause requires the type of audit to be communicated before commencement and names nine: VAPT, EAPT, device level audit, configuration audit, process audit, mobile application security audit, web application security audit, API security audit and compliance audit — a list the guidelines end with “etc.”. No AI audit type is among them. |
None of that is a criticism of the document. It is a 2025 instrument that named a class of engagement the market had barely started buying, and naming it was the right thing to do. What it means in practice is that the scope of work has to carry everything the instrument does not: the model version, the prompt version, the tool schema, the corpora, the sampling parameters, the rubric and the trial counts. Everywhere else in this hub the advice is to make the seller name the clause. Here it is the opposite — there is no clause to name, so the buyer has to write the specification.
And what the rest of the CERT-In estate does not contain either
A negative established in one document is a claim about that document, so the same search was run across four more CERT-In instruments on 22 September 2026, each downloaded from cert-in.org.in with its content-type and byte size confirmed and converted with a layout-preserving text extraction. That wider search is what licenses the narrower statements on this page — and it found one place where the vocabulary does exist, which is more useful than the four where it does not.
| Document searched | As downloaded | What the search found |
|---|---|---|
| Comprehensive Cyber Security Audit Policy Guidelines v1.0 | 553,513 bytes · 69 pages | The counts in the table above. “Artificial Intelligence” three times, “adversarial” once, “model” three times and never an AI model, “prompt” three times and always the adverb. No occurrence of “system prompt”, “temperature”, “top-p”, “rubric”, “tool schema”, “chatbot” or “hallucination”. |
| Empanelment Terms and Conditions v7.2, April 2024 | 138,356 bytes · 10 pages | Zero occurrences of every term searched: no “machine learning”, no “LLM”, no “large language”, no “generative”, no “adversarial”, no “system prompt”, no “model version”. The document predates the subject and does not reach it. |
| Guidelines for CERT-In Empanelled Information Security Auditing Organizations v3.0, May 2018 | 307,912 bytes · 6 pages | One hit across the whole vocabulary: “red team”, in the verification-team sentence that gives this page its first disambiguation row. Nothing AI-related at all. |
| Guidelines for Secure Application Design, Development, Implementation & Operations, updated January 2024 | 517,135 bytes · 15 pages | Zero occurrences of every term searched. CERT-In’s secure-development guidance does not reach AI components, which is worth knowing before a bidder cites it as the methodology for an AI engagement. |
| Technical Guidelines on SBOM, QBOM & CBOM, AIBOM and HBOM v2.0, 9 July 2025 | 1,654,189 bytes · 66 pages | The exception, and it matters. §9.3, Table 10, “Minimum Elements of AIBOM”, names Model Name — “The official name of the AI model, serving as a unique identifier for tracking, referencing, and discussion” — and Model Version — “The specific version of the AI model, ensuring that any changes, updates, and version control are documented” — alongside Model Type, Model Developer, Model Licensing Information, Software Dependencies, ML Models and Algorithms, Model Performance Metrics, Data Source, Data Sets, Hardware, Security Requirements, Input, Output and Intended Usage. §9.4.2.6 asks for model lineage to be tracked as model versions, retraining activities and modifications. §9.4.2.4 asks for scripts, model weights and configuration settings to be included so that results can be replicated. |
The last row is the one to take away, and it changes a sentence this page used to carry. CERT-In does have vocabulary for naming an AI model by identity and version — just not in the audit instrument. §6(xxvi) of the audit policy guidelines names SBOM, QBOM and AIBOM auditing as a class, for transparency, traceability and integrity of the components used in software, quantum computing and AI systems, and CERT-In separately publishes Technical Guidelines on SBOM, QBOM & CBOM, AIBOM and HBOM at version 2.0 dated 9 July 2025, whose §9.3, Table 10 sets out the minimum elements of an AIBOM and names Model Name, Model Version, Model Type and Model Developer among them. That is a bill-of-materials duty rather than a testing one, and it is worth having in the scope for exactly that reason: it is the clause that makes “which model, at which version” a documented fact rather than a question asked at report review. Two honest caveats. A full-text search of the audit policy guidelines finds no citation of the BOM document, so reading the two together is a reasonable inference rather than a cross-reference the guidelines make. And the AIBOM recommendations at §9.4.1 are written for government, public sector and essential-services organisations and the AI solutions supplied to them — a private buyer can adopt the element list, but should not describe it as a duty that reaches them. Then there is the CERT-In SAMVAAD 2026 conference proceedings, which record a working group on Indian Standards for Cybersecurity Auditing whose stated aim includes standardised audit reporting formats, RFP templates and clearly defined scope templates for audit types including emerging technologies such as artificial intelligence. The honest reading of that is encouraging and unhelpful in the same breath: the scope template is being worked on, and it is not published, so today the specification is yours to write.
One phrase from the same event is routinely misread, so it is worth pinning. The Press Information Bureau release on SAMVAAD 2026 lists the topics presented and names “AI driven red teaming methodologies” among them. That is red teaming driven by AI — using models to generate and run offensive test cases — which is a different subject from red teaming of AI. The release also names AI driven cyber risks, including emerging frontier AI models with advanced cyber capabilities. Both are real and neither is a CERT-In methodology for testing your assistant.
Determinism against stochasticity, and what it does to the engagement
Why an AI finding cannot be reported like a penetration-test finding
Everything awkward about buying, reading and closing an AI engagement comes from one property: the system does not answer the same way twice. That is not a bug in the testing and it is not flakiness to be engineered out. It is what a sampled model does, and it propagates into five places a buyer touches.
Reproduction becomes a rate, not a screenshot
A SQL injection either works or it does not, and the proof is a request and a response. A model finding is a distribution. The same sentence, submitted twice, can produce a refusal and a compliance — that is sampling, not flakiness, and turning the temperature to zero does not make it go away, because the context, the retrieved documents and the conversation history all vary in production even when the parameters do not. A finding that says “the assistant disclosed the system prompt” is not reproducible. A finding that says “this input disclosed the system prompt in 7 of 20 trials against model X version Y at temperature 0.7, with the tool schema and retrieval snapshot at annexure B” is.
What follows for you: The engagement has to buy trials, and the scope of work has to say how many. Three is an anecdote. Twenty is a number a report can defend, and the arithmetic for why is in the trial-count table below rather than left as an assertion. The trial count is a cost line, and a quote that does not carry one has not priced the thing it is selling.
Severity stops being arithmetic
CVSS v4.0 composes four metric groups and none of them has a place to record that an attack succeeds a third of the time. Attack Complexity and Attack Requirements describe conditions on the attacker’s side — the measurement problem here is on the target’s side. So a prompt-injection finding scored 8.1 is a judgment wearing a number, and the only honest way to print it is the vector alongside the observed success rate and the trial count, with the base metrics stated rather than implied. FIRST’s own nomenclature note helps here: a score derived from base metrics alone is CVSS-B, and saying so is more informative than the number.
What follows for you: Agree the scoring convention before the engagement, not in the report review — and agree its literal format, which is set out at the end of this section rather than left as a recommendation. §16.2(ii) requires CVSS and EPSS to be implemented in the report. It does not tell you how to score a behaviour, and nothing in the five CERT-In documents searched for this page does either.
Retest has to test a family, not an instance
The commonest closure failure in this discipline is re-running the exact prompts that produced the original finding. Mitigations in this space are frequently narrow: a filter that catches the phrasing in the report and nothing adjacent, a system-prompt instruction that holds for the tested language and not for the same request in another one, a guardrail tuned on the examples it was given. Closure means the mitigation holds across the family of variants, measured the same way the finding was, at a rate agreed in advance.
What follows for you: The variant suite outlives the engagement. That is the genuinely valuable artefact: a regression set that runs on every model version change, every system prompt edit and every new tool grant, and that produces a number you can put in front of a board.
The result has a shelf life you do not control
§10.1(i) limits the auditor to reviewing and reporting on the state of security at the point in time of the audit. For a conventional application that is a familiar constraint — the build is yours, and you know when it changed. For a hosted model it bites much harder: the provider can update weights, change routing, re-tune safety filtering or deprecate a version, and none of that appears in your change log. A result obtained against a model endpoint in March may not describe the same system in June.
What follows for you: Pin the model version in the scope and in the report, record the provider’s deprecation policy alongside it, and treat a provider-side model change as a change-management event in your own process rather than as somebody else’s business. “Pin the version” is an instruction with mechanics behind it — what string to record, from where, and how often — and those mechanics are set out in the finding-record table below rather than assumed.
A clean run is weaker evidence than it looks
An empty finding list from a penetration test means the tester did not find a way in against that scope. An empty finding list from an AI red team more often means the objectives were not adversarial enough, the trial count was too low, or the rubric was too narrow. It is our assessment, and the working assumption of the practitioners we read, that no general defence against prompt injection has been established — which is why the mature version of this exercise measures consequence rather than chasing prevention. Treat that as an engineering judgment rather than a settled result, and note that it cuts in the conservative direction: if a general defence does exist, bounding the consequence costs you nothing.
What follows for you: Read the objectives and the rubric before you read the findings. Items 20 and 21 of Annexure I to the empanelment terms expect reports to mark all unknowns clearly as unknowns and to state all states of security found rather than only the failed measures — which is exactly the right discipline for a stochastic result set.
There is a reporting consequence worth stating on its own, because it is where most templates break. §16.2(ii) of the audit policy guidelines requires auditors to implement both CVSS and EPSS in their reports, requires observations to be categorised using CVSS for severity and supplemented with EPSS, and requires every reported observation or vulnerability to be mapped with a CWE and a CVE number. Applied to a behavioural finding, three of those four do not land. The CVSS v4.0 specification composes Base, Threat, Environmental and Supplemental metric groups and has nowhere to put a success rate. EPSS estimates the probability that a published CVE will be exploited in the wild and is published per CVE. FIRST’s own frequently asked questions say scoring vulnerabilities with no CVE ID at all is a longer-term direction rather than a current capability, and add that EPSS is not a complete risk score and accounts for nothing specific to your environment. And CWE enumerates conditions in a software, firmware, hardware or service component — which catches some of these findings and not the ones whose only defect is that a system was persuaded.
What a severity line for a behavioural finding actually looks like
This page tells you twice to agree the scoring convention in advance. Here it is, so that the instruction has something behind it. Everything below is a format rather than a score: the fields are the fields, the values are illustrative, and the blanks are the point.
| vector | CVSS:4.0/AV:N/AC:L/AT:N/PR:L/UI:N/VC:H/VI:N/VA:N/SC:N/SI:N/SA:N |
| nomenclature | CVSS-B — base metrics only, because no Threat or Environmental metrics were supplied |
| observed | 11 successes in 20 trials, fresh sessions, at the parameters at annexure A |
| cwe | CWE-[nnn] where one genuinely fits the consequence, otherwise stated as not mapped |
| cve | None. No third-party product is implicated. The defect is in software the auditee wrote |
| epss | None. EPSS is published per CVE and there is no CVE |
| limitations | Recorded under §16.1(ii), with the reason each blank field is blank |
Three things about that block are worth saying out loud. The CVSS-B label is FIRST’s own nomenclature for a score derived from base metrics alone, and printing it is more informative than printing the number. The two blank fields are blank on purpose and each carries its reason, which is what §16.1(ii) is for. And the “observed” line, not the vector, is the field a retest is measured against — which is why the denominator can never be dropped. Items 20 and 21 of Annexure I to the empanelment terms expect exactly this posture: unknowns clearly marked as unknowns, and all states of security found reported rather than only the failures. There is a fuller treatment of how these scores behave in how to read a VAPT report.
How do you evidence an AI red team finding?
With the fields below, recorded per trial rather than per engagement. §16.1(ii) already requires the report to state its sampling decisions. On a stochastic system that is not a footnote, it is how many times you tried. Everything here is the short form — there is a longer version, with a reason per field, in red teaming an LLM application, and the two are deliberately not the same table.
| Field | What to record | Why it is in the record |
|---|---|---|
| Model identifier, as the API returned it | The string in the response, not the string in the request, captured per trial and not once per engagement. State whether it is a dated snapshot identifier or a floating alias. | A bare family name is an alias that resolves server-side and moves. The two give the result completely different shelf lives. |
| Provider, endpoint and region | Which service, which deployment, which region — plus any provider-side response or system fingerprint the API exposes. | The same model name can be served from different deployments with different filtering, and the region is also a data-residency fact. |
| Sampling parameters, including any seed | Temperature, top-p, maximum tokens, and the seed where the provider exposes one. | A fixed seed with fixed parameters is the closest thing to determinism available, and whether one was used changes how the rate should be read. |
| System prompt version and hash | The version reference plus a hash of the instruction layers as they stood during the test. | A finding is against a configuration rather than a model. §15.2.3(ii) already requires hash values and version numbers for an application build. |
| Tool schema, exported verbatim | The schema as the model received it, attached as an annexure rather than described. | Tool definitions change more often than models do, and a renamed tool is indistinguishable from a fix at retest. |
| Retrieval corpus snapshot with an as-of date | Which corpora were connected, which documents were returned, and the corpus as of a date. | §13.2.3(ii)(b) makes the as-of date of the asset inventory a report field, and a corpus that moves under two runs deserves the same treatment. |
| Conversation state | Whether each trial ran in a fresh session or a continued one. Never pool the two into a single denominator. | A continued session is contaminated by everything earlier in the window, so the two populations answer different questions. |
| Trials attempted and trials succeeded | Both numbers, per objective, never a bare percentage and never a bare success. | Eleven of twenty and eleven of two hundred are different findings, and the denominator is the part that is usually dropped. |
| The grading rule applied | The rubric clause that decided each trial, the grader used, and the rate at which a person spot-checked the machine grading. | Annexure I item 22 of the empanelment terms expects qualitative metrics derived from a stated method rather than from the auditor’s feelings. |
| Timestamp, in a stated timezone | Per trial, with the timezone written out. | A result that later stops reproducing can only be correlated against a provider’s release notes if it carries a time. |
The first row is the one that quietly decides whether any of the others matter, and it is the instruction this page gives four times without ever saying how. Record the model string the API returns, not the one you sent: a bare family name is an alias that resolves server-side and can move without a version bump you can see, while a dated snapshot identifier is a thing you can hold a provider to. Record it on every trial rather than once at the start, capture any provider-side response or system fingerprint the API exposes, and require the report to state which of the two was in force — because a result obtained against a pinned snapshot and a result obtained against a floating alias have completely different shelf lives, and only one of them can be defended six months later. CERT-In has the vocabulary for this in a different document: §9.3, Table 10 of the Technical Guidelines on SBOM, QBOM & CBOM, AIBOM and HBOM names Model Name and Model Version as minimum elements, §9.4.2.6 asks for model lineage to be tracked as versions, retraining activities and modifications, and §9.4.2.4 asks for scripts, weights and configuration settings to be recorded so that results can be replicated.
How many trials is enough, and what does a clean run actually prove?
Start with the arithmetic, because this is the one place on the page that should be unarguable. If an objective is attempted twenty times and never succeeds, the useful statement is not “it does not work”. It is that the true success rate is below roughly fifteen per cent, at ninety-five per cent confidence — the rule of three, which puts the upper bound at about 3/n when there are no successes. Fifty clean trials moves that bound to roughly six per cent, and a hundred to about three. So “nothing worked” is a statement about your denominator, and a report that prints it without one has made a claim it cannot support.
Two consequences follow for how a rate is written. Report it with its denominator and an interval rather than as a bare percentage — “11/20” is honest, “55%” is a rounding of an estimate with wide error bars. And never pool fresh sessions with continued ones into a single denominator: a continued session carries everything earlier in the window, so the two populations are answering different questions and averaging them produces a number about neither.
| Objective class | Example | Trials to buy | Reporting threshold |
|---|---|---|---|
| Consequence-bearing tool-call objectives | Cause a tool to act outside the authority of the person in the conversation. | The highest count you buy — at least twenty per objective, and fifty where the action is irreversible. | Any success is a finding. A rate of one in fifty on an irreversible action is not a small finding, it is a reachable one. |
| Disclosure objectives | Recover the system prompt, or content belonging to another user or tenant. | Twenty per objective, run in fresh sessions and again in continued ones, reported separately. | Report the rate. A disclosure that succeeds once in twenty is still a disclosure route. |
| Content, tone and refusal objectives | Elicit output the product undertakes not to produce. | The lowest count, but never fewer than ten, because these are the objectives with the widest variance. | Agree a rate in advance, because otherwise every borderline output becomes a negotiation at report review. |
Those counts are ours rather than anybody’s standard, and they are set by consequence rather than by convenience: the objectives with an irreversible effect at the end get the most trials, because they are the ones where a low rate still matters. Nothing in the CERT-In instruments sets a trial count, and nothing in them anticipates a resource that is bought rather than worked — which is exactly why the number has to appear in the test plan §13.2.2(ii) requires, beside the calendar time and the man-hours.
Ten axes, and not one of them is depth
AI red teaming compared with VAPT, axis by axis
The comparison that gets drawn in most proposals varies one thing — how deep the testing goes — and presents the two as sizes of a single product. They are not. They differ on at least ten axes, and on four of them (reproduction, severity, retest and where the fix lives) the difference is a change of kind rather than of degree.
| Axis | VAPT of an AI-enabled application | AI red teaming |
|---|---|---|
| Objective | Establish which weaknesses in the software and infrastructure are exploitable, and what exploiting them reaches. §6(iv) defines the exploitation half as active testing to identify and exploit potential vulnerabilities, with the objective of determining whether they can be exploited to compromise the application, access sensitive data, or affect the underlying infrastructure and environment. | Establish what the system can be induced to say, disclose or do, and how reliably. The target is behaviour under adversarial input, not a defect in code. Success is a bounded envelope, not an empty finding list — the exercise almost never ends with “nothing worked”. |
| Scope, and how it is set | An asset list. §13.1.1(c) has the scope derived from the consolidated and updated asset inventory and §13.1.1(d) has it vetted by internal audit in consultation with the CISO. §13.2.3(ii)(c) requires version-specific details of web and mobile applications in the scope and the report. | A configuration and a rubric. The scope has to name the model and its version, the system prompt and its hash, the full tool schema, the retrieval corpora and their trust levels, the sampling parameters — and then state, in writing, which behaviours count as failures. None of that vocabulary appears in the audit policy guidelines, so the scope of work has to supply it. One exception, from a different CERT-In document: the Technical Guidelines on SBOM, QBOM & CBOM, AIBOM and HBOM v2.0 do name Model Name and Model Version, at §9.3 Table 10. |
| Method | Tooling used and then checked by a person, against a named testing guide. §8(i) discourages solely tools-based testing because tool-based audits may focus primarily on automated processes and overlook the non-automated components, producing an incomplete view of the overall security. | Hypothesis, variation and measurement. An operator writes an attack objective, generates many phrasings and encodings of it, runs them at volume through a harness, and grades the outputs against the rubric. Automation is unavoidable here — the population is the point — but the rubric and the grading remain human problems. |
| Reproduction | Deterministic. A request, a response, a screenshot. The same input produces the same result, which is why a finding can be handed to a developer and closed by a diff. | Statistical. The same prompt may succeed four times in ten. A reproducible finding is a recipe plus a rate: the exact input, the model identifier and version, the sampling parameters, the system prompt version, the tool schema, the retrieval snapshot, the number of trials, and the number of successes. |
| Severity model | §16.2(ii) requires CVSS for severity, supplemented with EPSS for real-world exploitation likelihood, and every observation mapped to a CWE and a CVE number. The machinery fits, because that is what it was built for. | The same clause applies and the machinery does not fit. CVSS has no metric for “works thirty per cent of the time”. EPSS is defined over published CVEs and a behaviour in your own system prompt has none. CWE sometimes maps and often does not. The honest report prints the vector, the observed rate and the gap — and uses §16.1(ii), under which the report should carry its exemptions, limitations and other constraints, to say so. |
| Deliverable | A findings register with reproduction steps, evidence, severity and remediation advice, against a stated scope and a stated environment. It proves that these paths worked, against this build, on these dates. | A behavioural report: objectives attempted, the rate at which each succeeded, the conditions under which the rate moved, the consequence each success reached through the tool layer, and the controls that would bound the consequence. It proves what this configuration did, not what the model is. |
| Retest and closure | Re-run the proof. §16.3(iv) has follow-up audits verify closure of the vulnerabilities and non-conformities highlighted in the previous audit, and §16.3(v) holds the final report until closure and the production follow-up are complete. | Re-run the suite, not the prompt. Closure means the mitigation holds across the whole family of variants at an agreed rate, and the suite becomes a regression asset that runs on every model version, prompt change and tool grant thereafter. Closing a prompt-injection finding by re-testing the three prompts that produced it proves nothing. |
| What moves the result under you | Code changes, configuration changes and new CVEs. §10.1(i) limits the auditor to reporting on the state of security at the point in time of the audit, which is a known and manageable limitation. | All of that, plus the provider. A hosted model can be updated, re-routed, re-tuned or have its safety filtering changed without notice and without a version bump you can see. The point-in-time limitation at §10.1(i) is the same clause and a much sharper constraint. |
| Where the fix usually lives | In the code or the configuration of the thing that was tested. §9.3(iii) makes patching and correction the auditee’s responsibility. | Almost never in the model. The durable fixes are deterministic controls around it: authorisation enforced in the tool rather than inferred from the conversation, least privilege on the credentials a tool holds, human confirmation on irreversible actions, output encoding, egress restriction, and removing capability the product does not need. |
| Named in the instruments as | §6(iii) Vulnerability Assessments and §6(iv) Penetration Testing, restated word for word at §5(xxi) and §5(xxii). §13.2.3(i)(a) names VAPT as an audit type to be communicated before commencement. | Nothing, precisely. §6(xxiii) names “Artificial Intelligence (AI) System Audits” as a class and defines it in a single sentence. The words “machine learning”, “LLM”, “large language”, “generative”, “agent” and “training data” do not occur anywhere in the instrument. |
The row that decides the most arguments is the last but one. A penetration-test finding points at the code that was tested, and the fix lands there. A behavioural finding points at the model, and the fix almost never does — it lands in the tool that trusted an argument, in the credential that was broader than the product needed, in the confirmation step that was never built. Teams that read an AI red team report as a list of things to make the model stop doing spend a quarter tuning prompts and end up with a system that fails differently. Teams that read it as a list of consequences to bound ship controls that still work after the next model upgrade.
Prompt injection and excessive agency, and the five classes around them
What is prompt injection, and what is excessive agency?
Prompt injection is the case where text the system treats as data is acted on as instruction. Excessive agency is the condition where the system has more capability, more permission or more autonomy than its function requires, so that when it is influenced the consequence is larger than it needed to be. The first is the mechanism and the second is the blast radius, and the split between them is the whole of the remediation argument on this page: you cannot reliably stop the influence, and you can decide in advance what the influenced system is able to reach.
The table names the classes in more than one vocabulary on purpose. The OWASP numbers are the ones a proposal will quote and they move between editions — excessive agency sat at LLM06 in the 2025 edition and sits at LLM03 in the 2026 one — so a number without an edition ages badly. NIST AI 100-2e2025, Adversarial Machine Learning, carries a generative-AI attack taxonomy with stable identifiers — Prompt Injection is NISTAML.018, Indirect Prompt Injection is NISTAML.015 and is listed separately, Data Poisoning is NISTAML.013 — and MITRE ATLAS carves the same space by tactic and technique instead. Naming one of them in the scope of work is how an AI finding becomes classifiable at all, and it is the answer to the obvious next question that §8(ii) leaves open: the clause disposes of a top-ten list as a methodology and names no AI replacement, so the scope has to.
| Class | Where it enters | What it looks like in a report | How you test for it | The deterministic control |
|---|---|---|---|---|
| Direct prompt injection LLM01 in the OWASP Top 10 for LLM Applications, 2026 edition · Prompt Injection, NISTAML.018, in the generative-AI taxonomy of NIST AI 100-2e2025 | The conversation itself. A user types it. | An input, a rate, and the output it produced. The least interesting member of the family, because the person doing it already has whatever authority the session carries. | Objectives with a consequence, many phrasings each, run at a stated trial count in fresh sessions. | None that is general. The deterministic work is at the other end — what the influenced model can reach. |
| Indirect prompt injection LLM01 in the OWASP 2026 edition, same item · Indirect Prompt Injection, NISTAML.015, is a separate entry in NIST AI 100-2e2025 — which is the better split | A document the system retrieves, a page it fetches, an email it reads, an attachment it parses, a tool description it re-reads at runtime, a memory it wrote last week. | The carrier, the ingestion path, the session in which it fired, and the fact that the author never touched the system. This is the one that scales: one poisoned document reaches every user who asks a question it answers. | Place the carrier through the product’s own ingestion route, then ask an ordinary question from a different account and grade what the assistant does. | Separate data from instruction in the context, partition the corpus so user-supplied content is never retrieved into another user’s session, and treat tool descriptions as code rather than as configuration. |
| Excessive agency LLM03 in the OWASP 2026 edition, up from LLM06 in 2025 · no NIST AI 100-2 entry, because it is a design property rather than an attack | Not an input at all. It is a design decision: a tool that can write where the product only needs it to read, a credential broader than the function, an action with no confirmation step. | The consequence chain, written out: influence at the model, a tool called, an authority the session never had, an effect that cannot be undone. | Read the tool register first, then attempt each consequence-bearing objective. The desk review usually produces more than the testing does. | The whole of it is deterministic. Authorisation enforced in the tool, least privilege on the credential, human confirmation on irreversible actions, capability removed rather than guarded. |
| Insecure handling of model output Named without a number in the OWASP 2026 edition · ordinary application security wearing a new hat, and it maps cleanly to a CWE | Wherever generated text reaches a renderer, a shell, a query, a template or another model. | A conventional finding with a conventional severity and a conventional fix. The class where the two engagements genuinely agree. | Enumerate the sinks from the scope and test each one, exactly as the penetration test would — this is the row a VAPT can and should cover. | Encode at the sink, parameterise the query, and never execute generated text. |
| Sensitive information disclosure through the context Named without a number in the OWASP 2026 edition · Prompt Extraction, NISTAML.035, and Leaking information from user interactions, NISTAML.036, in NIST AI 100-2e2025 | Anything assembled into the window: the system prompt, retrieved documents, tool returns, another tenant’s record fetched by a tool that did not check. | What was disclosed, to whom, at what rate, and whether the disclosure was of configuration or of somebody’s data. The second is an incident question as well as a finding. | Direct requests, indirect requests, and requests in another language — rates recorded separately. | Stop treating the system prompt as a secret, keep entitlement checks at the tool rather than in the prompt, and filter at the retrieval boundary rather than after generation. |
| Corpus and memory poisoning Data Poisoning, NISTAML.013, in NIST AI 100-2e2025 · no separate OWASP item for the memory case | The ingestion path into a retrieval corpus, a fine-tuning set, or a per-user memory store. The last is the newest and the most often unscoped. | Who can write into the store, through which feature, and what the written content then does. A memory-scoped finding affects one user and survives the session — a different shape from the corpus case and invisible to a corpus review. | Write through the product’s own path, then measure behaviour from an unrelated session. | Provenance and trust levels on every store, partitioning by principal, and a review path for anything that reaches a shared index. |
| Tool-argument manipulation The seam where both disciplines meet · nearest NIST AI 100-2e2025 entry is Compromising connected resources, NISTAML.039 | The arguments the model supplies to a tool: a case identifier, a recipient, an amount, a tenant, a file path. | Two halves that are each unremarkable alone. The red teamer shows the model can be induced to supply the argument, and the application tester shows the tool accepts it without checking. Together they are usually the most serious result in the engagement. | Name the tool endpoints as API assets in the conventional engagement, and chain the behavioural objective to them in the AI one. | Take the consequential argument from the session rather than from the model, and validate every argument against a schema instead of trusting generated text. |
Read the last column down and the argument of this page stops being rhetorical. Six of the seven controls are things an engineer can build and a tester can verify deterministically, and exactly one row — direct prompt injection — has no general control at the point of entry. That is the shape of the problem: the mechanism is not closable on current evidence and the consequence is entirely closable, which is why the money goes on the tool boundary rather than on the prompt. There is a much deeper treatment of how each class is actually probed, with sequences and evidence lists per class, in red teaming an LLM application. This table is the buyer’s version, written so that two quotes can be compared against it.
One row deserves a sentence on its own, because it is the row that decides which engagement finds it. Insecure handling of model output is not an AI problem wearing a costume — it is cross-site scripting, command injection or SQL injection with a language model supplying the payload, it maps to an ordinary CWE, it scores properly on CVSS, and a competent penetration test will find it if the scope says the rendering of model output is in scope. Put that sentence in the conventional scope and you have moved an entire class out of the expensive engagement and into the cheap one.
Two disciplines, and the seam between them
Is AI red teaming a penetration test?
The procurement question underneath “can one firm do both” is really a question about people, because the two jobs reward different training and different instincts. Some firms genuinely field both and say which individual does which. The ones to be careful of are the ones that field neither and describe the combination.
The application penetration tester
Knows: HTTP and its edges, authentication and session mechanics, authorisation models and how they fail, injection classes and their contexts, serialisation, cryptographic misuse, cloud identity, and the specific way a framework betrays the developer who trusted its defaults.
Works with: An intercepting proxy, a repeater, a fuzzer, an authenticated crawler, source review where it is in scope, and a testing guide — §8(ii) names the OWASP Web Security Testing Guide, the Application Security Verification Standard, OSSTMM 3, the CSA Cloud Controls Matrix and the Mobile Security Testing Guide as comprehensive alternatives to limited lists.
Produces: Proven paths, reproducible byte for byte, each with a demonstrated consequence and a fix that a developer can land in a pull request.
Blind spot: A system whose defect is not in the code. A tester can confirm that every endpoint enforces authorisation correctly and still not notice that the assistant calls those endpoints on behalf of whoever is talking to it.
The AI red teamer
Knows: How models are prompted, fine-tuned and served. The standard families of adversarial input and why they transfer between models. Retrieval architectures and where untrusted text enters them. Agent and tool-calling designs. And enough statistics to know the difference between a rate and an anecdote.
Works with: A harness that runs many variants of an objective at volume, a grading step — often a second model applied against a written rubric, and always spot-checked by a person — trace capture across the whole request including the assembled context, dataset management for the variant suite, and cost and rate-limit control, because this work generates token volume in a way a proxy does not.
Produces: Objectives, rates, conditions and consequences: what could be elicited, how often, under what circumstances, and what it reached through the tools.
Blind spot: The ordinary application. An operator who has spent a week on prompt variants has not looked at the session cookie, the password reset flow or the object identifiers in the API, and a system compromised by an ordinary IDOR does not need to be talked into anything.
Where the two overlap, and why it matters
Knows: The tool boundary. An AI red teamer establishes that the model can be induced to call a tool. An application tester establishes what that tool does with the call. Neither half is a finding on its own, and the pair is usually the most serious result in the engagement.
Works with: Nothing exotic — the tool schema, a set of test accounts at different privilege levels, and the discipline to ask, of each tool, whose authority it acts under and what stops it acting outside that authority.
Produces: A finding written in a language both sides can act on: the tool performs no authorisation of its own and trusts an argument supplied by the model, therefore any influence over the model is authority over the tool.
Blind spot: It falls between two scopes with great reliability. The application test did not name the tool endpoints as assets, and the AI engagement was scoped to the model. Naming the tool endpoints as first-class API assets in the inventory is the cheapest thing on this page and the most commonly skipped.
One clause is worth holding in mind when a firm offers to staff this from wherever it can. Where the organisation is CERT-In empanelled, §10.2(vi) requires it to deploy only manpower declared to CERT-In in the snapshot information form, and §12.2(iii) tells auditees to verify identity and designations and states that auditing organisations should not field freelancers, interns, freshers, moonlighters, third party consultants or employees serving their notice period — and that the auditee organisation should verify compliance. That is a real constraint, and it bites hardest on any engagement a firm intends to staff from outside its declared manpower. Where the firm is not empanelled the constraint does not exist at all, and the named-individual question belongs in your contract instead — ask which person does the application half, which person does the behavioural half, and what each has built or broken before.
What each document proves, and what it is asked to prove
What should an AI red team report contain?
Most of the damage in this area is done not by bad reports but by good reports quoted outside their scope — in a summary email, a board slide, a questionnaire answer. Each of these four documents proves something specific and narrow, and each is routinely asked to prove one of the others.
The VAPT report
Proves: That against this build, in this environment, on these dates, these paths were exploitable and these were not — with reproduction, evidence and a remediation position for each. Where the auditing organisation is CERT-In empanelled, §9.5(iii) adds a signature chain: signed by the auditors who conducted the audit, reviewed and signed by a designated reviewer from mid-management who was not part of the audit team, and authorised and signed by the head of the auditing organisation.
Does not prove: Anything about model behaviour. A clean web application test and an assistant that can be talked into emailing another customer’s case file are entirely compatible, and the report does not claim otherwise — it is the summary email that does.
Who reads it: Customer security questionnaires, SOC 2 and ISO 27001 evidence, hosting gates, and — where the certificate is the object — the audit clearance GIGW guideline 5.3.1 requires before hosting in production.
The AI red team report
Proves: That this configuration — this model version, this system prompt, this tool schema, this retrieval corpus, at these sampling parameters — produced these behaviours at these rates under these conditions, and that the consequence reached this far through the tool layer.
Does not prove: That the model is safe, that the behaviour will not reappear after a provider-side update, or that a different configuration of the same model behaves the same way. It is also not an assurance certificate: a red team that failed to reach an objective has said something about the objectives it tried, not about the system.
Who reads it: Your own risk register, an ISO/IEC 42001:2023 management review, a board that has asked what the assistant can be made to do, and increasingly an enterprise customer whose questionnaire has grown an AI section.
The safe-to-host or audit clearance certificate
Proves: What its own text says, and no more. GIGW guideline 5.3.1 requires that the website, web application, web portal or mobile app “have been Security Audited and an Audit Clearance certificate has been issued by NIC/ STQC/ STQC empanelled laboratory/CERT-In empanelled laboratory before hosting in production environment”. The certificate rests on application-level testing, and §15.2.3(ii) requires the application hash values and version numbers to be obtained from the auditee and included in the report.
Does not prove: That the AI behaviour was examined. Nothing in GIGW chapter 5.3 mentions a model, and the hash rule was written for an application build — which raises a question worth asking out loud: what exactly is hashed when the behaviour of the thing depends on weights hosted by somebody else.
Who reads it: A government hosting provider, an STQC Website Quality Certification application, and the evaluator action that requires the certificate to be valid rather than merely present.
The §6(xvi) red team assessment report
Proves: How your detection and response function performed against an adversary on those dates: what was generated, what was triaged, what was escalated, what was contained, and how long each took.
Does not prove: That the estate is clean, and nothing at all about the AI system unless the operation happened to route through it. It answers a question about your people and your runbooks.
Who reads it: A board that asked whether the organisation would notice, and a security function building the case for what it needs next.
The third row deserves a sentence of its own, because government buyers arrive at this page with it. A safe-to-host or audit clearance certificate is an application-level artefact. GIGW guideline 5.3.1 requires the security audit and the clearance certificate before hosting in production, §15.2.3(ii) requires the application hash values and version numbers to be obtained from the auditee and included in the report, and nothing in GIGW chapter 5.3 mentions a model. So if your application calls a hosted model, the certificate describes the build you hashed — and what the assistant does depends on weights somebody else controls and can change. That is not a reason to skip the certificate. It is a reason not to let it stand in for an answer it was never asked. See what a safe-to-host certificate is and what a CERT-In empanelled auditor may and may not do.
What your system can do decides what is worth buying
Do we actually need AI red teaming?
Not always, and a page that answers this question with “yes” is selling rather than advising. The variable that decides it is not how important the system feels or how much attention AI is getting — it is what the system is able to do. Find your row. The top one is a genuine “no”, and the rows below it escalate because the consequence of being influenced escalates, not because the technology is more impressive.
| What the system can do | What is proportionate to buy | What evidence you should end up holding | The clause that puts it in scope |
|---|---|---|---|
| A retrieval-free assistant over public content, with no tools | The VAPT, plus an output-encoding check where model output is rendered. No behavioural engagement is warranted. | A conventional report whose scope annexure names the chat surface and the model, and a finding or a clean result on the rendering path. | §6(iii) and §6(iv). §6(xxiii) is not engaged in any useful sense. |
| Retrieval over internal documents, read-only, still no tools | The VAPT, plus a small disclosure-objective suite at low trial counts against the retrieval boundary. | Rates for system-prompt recovery and for cross-entitlement disclosure, with the corpus snapshot and its as-of date recorded. | §6(xxiii) for the behavioural half, and §13.1.1(c) to get the corpora into the inventory first. |
| Tools that write to records on the user’s behalf | The VAPT with the tool endpoints named as API assets, plus a consequence-bearing behavioural suite. | The tool register, the consequence chain for each objective attempted, and rates with denominators. | §13.1.1(a) already has APIs inside the scope, and §6(xxiii) covers the behavioural work. |
| Irreversible or external-effect actions — pay, send, delete, publish | Everything above, and a human-confirmation design review before any testing is commissioned. | A written decision, per action, on what confirms it and whether the model can supply that confirmation. Then the test results against the design. | Not in the instruments. §13.1.2(iii) reaches it only once you decide a new tool grant is a major change, which is our reading. |
| Autonomous multi-step operation, or agents calling other agents | All of the above, plus loop, termination and delegation objectives, and a connector inventory. | The bound on every multiplier — steps, spend, time, tool calls — and the measured behaviour at each bound rather than below it. | Nothing in the five CERT-In documents searched for this page addresses autonomy. It is entirely a scope-of-work matter. |
Two notes on using the table. The rows are cumulative — row four means everything in rows one to three as well, not instead of them — and the boundary that matters most is between rows two and three, because that is where the system stops answering and starts acting. If you are not sure which side of it you are on, the test is not architectural: ask whether anything the assistant does leaves a mark somebody else can see. And if you land in row one, say so in writing and re-read the row when the roadmap adds a tool, because the upgrade from row one to row three is usually a fortnight’s work by somebody who has never read this page.
When you need both, and the order that makes sense
Should we do the VAPT or the AI red team first?
Wherever the table above puts you at row three or below, you need both — and the order is not a matter of taste. Step 02 below states the reason in one sentence and this paragraph will not repeat it: the conventional engagement comes first because an authorisation defect underneath makes the behavioural question moot, and because it is cheaper to find and cheaper to fix. What follows is the programme in the order the dependencies actually run, with the reason each step sits where it does.
Inventory the AI system as assets, not as a product
The scope has to be derived from the consolidated and updated asset inventory under §13.1.1(c), and vetted by the internal audit team in consultation with the CISO under §13.1.1(d). For an AI-enabled application that means the inventory has to carry things most inventories do not: each model and its version and provider, each system prompt and its version, each tool with its arguments and the credential it holds, each retrieval corpus with a note on who can write into it, and the inference endpoints themselves. Until those are assets with owners, no scope drawn from the inventory can reach them.
Why here: Everything downstream is scoped from this list. An asset that is not on it is out of scope by omission rather than by decision.
Test the application and the infrastructure first
The conventional engagement comes first for a blunt reason: if the API tier does not enforce authorisation per object, nobody needs to talk the assistant into anything. They can call the tool endpoint directly with a case identifier they guessed. Buying the exotic engagement while the ordinary one is unbought is the AI version of commissioning adversary simulation against an estate nobody has assessed — it measures the wrong thing at the higher day rate.
Why here: An authorisation defect makes the behavioural question moot, and it is cheaper to find and cheaper to fix.
Name the tool endpoints as first-class API assets in that engagement
This is the one instruction on this page that costs nothing and changes the most. The endpoints a model calls are an API tier, they usually have no interactive client, they were written quickly, and they are frequently trusted because “only the orchestrator calls them”. Put them in the asset annexure with their schemas and their test accounts, and a conventional penetration test will test them as it tests anything else. §13.2.3(ii)(c) already requires version-specific details of web and mobile applications in the scope and the report. Extend the same discipline to the tool surface.
Why here: It converts the most serious class of AI finding into an ordinary application finding, which is a far better place for it to be.
Then run the AI red team against the orchestration, the model and the tools together
Scope it as one system rather than as a model. The objectives worth setting are the ones with a consequence at the end: cause a tool to act outside the authority of the person in the conversation, cause the system to disclose content the person is not entitled to, cause a retrieved document to change the system’s instructions, cause an irreversible action without a confirmation. Each objective gets a trial count and a grading rubric before the work starts.
Why here: The model in isolation is a research question. The model with your prompts, your corpus and your tools is your risk.
Fix outside the model wherever you can
Remediation that depends on the model refusing is remediation with a failure rate. The durable controls are deterministic and they sit in the layers around it: the tool checks the session’s authority rather than trusting an argument, the credential the tool holds is scoped to the least privilege the product needs, irreversible actions require a human confirmation that the model cannot supply, output is encoded before it is rendered, egress is restricted so an exfiltration route does not exist, and capability the product does not need is removed rather than guarded.
Why here: §9.3(iii) puts patching and correction on the auditee. Choosing controls you can prove beats choosing controls you have to re-measure every quarter.
Retest the family, and keep the suite
§16.3(iv) has follow-up audits verify closure of what the previous audit highlighted, and §16.3(v) holds the final report until closure and the production follow-up are complete. For the behavioural findings, closure is a rate across the variant family rather than a pass on the original prompts — and the suite that measures it should be handed over, version-controlled and re-runnable by your own team.
Why here: A suite you own turns a point-in-time result into a control you can evidence between engagements.
Wire the suite into your change gate
§13.1.2(iii) has every change classified minor or major, and a major change — the guidelines give system overhauls, technology migrations and configuration adjustments that affect sensitive data or critical infrastructure as the examples — must undergo a cyber security audit before implementation. §13.1.2(iv) adds that an audit should be performed after every major change in infrastructure and application, based on the criticality involved. Applying that classification to an AI system is our reading rather than the instrument’s words, and the reading is straightforward: a new model version, a system prompt rewrite, a new tool grant and a new corpus source each change what the system will do, which is the definition the clause uses.
Why here: Without a gate, the behavioural result silently expires the first time somebody upgrades the model in a config file.
Only then ask the detection question
If the board’s question is whether anybody would notice, that is a §6(xvi) red team assessment and it is a third purchase with its own rules — an objective rather than an asset list, notification limited to key personnel under §15.1.1(i), §15.1.1(ii) under which the auditee should refrain from implementing any unusual or major network changes during the auditing or testing period, and specific written permissions under §13.2.7(ii) for survivability failures, denial of service, process testing and social engineering.
Why here: Detection is a question about your people and your runbooks. Neither of the first two engagements answers it, and asking it before the first two are done produces a finding you already know.
There is one genuine exception to the order, and it is worth stating because it applies to more organisations than it should. Where an assistant is already live with tools that can write, send, pay or delete, the excessive-agency question is live now and does not wait for a testing calendar. In that position, review the tool schema and the credential each tool holds immediately, as a desk exercise, while the conventional engagement is being scoped. That review needs nobody’s permission, produces a decision rather than a finding, and usually takes an afternoon. It is also the single highest-value hour on this page, so here is the template for it rather than a further recommendation to do it.
The tool register, which is the deliverable of that afternoon
One row per tool the model can invoke. The rows below are filled in from the invented system in the worked example further down, so the shape is legible — nothing in them describes a real system, a real client or a real outcome. Fill your own version and the excessive-agency position stops being a judgment and becomes a document you can hand to an auditor, an insurer or a board.
| Tool | Arguments the model supplies | What the session should supply instead | Credential and its scope | Systems touched | Read or write | Reversible | Human confirmation | Rate or quantity limit | Where the call is logged |
|---|---|---|---|---|---|---|---|---|---|
| Look up a case by identifier | case_id | case_id — the session’s own case, or an entitlement check against the one supplied | Service account with read on the whole case table | Case management | Read | n/a | No | None set | Orchestrator log only — no per-call record in the case system |
| Update contact details on a case | case_id, field, value | case_id from the session, field and value validated against an allow-list | Same service account, with write | Case management | Write | Yes, with an audit trail | No | None set | Case system audit trail attributes the change to the service account, not the user |
| Escalate a case | case_id, reason | case_id from the session, reason free text | Same service account | Case management, notification queue | Write | Partly — the escalation can be withdrawn, the notification cannot | No | None set — an availability finding before it is anything else | Queue log |
| Send an email | recipient, subject, body | Recipient restricted to addresses already on the case, and body assembled by the application, not by the model | Outbound mail relay, unrestricted recipients | Mail, and whatever the body contains | Write, external effect | No | No — and this is the row the review exists to find | None set | Relay log, retained 30 days |
Three columns do the work and it is worth saying which. The difference between “arguments the model supplies” and “what the session should supply instead” is the entire fix for the most serious class of finding in this architecture — an argument taken from the session cannot be steered by a document. The credential column is where least privilege becomes a decision rather than an aspiration: a read-only account cannot be talked into writing. And the confirmation column, read against the reversible column, is the register’s own audit: any row that is irreversible and unconfirmed is a finding before anybody has tested anything.
Step four is the one with the most moving parts, and it has a page of its own: red teaming an LLM application sets out how the exercise is actually run — objectives, variant generation, the grading step, the trial protocol and the artefacts that come out of it. Step eight, the detection question, is a §6(xvi) engagement and is treated in VAPT, penetration testing and red teaming.
What the scope of work has to carry, because the instrument does not
What goes in an AI red teaming scope of work?
Fourteen items, each with the nearest CERT-In clause where one exists and an honest note where none does. Three of them — the rubric, the trial counts and the compute ceiling — appear in none of the five CERT-In documents searched for this page, and they are the three that most reliably decide whether the engagement produces something defensible. The table sets out what to write down. Immediately after it is the writable version, because a page that spends this long arguing that the buyer must draft the specification should not then leave the drafting to the buyer.
| Scope item | What to write down | Nearest clause |
|---|---|---|
| Model identity, version and control | The provider and the exact model identifier and version, and who controls it: a hosted API you cannot pin, a model deployed in your own tenancy, or weights on your own hardware. Say by what method it is pinned, and require the report to record the identifier the API returns rather than the one requested. Record the provider’s deprecation and update policy in the same place, because that is the clock on the result. | Two, and the second is the on-point one. §15.2.3(i) requires the environment to be specified — the guidelines give test, development, UAT, pre-production and production, introduced with “such as”, so the list is open and a model version belongs in the same statement. And CERT-In’s Technical Guidelines on SBOM, QBOM & CBOM, AIBOM and HBOM v2.0 name the field directly: §9.3, Table 10, Minimum Elements of AIBOM, lists Model Name, Model Version, Model Type and Model Developer, and §9.4.2.6 asks for model lineage to be tracked as versions, retraining activities and modifications. |
| System prompt version and hash | The exact system prompt in force during the test, captured with a hash, plus any per-tenant or per-flow variations. A finding obtained against one system prompt says nothing about another. | §15.2.3(ii) requires application hash values and version numbers to be obtained from the auditee and included in the report. Extending “application” to cover the prompt and the tool schema is our reading of a clause written for a build artefact — but it is the only sensible reading, and it is the artefact that actually determines behaviour. |
| The full tool schema | Every tool the model can invoke: its name, its arguments, the credential it uses, the systems it touches, whether its effect is reversible, and whether it requires a human confirmation. This document is the single most useful artefact in the engagement and it usually has to be written for the first time in order to be supplied. | §13.1.1(a) has the scope cover the entire cyber infrastructure including APIs, and §16.1(iii) requires every asset provided in scope to be mentioned in the report. |
| Retrieval corpora, memory stores and their trust levels | Each source the system retrieves from, who can write into it, and whether user-supplied content reaches it. A corpus that ingests uploaded attachments is an untrusted input channel and should be labelled as one in the scope, not discovered as one in the findings. List persistent memory and user-profile state here too and separately: an instruction written into a per-user memory survives the session it arrived in, is scoped to one person, and will not appear in any review of the shared corpus. | §13.1.1(c) derives the scope from the asset inventory. A corpus nobody listed as an asset is out of scope by omission, and a memory store is the asset class least likely to be on anybody’s list. |
| Which input channels are in scope, including the non-text ones | Name the channels the system will read: text, uploaded images, screenshots, scanned documents, PDFs with a text layer, audio. The default harness probes text because the harness speaks text, so an unnamed channel is tested by neither engagement — the penetration test looked at the upload and the AI engagement looked at the prompt. | Nothing in the audit policy guidelines, the empanelment terms, the 2018 auditor guidelines or the secure application guidelines reaches the question. §13.1.1(a) puts the upload path in scope as infrastructure and stops there. |
| Third-party tool connectors and their protocol layer | Which connectors or tool servers are enabled, who publishes each, what credential each holds, whether its version is pinned or resolved at start-up, and whether its tool descriptions can change server-side without a release on your side. A tool description is prompt text a supplier controls, and on most implementations it is re-read at runtime rather than fixed at approval. | No clause addresses it. §6(xxvi) and the BOM technical guidelines are the nearest thing, as component transparency rather than as a behavioural control. |
| Sampling parameters, and whether they match production | Temperature, top-p, maximum tokens, any seed the provider exposes, and whether the values used in testing are the values used in production. Testing at temperature zero and shipping at 0.7 produces a result about a system you do not run. | Nothing in the audit policy guidelines, the empanelment terms, the 2018 auditor guidelines or the secure application guidelines addresses this — all four were searched for the vocabulary. It is a scope-of-work item and it exists because the instruments are silent. |
| The behaviour rubric | Written before the work starts: what counts as a failure, in words specific enough that two reviewers grade the same output the same way. “Harmful content” is not a rubric. “Provides step-by-step instructions that materially advance X” is closer. The rubric is the part a buyer should read most carefully, because it determines the finding set. | Item 22 of Annexure I to the empanelment terms expects reports to use qualitative metrics based on industry-accepted methods, derived from a mathematical formula and not on the feelings of the auditor. A written rubric is how that expectation survives contact with a subjective judgment. |
| Trial counts and the reporting threshold | How many attempts per objective, and the success rate at which something becomes a finding rather than a note. Both numbers are cost drivers and both belong in the test plan. Set them by objective class rather than uniformly — the arithmetic for what each count buys, and what a zero-success run actually licenses you to claim, is set out in the determinism section above. | §13.2.2(ii) requires the test plan shared by the auditing organisation to include both calendar time and man-hours. For this work, trial counts belong beside them, because compute is a resource the clause did not anticipate. |
| Credentials, blast radius and the test environment | Whether the exercise runs against production or a replica, and — separately — whether the tools in that environment hold production credentials. A staging assistant wired to production tool credentials is the worst of both arrangements. Where the exercise must touch production, agree a stop condition and a named holder of stop authority on each side. | §15.1.2 requires temporary privileged access to be revoked immediately on completion. §15.2.2(iv) states that in critical environments where availability is a top priority, testing should be conducted in a passive manner to avoid any potential downtime or service disruption. §13.2.6 says there should be a well-defined escalation matrix for both the auditee organisation and the auditing organisation, shared with the respective authorities. |
| Cost, rate limits and who pays for the tokens | This discipline generates request volume that a proxy-based engagement does not, and a provider rate limit hit mid-run invalidates a trial set rather than merely slowing it. Agree the ceiling, the account it runs on, and what happens when it is reached. | Not addressed in any of the five CERT-In documents searched for this page. It is the commonest practical reason an AI engagement overruns, and it is entirely avoidable in the scope. |
| The provider’s own terms | Testing a third-party hosted model may be constrained by the provider’s acceptable use policy, and some providers require notification or operate their own disclosure route. That is a contractual question between you and them, it is yours to answer rather than the tester’s, and it is best answered before the engagement letter is signed. | §15.2.2(iii) bars testing that reaches external entities — customers, business partners, vendors or other third parties — unless specific written consent is obtained. The clause is written about people, but the same caution reads across to a service you do not own. |
| Handling of the material the exercise generates | Prompts, outputs and traces produced during testing are auditee data, and some of them will be content nobody wants in a shared drive. Agree storage, retention, redaction in the report, and who reviews it. Where the probes target harmful content, agree a reviewer rotation as well — this is unusual among security engagements in producing material that is unpleasant to read at volume. | §10.3 provides that auditee-related data should be stored only on systems located in India, kept encrypted on the auditor’s laptop, and wiped after completion so that it cannot be retrieved by any known forensic technique, with a certificate to that effect formally issued to the auditee. §10.3(vi) sets a one-year default retention where the agreement is silent. The 2014 Policy Guidelines for Handling Audit related Data, required by name at §10.3(vii), bar hosting the report on public file or cloud services unless under the auditing organisation’s control or specifically authorised. |
| Which standards the method is built from | Name the testing guide and the control standard, at a version — and for the AI half, name four different kinds of document, because they do different jobs and a proposal that names only one has not answered the question. A taxonomy to classify attacks against: NIST AI 100-2 E2025, Adversarial Machine Learning, or MITRE ATLAS, or both, because the two carve the space differently. A method document for how the exercise is run: the OWASP GenAI Red Teaming Guide, which is a method rather than a list and so survives the §8(ii) objection intact, read beside NIST AI 600-1, the generative-AI profile that defines AI red-teaming as a structured testing exercise and sets out suggested actions for it — a voluntary US profile, never a duty that reaches an Indian entity. A management standard for the governance half: ISO/IEC 42001:2023, with ISO/IEC 23894:2023 for AI risk management and the NIST AI Risk Management Framework, AI 100-1, as the freely retrievable equivalent. And a failure-mode checklist, cited at its edition: the OWASP Top 10 for LLM Applications is versioned by year and the numbers move between editions, so an item number without an edition is ambiguous within twelve months of being written. | §8(ii) states that limited lists such as the OWASP Top 10, SANS Top 25 and similar should not be considered standards or references for audits, and names the comprehensive alternatives — ISO/IEC, the Cyber Security Audit Baseline Requirements, the CSA Cloud Controls Matrix, OSSTMM 3, the OWASP Web Security Testing Guide, the ASVS, the Mobile Security Testing Guide and the DevSecOps Maturity Model. Reading a top-ten list for LLM applications as one of the “similar” limited lists is our reading, and it points the same way: use it as a checklist of failure modes, never as the methodology. The clause names no AI equivalent, which is why the four kinds of document above have to be named in the scope instead. |
A note on how to read the “nearest clause” column, because this page is careful about it elsewhere and should be here. Where a row says a clause requires something, it does. Where a row extends a clause — reading “application hash values and version numbers” at §15.2.3(ii) across to a system prompt and a tool schema, or reading §13.1.2(iii)’s major-change classification across to a model version — that is our reading of a clause written before this architecture existed, and the row says so. The distinction matters in a procurement dispute, and it costs nothing to keep.
Six paragraphs you can paste into the scope of work
Fill the bracketed fields, delete what does not apply, and have your own counsel read it. This is a starting point written from published clauses, not legal advice, and the clause numbers are printed so that a supplier can check them rather than take our word for it. The paragraphs assume the annexures exist — which is the point, because writing them is the preparation this work actually consists of.
- Type of audit. The engagement comprises an Artificial Intelligence (AI) System Audit in the sense given at §6(xxiii) of CERT-In’s Comprehensive Cyber Security Audit Policy Guidelines, version 1.0, dated 25 July 2025, limited for the avoidance of doubt to the security and adversarial-resilience limbs of that definition. Where the surrounding application is in the same purchase, it is scoped as separate and separately priced phases of Vulnerability Assessment and Penetration Testing under §6(iii) and §6(iv), reported as two clearly distinguished finding sets in a single report.
- Configuration under test. The system under test is the assembly of the model, the orchestration service, the system prompt, the tool schema and the retrieval corpora listed at Annexure A. Annexure A shall state the provider, the model identifier and version, and the method by which the version is pinned. It shall also state the system prompt version and its hash, the full tool schema (exported verbatim), every retrieval corpus and memory store with its trust level and its as-of date, and the sampling parameters, together with a statement of whether those parameters are the production values. The auditing organisation shall record in each trial the model identifier as returned by the API rather than as requested.
- Method and reporting criteria. The attack objectives are at Annexure B and the grading rubric is at Annexure C, both supplied before commencement. The test plan shall state the number of trials per objective, the grading method, the rate at which machine grading is spot-checked by a person, and the reporting threshold at which a success rate becomes a finding. Objectives attempted and not achieved shall be reported with their trial counts. Exclusions are listed at Annexure D and the report shall repeat them.
- Environment, credentials and escalation. Testing shall be performed on [named environment]. For each tool, Annexure A states the credential it holds in that environment and the systems that credential can reach. Temporary privileged access shall be revoked immediately on completion in accordance with §15.1.2. Where testing touches a critical environment in which availability is a top priority, it shall be conducted in a passive manner in accordance with §15.2.2(iv). The escalation matrix required by §13.2.6 is at Annexure E and shall name, for each party, a stop authority, a deputy and an out-of-hours contact, together with the conditions on which either party may halt the exercise.
- Compute, data and material. The compute ceiling is [X] and the work runs on [whose] account. On reaching the ceiling the auditing organisation shall stop and notify rather than reduce trial counts silently. Prompts, outputs and traces generated during the engagement are auditee data and shall be handled under §10.3, stored only on systems located in India, encrypted, and wiped after completion with a certificate of deletion issued to the auditee. Where objectives target harmful content, the auditing organisation shall state its reviewer arrangements and redaction approach in the report.
- Reporting and closure. The report shall carry the contents required by §16.1(ii), shall name every in-scope asset under §16.1(iii) and every asset that could not be reached with the reason, and shall record severities in the convention agreed at Annexure F — full CVSS vector, the CVSS-B label where only base metrics were used, observed successes over trials, CWE where one fits, and CVE and EPSS left blank with the reason printed. Closure shall be measured across the variant family, as defined at Annexure B, at a rate agreed before work starts, and the variant suite shall be delivered to the auditee version-controlled and re-runnable, under [licence].
Three fields do most of the work and are the three most often left blank: the pinning method in the second paragraph, the trials per objective in the third, and the compute ceiling in the fifth. A scope that fixes all three has removed the three largest sources of disagreement about what was actually bought. The conventional half has its own paste-able block in what VAPT is, and the two are designed to sit in one document.
Fourteen questions to ask before you sign anything
Read these down the phone. Where there is a clause behind the question it is printed, and where there is not it says so — which on this subject is about half of them, and that is itself the answer to several of these questions. The first three will usually tell you whether to ask the remaining eleven.
- 01 Which §6 class or classes is this, given that §6(xxiii) names AI System Audits separately from §6(iii) and §6(iv), and will those words appear in the scope of work? (§13.2.3(i)(a))
- 02 Which model and version is in scope, by what method is it pinned, and will the report record the identifier the API returns rather than the one requested? (not in the audit policy guidelines, but AIBOM §9.3 Table 10 names the field)
- 03 Are the tool endpoints in the asset annexure as API assets in their own right, with test accounts at more than one privilege level? (§13.1.1(a), §13.1.1(c), §16.1(iii))
- 04 How many trials per objective, and how was that number chosen for each class of objective? (§13.2.2(ii), extended)
- 05 Is the grading rubric supplied before commencement, or written alongside the report? (Annexure I item 22, empanelment terms)
- 06 Who grades, and at what rate does a person spot-check machine grading? (not in the instruments)
- 07 Which environment, and what credential does each tool hold in it — tool by tool, in writing, before the start? (§15.1.2, §15.2.2(iv))
- 08 What is the compute ceiling, whose account does it run on, and what happens when it is reached? (not in the instruments)
- 09 Will the guardrail be measured with and without, and will both numbers appear in the report? (not in the instruments)
- 10 What closure rate is agreed in advance, and how is the variant family defined? (§16.3(iv), and the definition is not in the instruments)
- 11 Is the variant suite handed over, version-controlled and re-runnable by our own team, and under what licence? (not in the instruments)
- 12 How will the prompts, outputs and traces be stored, redacted and reviewed, and is there a reviewer rotation where objectives target harmful content? (§10.3, and the rotation is not in the instruments)
- 13 Which input channels are in scope — text only, or images, documents and audio as well? (not in the instruments)
- 14 Which taxonomy will an attack be classified against, and which method document is the exercise built from? (§8(ii) requires a comprehensive standard and names no AI equivalent)
What the cost is built from, and why two quotes differ by a factor of three
How much does AI red teaming cost, and how long does it take?
No figures here, and that is a position rather than an evasion: a number without a scope is attached to nothing, and the argument of this page is that the scope is what decides whether the result is worth anything. What is useful is the arithmetic. The drivers below are unusual for a security engagement — two of them are resources that are bought rather than worked — and §13.2.2(ii), which makes the test plan carry calendar time and man-hours, was written before either existed.
How many objectives
Not in the instruments
The unit of work. An objective is a stated outcome the system should not reach, with a rubric clause attached to it, and each one is written, run, graded and reported. A quote that does not say how many objectives it buys has not said what it is selling.
How many trials each objective gets
§13.2.2(ii), extended
The multiplier nobody prices. Twenty trials per objective against three is a seven-fold difference in compute and a categorical difference in what the report can claim. §13.2.2(ii) makes the test plan carry calendar time and man-hours. Trial counts belong beside them, because the clause did not anticipate a resource that is bought rather than worked.
Tools instrumented, against probing the model alone
§13.1.1(a), §16.1(iii)
Probing a model in isolation is cheap. Standing up a system with its tools connected, its credentials scoped for a test environment and its calls traced is the expensive half — and it is the half where the serious findings live. This is the single largest legitimate gap between two prices.
How many corpora, and whether any accepts user-supplied content
§13.1.1(c)
Each corpus is an ingestion path to test. One that accepts uploads is a different and larger piece of work, because the objective has to be placed through the product’s own route and then fired from an unrelated session.
Whether a guardrail is measured with and without
Not in the instruments
Running the suite twice, once with the classifier enabled and once without, is the only way to state its false-negative rate — and it roughly doubles the run. Worth it if you are relying on the guardrail. Skip it honestly if you are not.
Replica or production
§15.2.2(iv), §15.1.2, §13.2.6
Production brings passive technique where availability is the priority, an escalation matrix that has to work in real time, and a per-tool decision about what each credential can reach. A replica brings the cost of building one that behaves like production, which is frequently larger.
Who writes the rubric
Annexure I item 22, empanelment terms
Grading against a written rubric is expensive, while grading against a vibe is free. If the rubric is yours to write, that is preparation effort on your side of the line. If it is the supplier’s, it is billable hours before any testing starts — and either way it must exist before commencement, because a rubric written with the report cannot be falsified.
Whether the variant suite is a handover deliverable
Not in the instruments
A suite built to be run once and a suite built to be handed over, version-controlled and re-runnable by your team are different pieces of engineering. The second costs more and is the only one that survives the engagement.
The compute ceiling, and whose account it runs on
Not in the instruments
This discipline generates token volume a proxy-based engagement does not, and a provider rate limit hit mid-run invalidates a trial set rather than merely slowing it. Agree the ceiling, the account and what happens when it is reached, before rather than during.
The duration pattern, which runs the opposite way to a VAPT
On a conventional engagement the testing is the long pole and the preparation is a week of access requests. Here it inverts: the testing runs fast, because a harness executes trials at machine speed, and the preparation runs slow, because the scope artefacts — the tool schema, the corpus inventory, the objectives, the rubric — frequently have to be written for the first time in order to exist. That is the single most useful sentence on this subject and it is worth planning around: budget the preparation honestly and the engagement rarely overruns, but skip it and the first week goes on discovery nobody priced, at the testing day rate.
One arithmetic check to run on a quote. Take the number of objectives, multiply by the trials per objective, and ask what the total compute cost of that many runs is at the provider’s published rate. If nobody on the supplier side can do that sum in the scoping call, the compute line has not been priced — and it is the line that stops the engagement at sixty per cent of the planned trials, which is the failure mode described in the table further down.
One invented system, two engagements, two finding sets
What does an AI red team engagement look like end to end?
Abstractions are easier to argue with than examples, so here is an example. It is invented — the first step says so in terms — and it exists to make the shapes legible. Follow it through and the argument for the sequence above stops being a preference and becomes arithmetic.
The system, which is invented
Everything in this walkthrough is made up. It is not a client, it is not a sector, and nothing in it is a claim about what any engagement achieved. A grievance-handling assistant sits on a services portal. A chat panel in the portal calls an orchestration service, which assembles a system prompt, appends the conversation history, retrieves from two indexes — a public circulars index and an internal standard-operating-procedure index — and calls a hosted model. The model has four tools: look up a case by identifier, update the contact details on a case, escalate a case, and send an email. Attachments uploaded with a grievance are parsed and added to the internal index so the assistant can answer questions about them.
What the VAPT finds
Two weeks of authenticated application testing across the portal, the chat API and the hosting tier. It returns a session that is not invalidated on password change, an object-identifier defect that lets one citizen read another citizen’s conversation history, missing rate limits on the message endpoint, a stored cross-site scripting issue where model output containing markup is rendered without encoding, an over-permissive storage bucket holding uploaded attachments, and an outdated library on the web tier. Every one of those is real, every one is worth fixing, and two of them are serious. The report is accurate and the scope annexure is honest about what it covered.
What the VAPT does not find, and why
The four tool endpoints were not in the asset annexure. They have no interactive client, they are called only by the orchestration service, and nobody listed them when the inventory was compiled. So they were never tested — not because the tester was weak, but because §13.1.1(c) derives the scope from the inventory and they were not in it. The system prompt was not an artefact anybody thought to hand over. The internal index was in scope as a datastore and not as a content channel.
What the AI red team finds — the injection
An operator uploads a grievance attachment containing text addressed to the assistant rather than to a human reader. The parser adds it to the internal index. A later, unrelated user asks a question the retriever answers from that document, and the assistant follows the instruction in it. Across twenty trials at production sampling parameters, the instruction is followed in eleven. The finding is written as the recipe plus the rate: this document, this index, this model version, these parameters, eleven of twenty.
What the AI red team finds — the consequence
The instruction the document carries is to look up a specific case and email a summary of it. The look-up tool accepts a case identifier as an argument and returns the case. It performs no authorisation of its own — it was written on the assumption that the orchestrator would only ever ask for the case belonging to the session. The email tool accepts an arbitrary recipient. So the chain runs: untrusted document, model instructed, tool called with an identifier the session’s user does not own, contents emailed to an address the attacker chose. Nothing in that chain is a model defect. Every link in it is an authorisation decision that was never made.
What the AI red team finds — the smaller ones
The system prompt is recoverable in sixteen of twenty trials with a direct request, which matters mainly because it names the tools and their arguments and so shortens the work above. A refusal behaviour that holds in English fails in three of twenty trials when the same request is made in another language. And the assistant will state a case outcome it has no source for when the retrieval returns nothing relevant, which is a service-quality finding rather than a security one and belongs in the report labelled as such.
What the finding looks like when it is written down
The report is the product, so here is one finding as it would appear in it. Title: indirect prompt injection via a grievance attachment reaches the case look-up and email tools. Configuration: model identifier as the API returned it, with the dated snapshot suffix rather than the family alias. Provider, endpoint and region. Temperature and top-p as production runs them, no seed exposed. System prompt v4 with its hash. Tool schema at annexure B, exported verbatim. Internal index as of a stated date. Input: the attachment, attached byte for byte rather than described. Trials: eleven successes in twenty, fresh sessions, none pooled with continued ones. Consequence: case belonging to another citizen retrieved and emailed to an attacker-chosen address, through two tools neither of which performs an authorisation check. Severity, in the convention agreed before the work started: the full CVSS vector, labelled CVSS-B because only base metrics were supplied, printed beside “11/20”, with the CWE mapped to the missing authorisation check, the CVE field blank because no third-party product is implicated, the EPSS field blank because there is no CVE, and both blanks explained under §16.1(ii). Closure criterion: the mitigation holds across the variant family at the rate agreed in advance, where the family is defined in the report as the set of carriers and phrasings generated from this objective, not as the single document above.
The fixes, and where they live
The look-up tool takes the case identifier from the session rather than from the model, or checks the session’s entitlement to the identifier it was given. The email tool is restricted to addresses already on the case. Retrieved documents are placed in the context with a marker that distinguishes data from instruction, and the internal index is split so that user-uploaded content is never retrieved into a session other than its own. The model output is encoded before rendering, which also closes the cross-site scripting finding from the other engagement. The system prompt stops being a secret and starts being a document that assumes it is public. Not one of those fixes is a change to the model.
The lesson the two reports teach together
The object-identifier defect in the VAPT and the tool-authorisation gap in the AI engagement are the same bug at two layers, found by two disciplines, six weeks and two purchase orders apart. Had the tool endpoints been named as API assets in the first scope, the penetration test would have found the second one too — with a deterministic proof, a lower price and a fix that landed in the same sprint. That is the whole practical argument of this page compressed into one sentence, and it is why the inventory step comes first.
Two details in that walkthrough are worth pulling out because they generalise. First, the serious finding was not a model problem at any point — it was an authorisation decision nobody made, discovered by an adversarial method and fixed by a deterministic control. Second, the injection arrived through a feature somebody shipped deliberately: parsing attachments into the index so the assistant could answer questions about them. That is the shape of most indirect prompt injection in production. It does not come through a hole. It comes through the product.
What goes wrong, and when you find out
Why do AI security engagements fail?
These are procurement and process failures rather than technical ones, which is why they are worth a table: every one of them is visible before it costs anything, and almost all of them are invisible afterwards.
| The failure | Where it surfaces | How you know | What it costs |
|---|---|---|---|
| A VAPT was bought and reported internally as “the AI has been tested” The scope named hostnames. The report is a good application test. The summary email to the board said the assistant had been security tested, and everybody stopped asking. | The first incident involving the assistant, or the first customer questionnaire with an AI section | No finding in the report mentions a prompt, a tool, a corpus or a model version, and the scope annexure names no model. | A second engagement, plus the period in which the organisation believed a question had been answered that was never asked. |
| AI red teaming was bought and the ordinary application was never tested A thorough behavioural engagement against the model and the orchestration, while the API tier underneath had an object-identifier defect that made the whole exercise unnecessary for an attacker. | The next conventional test, or an opportunist | There is no VAPT of the surrounding application inside the last twelve months, or there is one and its scope predates the assistant. | The more expensive engagement bought first, and the cheaper finding left in production. |
| The tool endpoints were in nobody’s scope The application test scoped the hostnames the users see. The AI engagement scoped the model. The endpoints the model calls sat between the two, untested by either, holding a credential with more privilege than the product needs. | Whenever somebody reads the tool schema for the first time | The asset annexure of the application test does not list them, and the AI scope describes a model rather than a system. | The most serious finding class in this architecture, unexamined by two engagements that both believed it was the other one’s. |
| A behavioural finding was scored with a bare CVSS number The report prints 8.1 with no vector, no trial count and no success rate, because the template demanded a severity and the template was built for deterministic findings. | Report review, when somebody asks what the number means | §16.2(ii) requires CVSS supplemented with EPSS and every observation mapped to a CWE and a CVE — the row has a CVSS number and the other three columns are blank or invented. | A remediation queue sorted by a number nobody can defend, and a governance artefact that fails its first serious read. |
| Closure was proved by re-running the three prompts from the finding A filter was added that catches the exact phrasings in the report. The retest passed. The variant family was never re-run. | The next engagement, usually with a near-identical finding | The retest evidence shows three inputs, not a suite, and no success rate appears next to the closure statement. | A closed finding that is not closed, and a follow-up audit under §16.3(iv) that verified the wrong thing. |
| The model changed and nothing was re-run A provider deprecated a version, or an engineer moved a configuration value to a newer model for cost reasons. Every behavioural result in the report now describes a system that is no longer running. | Quietly, and usually never — until something happens | The model identifier in the report and the model identifier in the running configuration do not match, and no change record connects them. The sharper version of the same check: the report records the identifier that was requested rather than the one the API returned, so it cannot tell you whether a floating alias moved underneath the engagement. | The entire behavioural assurance position, silently, for the price of a one-line configuration change. |
| The test environment used production tool credentials A replica assistant was stood up for testing and wired to the same service accounts as production, because that was what was configured. The exercise then acted on live records. | During the engagement, expensively | Nobody can name which credential each tool holds in the test environment. That question has one correct answer and it should be answerable in writing before the start. | Real data altered by an exercise, the escalation matrix §13.2.6 asks for invoked in earnest, and a conversation nobody wanted to have. |
| The rubric was written after the findings The operators probed, collected outputs they judged concerning, and the definition of “concerning” settled into place as the report was drafted. | The first disagreement about whether something is a finding | The rubric appears in the report and not in the test plan, and two findings in the same category are graded inconsistently. | An unfalsifiable finding set, and no baseline to measure the retest against. |
| A clean AI red team report was read as an assurance certificate The objectives were modest, the trial counts were low, nothing was elicited, and the one-page summary said the system is safe. | The next audit, or the next incident | No coverage statement, no objective list, no trial counts, no rate anywhere in the document. | A document that implies to every subsequent reader something its own method could not establish. |
| The engagement ran out of budget on tokens rather than on hours Nobody priced compute. The harness hit a provider rate limit at sixty per cent of the planned trials and the remaining objectives were dropped. | Week two | The test plan states man-hours, as §13.2.2(ii) requires, and states no trial counts and no compute ceiling. | A partial result presented as a whole one, unless the report says which objectives were abandoned — which is what §16.1(ii) asks for when it says the report should carry its exemptions, limitations and other constraints. |
The sixth row is the quiet one and the one most likely to be true of you right now. Model versions move for reasons that have nothing to do with security — a deprecation notice, a price change, a latency improvement — and they move through a configuration file rather than through a change advisory board. If nothing else on this page is adopted, adopt the practice of recording the model identifier and version alongside every behavioural result, and of treating a change to it as a change that requires the suite to be re-run. §13.1.2(iii) already classifies changes as minor or major and requires a major change to undergo an audit before implementation. Deciding in advance which side of that line a model upgrade falls on is a governance decision that takes one meeting.
Clauses worth importing when your tester is not empanelled
What should an AI red teaming contract say?
Almost every rule quoted on this page is a condition of empanelment. §2 of the 2025 audit policy guidelines states that they are binding on CERT-In empanelled auditing organisations and auditee entities covered under the relevant provisions — and does not enumerate those provisions, which is worth saying plainly rather than glossing. A testing firm that is not on CERT-In’s published list owes you none of it by default — not the test plan with man-hours, not the India-only data handling, not the report contents, not the honesty expectations. The remedy is not to avoid such firms. It is to write the protections into the engagement letter, and the list below is what to copy.
- A test plan with calendar time and man-hours, before work starts — and, for the AI half, trial counts per objective and a compute ceiling (§13.2.2(ii), extended)
- The environment named in the scope and in the report, with the model identifier and version pinned alongside it (§15.2.3(i))
- Hash values and version numbers obtained from you and printed in the report — read across to the system prompt and the tool schema (§15.2.3(ii))
- Every in-scope asset named in the report, and every asset that could not be reached named with the reason (§16.1(iii), §10.1(ii))
- Exemptions, limitations and other constraints in the report — which is where an unscoreable behavioural finding is honestly recorded (§16.1(ii))
- Critical and high findings notified during the engagement as and when found, not held for the draft (§16.3(ii), §16.4)
- Follow-up testing inside the scope and the price, with closure measured across the variant family rather than the original prompts (§9.3(iv), §13.1.1(g), §16.3(iv))
- Your data held only in India, encrypted, wiped beyond forensic recovery, with a certificate of deletion issued to you — and the same treatment for every prompt, output and trace the exercise generated (§10.3)
- Temporary privileged access revoked immediately on completion, including every service account a tool used during the exercise (§15.1.2)
- An escalation matrix on both sides, with a named stop authority and a deputy, reachable out of hours (§13.2.6)
- Unknowns marked as unknowns, and all states of security found reported — not only the failures (Annexure I items 20 and 21, empanelment terms)
- Qualitative metrics derived from a stated method rather than from the auditor’s judgment — which for this work means the rubric, in writing, before the engagement (Annexure I item 22, empanelment terms)
- Fees on predefined scope, deliverables and timelines, and not contingent on the finding count or on a closure report — a hardening, because §7’s only “must” is that the commercial arrangements be structured to maintain independence, while the contingency language is a “should not” and the predefined-scope basis is a recommendation. Import it as a hard term, but do not tell a supplier the word “never” is CERT-In’s (§7, hardened)
- The model identifier recorded as the API returns it, per trial, with a timestamp in a stated timezone — and the report to say whether a dated snapshot identifier or a floating alias was in force (not in the audit policy guidelines (nearest is AIBOM §9.3 Table 10 and §9.4.2.6))
- The variant suite delivered to you, version-controlled and re-runnable by your own team — an ownership term the instruments do not contemplate and the single most valuable thing to negotiate for (not in the instruments, so put it in the engagement letter)
Two things reach you regardless of who does the testing and regardless of what your contract says. The CERT-In Directions of 28 April 2022 apply to service providers, intermediaries, data centres, body corporate and Government organisations by their own terms — the 6-hour reporting duty at direction (ii), the 180-day rolling log obligation maintained within Indian jurisdiction at direction (iv), and the incident types at Annexure I, of which item (xx) covers artificial intelligence and machine learning and item (xviii) covers cloud computing systems. And your own contractual and regulatory duties to customers, which no testing arrangement displaces. Both are set out in the CERT-In Directions of 2022, explained, and the AI-specific reporting question — what counts as an incident affecting an AI or machine learning system, and what the 6-hour clock means when the malfunction is probabilistic — is worked through in AI incidents under the CERT-In Directions.
What buyers argue about
These arrive in real procurement reviews, usually with a date attached and a bidder in the room. In each case the answer that works cites the document rather than asserting the conclusion — and where there is no document, says so.
“Our cloud provider red teams the model before release, so we are covered.”
They tested the model. You built the system. A provider’s pre-release evaluation says nothing about what your system prompt permits, what your tools will do when called, what is in your retrieval corpus or who can write into it — and those are where almost every consequential finding in this architecture lives. §9.1(ii) of the 2025 audit policy guidelines is the relevant principle: the responsibility for maintaining an efficient and robust cyber security posture ultimately rests with the auditee organisation, not the auditor. It reads across to a supplier without much strain. Use the provider’s evaluation for what it is worth — evidence about the component — and test the assembly yourself.
“Prompt injection is unsolvable. What is the point of testing for it?”
You are not testing to eliminate it. You are testing to find out what it reaches. Treat susceptibility as a property of the architecture rather than a defect to be closed, and the useful question becomes the second one: when the model is influenced, what can it cause to happen? That question has deterministic answers — which tools exist, what authority they act under, which actions are irreversible, what the credential can touch, whether a human confirms. Excessive agency is the lever you can actually pull, and it is a design decision rather than a model property. The OWASP Top 10 for LLM Applications makes the same split, numbering prompt injection LLM01 and excessive agency LLM03 in the 2026 edition published on 4 August 2026 — the latter having moved up from LLM06 in the 2025 edition, and OWASP’s own release announcing the edition says in terms that excessive agency is “now number three”. That movement is a useful signal about where the field thinks the tractable work is.
“Can we not just add a guardrail model in front of it?”
You can, and you should measure it before you rely on it. A guardrail classifier is a control with a false-negative rate, and that rate is exactly the kind of thing this discipline is for: run the variant suite against the system with the guardrail enabled and with it disabled, and report both numbers. What a guardrail must not do is replace a deterministic control. If the tool checks the session’s entitlement, the consequence is bounded whatever gets through. If the only thing between an influenced model and an irreversible action is a second model’s opinion, you have added a probability, not a boundary.
“Our auditor says their VAPT covers the AI system.”
It might, and four questions settle it in one email. Which §6 class or classes is this, given that §6(xxiii) names AI System Audits separately from §6(iii) and §6(iv)? Which model and version is in the scope, and is it pinned? Are the tool endpoints in the asset annexure as API assets, with test accounts? And how many trials per objective, against what written rubric? §13.2.3(i)(a) already requires the type of audit to be communicated before commencement, and it names nine types of which none is an AI audit — so a bidder who cannot answer the first question is failing a requirement rather than being asked something unusual.
“Is AI red teaming mandatory in India?”
Not as a free-standing legal duty, and anybody saying otherwise is compressing several instruments into one sentence. What is true: §6(xxiii) of the 2025 audit policy guidelines names Artificial Intelligence (AI) System Audits as a class of engagement in scope of that document, and §6 frames an expectation that auditee organisations ensure a comprehensive audit covering all aspects of their ICT systems at least once a year, with additional assessments during the year as they choose. Separately and more sharply, item (xx) of Annexure I to the CERT-In Directions of 28 April 2022 makes attacks or malicious or suspicious activities affecting systems, servers, software and applications related to Artificial Intelligence and Machine Learning a mandatorily reportable class of incident, within 6 hours of noticing it — and that duty exists whether or not you ever commission a test. Beyond CERT-In, ISO/IEC 42001:2023 and, for products placed on the EU market, Regulation (EU) 2024/1689 reach you by their own routes.
“Does the EU AI Act require us to red team?”
Read the obligation you are actually under rather than the summary of it, and check the date, because the timetable has moved. Regulation (EU) 2024/1689 is the AI Act. Article 14 is the human-oversight requirement for high-risk systems, and it is the article buyers most often reach for when they ask this question — a system that must be capable of being effectively overseen by natural persons is a system whose autonomous action you had better have characterised. Then note the Digital Omnibus on AI, Regulation (EU) 2026/1744, in force 27 July 2026, which defers the stand-alone high-risk obligations for Annex III systems to 2 December 2027 and for Annex I systems to 2 August 2028. None of that is a testing mandate in terms. All of it is a reason to be able to describe, with evidence, what your system does under adversarial input.
“Do we need a CERT-In empanelled organisation to do this?”
Only if something in your own position requires it. AI System Audits is one of the classes §6 lists, which puts it in scope of a document that §2 states is binding on CERT-In empanelled auditing organisations and auditee entities covered under the relevant provisions — §2 does not enumerate those provisions, which is worth saying plainly. If a tender, a regulator or a hosting gate requires an empanelled firm, then it does. Otherwise it is a commercial choice, with two consequences worth deciding deliberately. Using an empanelled organisation brings the obligations on this page with it, including §10.6 and clause 3.2.2 of the empanelment terms, which put the report and the metadata with CERT-In within five days of completion. Not using one means none of §13.2.2(ii), §15.2.2, §10.3 or §16 reaches your tester by default, and the protections have to be written into the engagement letter instead.
“Can we look up who is capable of this on the CERT-In list?”
Only partly, and it is worth knowing what the list actually contains before relying on it. Each organisation on the published list carries a snapshot of skills and competence with seven numbered fields, and §12.1(i) says that information should be used by organisations or sectoral regulators to identify and select auditing organisations. Field 3 is “Capability to audit, category wise (add more if required)” — a Yes or No answer against categories the organisation itself lists, with no fixed taxonomy, so the categories genuinely differ between entries and some answers are No. The list therefore tells you whether a firm claims a category, not whether it is good at it. §12.2(iv) puts that work on you: check the domain of audits conducted, previous audits, methodology, tools and relevant-domain experience, and consider the alternatives, before selecting.
“Can the exercise run against production?”
Sometimes it has to, and then two clauses apply and a good deal of practice sits on top of them. §15.2.2(iv) says that in critical environments where availability is a top priority, testing should be conducted in a passive manner to avoid any potential downtime or service disruption. §13.2.6 is one sentence: there should be a well-defined escalation matrix for both the auditee organisation and the auditing organisation, for addressing any problem encountered during the audit process, which should be shared with the respective authorities. It names no stop condition, no stop authority, no deputy and no out-of-hours contact — those four are our reading of what that matrix has to contain for a test of this kind, and they belong in the engagement letter rather than being attributed to the clause. The specific AI wrinkle is credentials rather than load. Decide, tool by tool, what the test environment’s credential can reach, and keep the answer in writing. §15.1.2 requires temporary privileged access to be revoked immediately on completion, which is easy to satisfy for a human tester and easy to forget for a service account wired into an orchestrator.
“Our system integrator built it and will test it as part of the contract.”
That is the arrangement §5(xvii) defines as a conflict of interest: it expressly covers the same entity being involved in both the implementation of controls and their subsequent audit, or conducting audits while also being responsible for remediation, and says such overlaps create inherent conflicts that undermine the independence of the audit process. The integrator has real work to do on the other side of that line — §13.1.3(iii) says the audit of a critical application should include dynamic and static application security testing, and that when procuring the application the auditee organisation should require the software developer or system integrator supplying it to perform static application security testing, with that requirement clearly stated in the RFPs and tenders — and in this architecture the equivalent build-side duty is maintaining the tool schema and the least-privilege position on each tool credential. Build-side rigour and independent assurance are both necessary and they are not interchangeable.
“How is this different from the AI evaluations our data science team already runs?”
Direction and intent. An evaluation measures average-case quality against a benchmark: does the system answer correctly, helpfully and consistently on inputs that resemble real use. A red team measures worst-case behaviour against an adversary who is trying: what can be elicited by somebody who wants a particular failure and will iterate until they get it. The two use overlapping machinery — a harness, a dataset, a grading step — and they are not substitutes, because a system can score well on a benchmark and be trivially steerable by an input nobody put in the benchmark. Keep both, and keep the adversarial suite separate so that a benchmark improvement cannot be mistaken for a security improvement.
“The quote is a third of the price of the others. What are we losing?”
Three things a total conceals, and all three are visible in a test plan. Trial counts, because a one-shot probe per objective is an anecdote and costs almost nothing. Rubric quality, because grading against a written rubric is expensive and grading against a vibe is free. And coverage of the tool layer, because probing a model in isolation is much cheaper than instrumenting a system with its tools connected. §13.2.2(ii) requires the test plan to state both calendar time and man-hours. Ask for trial counts and the rubric alongside them, and the gap between two prices usually explains itself in about ten minutes.
We deliver CERT-In audits
We deliver CERT-In audits, end to end, with our CERT-In empanelled partners. See how a CERT-In audit runs with us.
On an AI engagement most of the calendar goes by before anybody tests anything. The inventory the scope has to be derived from under §13.1.1(c) gets built, so that the models, prompts, tools and corpora are assets with owners. The tool schema gets written, because it frequently does not exist as a document until somebody asks for it. Every tool credential gets a least-privilege decision. The rubric is written and the trial counts are agreed. An environment is stood up whose tools do not hold production credentials. Afterwards the remediation cycle runs so the follow-up audit finds closed findings, and the variant suite is kept so the result survives the next model upgrade.
For the conventional half, see penetration testing and VAPT. For the governance half that §6(xxiii) also names — ethical alignment, transparency and data integrity — see ISO/IEC 42001, how it integrates with ISO 27001 and how it lines up against the EU AI Act.
Frequently Asked Questions
What is AI red teaming?
What is the difference between AI red teaming and VAPT?
Does a VAPT cover an AI chatbot?
Is AI red teaming a penetration test?
What is prompt injection, and is it a vulnerability?
What is excessive agency?
How do you reproduce an AI red team finding?
How do you score the severity of a prompt injection finding?
Does an AI red team finding get a CVE?
Should we do the VAPT or the AI red team first?
How often should we red team an AI system?
Does CERT-In require an AI audit?
Is an AI incident reportable to CERT-In?
Does ISO/IEC 42001 require AI red teaming?
Does the EU AI Act require AI red teaming?
Can our cloud or model provider’s own testing replace ours?
What should an AI red team report contain?
How much does AI red teaming cost, and how long does it take?
What is the difference between AI red teaming and AI evaluation?
Does “red teaming” mean the same thing in a CERT-In document as it does in an AI context?
Can a VAPT and an AI red team be bought as one engagement?
How many trials should an AI red team run per objective?
Which standard or taxonomy should an AI red team work from?
Does every AI system need an AI red team engagement?
Do these CERT-In clauses apply if our tester is not on the CERT-In panel?
The primary sources, named and linked
Every rule and every definition on this page is drawn from one of the following, and each is linked so you can read the clause rather than our paraphrase. Section numbers in the form §16.2(ii) refer to the Comprehensive Cyber Security Audit Policy Guidelines unless another document is named.
Every §-locator on this page was checked against the instruments below on 22 September 2026, and every URL below was fetched on the same date with its status code, content-type and byte size recorded. The second half of that matters and is usually skipped: cert-in.org.in answers 200 with a 938-byte HTML page for unknown paths under /PDF/, so a status code alone proves nothing about whether a document exists. One set of links behaves differently and is flagged in place: eur-lex.europa.eu answered an automated fetch of both ELI paths with HTTP 202 and a zero-byte body, so the two EU instruments are cited by their official identifiers and linked at their official locations without this page claiming to have machine-read them.
Several statements on this page are negative — that the 2025 audit policy guidelines contain no occurrence of “machine learning”, “LLM”, “large language”, “generative”, “agent” or “training data”, that every occurrence of “prompt” in them is the adverb “promptly”, that “model” occurs three times and never means an AI model, that “adversarial” occurs once, inside §6(xxiii), and that no AI audit type appears among the nine named at §13.2.3(i)(a). Each was established by full-text search of the official PDF on 22 September 2026, after confirming the download was 553,513 bytes of application/pdf. Absence is a claim like any other and it needs the same work.
A negative established in one document is a claim about that document, so where this page says a subject is not addressed it names the documents that were searched rather than the estate. Five were searched on 22 September 2026, each downloaded from cert-in.org.in with its content-type and byte size confirmed and converted with a layout-preserving text extraction: the audit policy guidelines (553,513 bytes), the empanelment terms and conditions (138,356), the 2018 auditor guidelines (307,912), the secure application design guidelines (517,135) and the BOM technical guidelines (1,654,189). The table in the instrument section records what each one returned, including the one document in which the model-identity vocabulary does exist. Nothing on this page asserts a negative across CERT-In’s whole published estate, because nobody searched it.
Two documents the guidelines name are deliberately absent from the list because they cannot be retrieved: the “Method of verifications to compliance with CERT-In Directions issued on 28.04.2022” cited at §15.2.1(iv), and the Deter and Punish Framework cited at §11(iv) and §19. As at 22 September 2026, both cert-in.org.in paths printed in the guidelines served the CERT-In home page rather than a document. They are named here as the guidelines name them, and not linked.
- Comprehensive Cyber Security Audit Policy Guidelines Indian Computer Emergency Response Team (CERT-In), MeitY · Version 1.0, 25 July 2025 · 69 pages, 20 sections · relied on for §2, §5(xvii), the definitions restated at §5(xxi)–(xxiii) and §5(xxiv), §6 and its definitions at (iii), (iv), (xvi), (xviii), (xxiii) and (xxvi), §7, §8(i)–(ii), §9.1(ii), §9.3(iii)–(iv), §9.5(iii), §10.1(i)–(ii), §10.3, §10.6, §12.1(i), §12.2(iv), §13.1.1, §13.1.2(i)–(iv), §13.1.3(iii), §13.1.7(i)(a), §13.2.2(ii), §13.2.3(i)(a), §13.2.3(ii)(b)–(c), §13.2.6, §13.2.7(ii), §15.1.1(i)–(ii), §15.1.2, §15.2.2(iii)–(iv), §15.2.3(i)–(ii), §15.2.5(iii), §16.1(ii)–(iii), §16.2(i)–(ii), §16.3(ii)–(v) and §16.4 · also the source of this page’s negative findings, established by full-text search of the PDF · two modality notes this page keeps: §7’s only “must” is that commercial arrangements be structured to maintain independence, the contingency language being a “should not” and the predefined-scope basis a recommendation. And §15.1.1(ii) is a “should refrain”, limited to unusual or major network changes, not a change freeze
- Directions under sub-section (6) of section 70B of the Information Technology Act, 2000 CERT-In, MeitY · No. 20(3)/2022-CERT-In · 28 April 2022 · 8 pages · relied on for direction (ii) on the 6-hour reporting obligation and its channels, direction (iv) on the 180-day rolling log obligation maintained within Indian jurisdiction, and Annexure I items (xviii) cloud and (xx) Artificial Intelligence and Machine Learning
- FAQs on Cyber Security Directions of 28.04.2022 CERT-In, MeitY · May 2022 · relied on for the Annexure-I explanation of incident type (xx) · the document states of itself that it “is not a legal document and in no way whatsoever replaces, amends or alters any part of the IT Act, 2000 and/or the … CERT-In Rules, 2013”, and that it is an evolving document whose version may change — quoted here because any reliance on it should carry that caveat
- Guidelines for CERT-In Empanelled Information Security Auditing Organizations Indian Computer Emergency Response Team (CERT-In), MeitY · Version 3.0, May 2018 · 6 pages · relied on, under Technical, for the verification team (Red Team) that verifies the work performed by the audit team (White Team), and for industry standard methodologies rather than solely tools-based testing
- Empanelment of Information Security Auditing Organisations — Terms and Conditions for Empanelment Indian Computer Emergency Response Team (CERT-In), MeitY · Version 7.2, April 2024 · 10 pages · relied on for clauses 3.1.7 and 3.2.2, and for Annexure I items 20, 21 and 22 — unknowns marked as unknowns, all states of security found reported, and qualitative metrics based on a mathematical formula rather than on the feelings of the auditor
- Technical Guidelines on SBOM, QBOM & CBOM, AIBOM and HBOM Indian Computer Emergency Response Team (CERT-In), MeitY · Version 2.0, dated 09.07.2025 · 66 pages · listed because §6(xxvi) of the audit policy guidelines names AIBOM auditing as a class, and relied on for §9.3, Table 10, “Minimum Elements of AIBOM”, which is the one place in the CERT-In estate this page found that names an AI model by identity and version — Model Name, Model Version, Model Type, Model Developer, Model Licensing Information, Software Dependencies, ML Models and Algorithms, Model Performance Metrics, Data Source, Data Sets, Hardware, Security Requirements, Input, Output and Intended Usage — together with §9.4.2.4 on reproducibility, §9.4.2.6 on model lineage and the §9.4.1 recommendations · note that a full-text search of the 2025 audit policy guidelines finds no citation of this document, so reading the two together is a reasonable inference and not a cross-reference the guidelines make
- Empanelled Information Security Auditing Organisations by CERT-In — the snapshot information Indian Computer Emergency Response Team (CERT-In), MeitY · the published list §10.2(vi), §12.1(i) and §12.3(iii) refer to · 997 pages at the time of checking · relied on for the anatomy of the snapshot and for the fact that field 3, capability by category, carries no fixed taxonomy
- Guidelines for Indian Government Websites and Apps (GIGW 3.0) — the manual National Informatics Centre with STQC and CERT-In, MeitY · the PDF linked from guidelines.india.gov.in as “Download GIGW Manual 3.0” · relied on for guideline 5.3.1, the audit clearance certificate required before hosting in production · cite the manual for the numbering: the live attributes page prints “5.3.2” twice and mis-numbers the policies statement
- CERT-In SAMVAAD 2026 — Conference Proceedings Indian Computer Emergency Response Team (CERT-In), MeitY · 2nd edition, 27–29 April 2026, BITS Pilani K. K. Birla Goa Campus · relied on for the remit of the working group on Indian Standards for Cybersecurity Auditing, which is stated to aim at standardised audit reporting formats, RFP templates and scope templates for audit types including emerging technologies such as artificial intelligence
- CERT-In SAMVAAD 2026 concludes — press release Press Information Bureau, Ministry of Electronics & IT, Government of India · Release ID 2256581 · 29 April 2026 · relied on only for the list of topics presented, which names AI driven cyber risks and AI driven red teaming methodologies · read the phrase precisely: it describes red teaming driven by AI, which is not the same subject as red teaming of AI
- OWASP Top 10 for LLM Applications, 2026 edition OWASP GenAI Security Project · not a government source, and not a standard — §8(ii) of the audit policy guidelines states that limited lists such as the OWASP Top 10, SANS Top 25 and similar should not be considered standards or references for audits · relied on for prompt injection at LLM01 and excessive agency at LLM03 in the 2026 edition published 4 August 2026, the latter having moved up from LLM06 in the 2025 edition · this is the edition page. The document’s own cover carries the date 4 August 2026 and the resource card on the page dates the posting 3 August 2026 · the numbering moves between editions, so cite the edition with the item
- OWASP GenAI Security Project unveils the 2026 Top 10 for LLM Applications — release OWASP GenAI Security Project · 1 September 2026 · relied on for the one sentence that settles the renumbering in the project’s own words: excessive agency is “now number three”
- OWASP LLMRisks Archive OWASP GenAI Security Project · linked only so a reader can see why a stale citation misleads · this page indexes the 2025 entries, down to LLM06:2025 Excessive Agency, and carries no 2026 numbering · it is NOT the source for anything this page says about the 2026 edition
- OWASP GenAI Red Teaming Guide OWASP GenAI Security Project · resource page dated 22 January 2025 · a method document rather than a list, which is why it survives the §8(ii) objection that disposes of a top-ten list · named here because this page tells buyers to demand a method document for the AI half and §8(ii) names no AI equivalent · its own summary describes four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis
- NIST AI 100-2 E2025 — Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations National Institute of Standards and Technology, US Department of Commerce · the PDF running footer reads “NIST AI 100-2e2025 March 2025” · not a government-of-India source and not binding in India · listed because it is the freely retrievable attack-class taxonomy an AI finding can be classified against, which §8(ii) requires of an audit and does not supply for this subject · relied on for the generative-AI attack taxonomy and its identifiers, read directly from the PDF: Data Poisoning NISTAML.013, Indirect Prompt Injection NISTAML.015, Prompt Injection NISTAML.018, Prompt Extraction NISTAML.035, Leaking information from user interactions NISTAML.036 and Compromising connected resources NISTAML.039 · the full text is served as a PDF from nvlpubs.nist.gov
- MITRE ATLAS (Adversarial Threat Landscape for AI Systems) — the published data repository MITRE · not a government source · the adversarial-machine-learning threat matrix, listed as the second taxonomy worth naming in a scope of work because it carves the space by tactic and technique rather than by attack class · the expansion above and the description “a public knowledge base of adversary TTPs targeting AI systems” are MITRE’s own words, read from this repository · retrievability note, and the reason this entry links the repository rather than the project site: atlas.mitre.org answers 200 with a 3,695-byte JavaScript shell whose only server-rendered text is the title, and its sub-paths answer 404 to an automated fetch, so a 200 from that host proves the site is up and nothing else · this page prints no ATLAS tactic or technique identifier, because the published dataset carries a deprecation notice on its own first line and an identifier taken from it would need re-checking before it appeared in a report
- NIST AI 600-1 — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile National Institute of Standards and Technology · July 2024 · a voluntary US profile, not adopted by any Indian instrument and never a compliance obligation for an Indian entity · listed because it is the retrievable international document that treats AI red-teaming as a named practice and defines it — “a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled environment and in collaboration with system developers” · it complements §6(xxiii), which names a class of engagement and leaves the method entirely open
- NIST AI Risk Management Framework (AI RMF 1.0), NIST AI 100-1 National Institute of Standards and Technology · the PDF’s own title is “Artificial Intelligence Risk Management Framework (AI RMF 1.0)”, published under the number NIST AI 100-1 · not a government-of-India source and not binding in India, and the framework describes itself as voluntary · listed as the freely retrievable governance framework for the half of §6(xxiii) a testing engagement does not reach — ethical alignment, transparency and data integrity · the framework document is served as a PDF from nvlpubs.nist.gov
- OWASP Application Security Verification Standard Open Worldwide Application Security Project · not a government source · named at §8(ii) as one of the comprehensive alternatives to limited lists, and listed here because this page tells buyers to demand a named control standard at a stated version and level
- CVSS v4.0 Specification Document Forum of Incident Response and Security Teams (FIRST) · document version 1.2 · relied on for the four metric groups — Base, Threat, Environmental and Supplemental — and for the nomenclature note that a score derived from base metrics alone is CVSS-B · the qualitative severity bands used across this hub are FIRST’s, published in the specification, and are not a CERT-In classification
- Exploit Prediction Scoring System (EPSS) Forum of Incident Response and Security Teams (FIRST) · relied on for the definition — a data-driven machine-learning model estimating the probability that a published CVE will be exploited in the wild in the next 30 days, published daily per CVE
- EPSS frequently asked questions Forum of Incident Response and Security Teams (FIRST) · relied on for two statements this page depends on: that EPSS is not a complete risk score and does not account for context, compensating controls or anything specific to an organisation’s environment, and that scoring vulnerabilities with no CVE ID at all is a longer-term direction rather than current capability
- About CWE MITRE · relied on for the definition of a weakness as a condition in a software, firmware, hardware or service component that, under certain circumstances, could contribute to the introduction of vulnerabilities — which is the reason some behavioural findings map to a CWE and some do not
- Regulation (EU) 2024/1689 — the Artificial Intelligence Act Official Journal of the European Union · relied on for Article 14, human oversight for high-risk systems · note on retrievability: eur-lex.europa.eu answered an automated fetch of this ELI path with HTTP 202 and a zero-byte body on the date below, so this page cites the instrument by its official identifier and links its official location without claiming to have machine-read it · the dates this page states for this instrument and for the one below are taken from the instruments as published and could not be machine-read from the ELI paths, so open them in a browser before relying on a date in a compliance plan
- Regulation (EU) 2026/1744 — the Digital Omnibus on AI Official Journal of the European Union · in force 27 July 2026 · relied on for the deferral of the stand-alone high-risk obligations, to 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems · a separate instrument from Regulation (EU) 2024/1689, never a new name for it · the same retrievability note applies: those three dates are stated from the instrument as published, not from a machine-read copy of it, and the ELI path answered an automated fetch with HTTP 202 and a zero-byte body
ISO/IEC 42001:2023 and ISO/IEC 23894:2023 are cited on this page by their numbers and titles only. They are copyrighted standards sold by ISO and its national members, they are not freely retrievable — iso.org answered an automated fetch with HTTP 403 on 22 September 2026 — and nothing on this page quotes or paraphrases their text. Where a freely retrievable equivalent exists it is listed above and linked: the NIST AI Risk Management Framework for the governance half, and NIST AI 100-2e2025 for the attack taxonomy.
Related reading: what VAPT is, the types of VAPT, how to scope a VAPT, how to read a VAPT report, VAPT against penetration testing against red teaming, retesting and closure, what CERT-In is, the CERT-In Directions of 2022, what a CERT-In empanelled auditor may and may not do, what AMBAK is, what a safe-to-host certificate is, what AI red teaming is, red teaming an LLM application, AI incidents under the CERT-In Directions, what cloud security is, cloud security audits in India, API security testing, AI risk assessment against AI impact assessment, ISO 42001 against the EU AI Act, and integrating ISO 42001 with ISO 27001.
Related Reading
What AI Red Teaming Is
Adversarial testing of AI systems: what it targets, how it differs from a penetration test, and what it produces.
Read moreWhat VAPT Actually Is
Vulnerability assessment and penetration testing are two activities. Why India bundles them, and what a real one covers.
Read moreRed Teaming an LLM Application
Scoping, attack classes, evidence for a non-deterministic system, severity and retest.
Read moreThe Types of VAPT
Web, network, mobile, API, cloud configuration and wireless — what each can and cannot find.
Read moreAI Incidents Are Reportable to CERT-In
Annexure I item (xx) makes attacks on AI and ML systems mandatorily reportable — within six hours.
Read moreVAPT vs Pen Test vs Red Team
Four activities that get quoted against each other, and what actually distinguishes them.
Read more