Red-teaming used to be a research-lab activity. In 2026, with boards, auditors and regulators asking pointed questions about AI, it has become a sensible pre-launch gate for any customer-facing AI feature in a regulated industry. Risk and compliance teams in banking, insurance and healthcare increasingly expect evidence that a system was tested adversarially before it met customers. Here is a practical playbook to run before your AI system ships.
What you are actually testing for
- Prompt injection. Can a user, or a piece of retrieved content, override the system prompt? Direct and indirect injection both matter; indirect injection through poisoned documents, emails or web pages is under-tested.
- Data exfiltration. Can the model be coaxed into revealing its system prompt, internal data or other customers' inputs?
- Jailbreaks. Can it be made to break its constraints — authorising refunds it shouldn't, producing content it shouldn't, giving advice it is scoped not to give?
- Authorisation bypass. Can it be talked into invoking tools or retrieving data the user isn't entitled to?
- Hallucination under pressure. Does it invent answers when given leading questions in domains where invention is costly?
- Toxicity, bias and culturally loaded outputs. Especially relevant for Indian audiences across religion, caste, region, language and politics.
The OWASP Top 10 for LLM Applications is a useful checklist to map these against; it covers prompt injection, sensitive information disclosure, excessive agency and more.
The methodology that works
- Threat model first. Who is the realistic attacker — an abusive customer, a competitor probing, a malicious insider, a journalist testing limits? What can the system reach: which tools, which data, which actions? Each answer shifts priorities.
- Mixed-method probing. Manual, creative attempts by testers with domain experience; automated attack tools such as Garak or PyRIT; and LLM-generated adversarial inputs at volume. No single method finds everything.
- Test in every language you serve. Guardrails tuned on English often behave differently with Hindi, Tamil or code-mixed Hinglish input. Probe each one.
- Per-finding reproduction. Every finding gets a minimal reproducer. "It refused last time" doesn't close a finding.
- Mitigation and re-test. Most mitigations are partial. Re-test under the same conditions and across closely related variants.
Who should be on the red team
A mix: an engineer who knows how the system is built, a security tester, and someone from the business domain — a claims handler, a credit officer, a clinician — who knows what a harmful answer looks like in practice. Include people who speak the languages your users speak. Keep at least one tester independent of the team that built the system; builders tend to test the paths they expect.
Rating severity
| Severity | Example | Expected response |
|---|---|---|
| Critical | The agent triggers a payment or exposes another customer's data | Block launch; fix and re-test |
| High | A reliable jailbreak that produces prohibited advice | Fix before launch or disable the feature |
| Medium | System prompt leakage; inconsistent refusals | Fix in the next release and monitor |
| Low | Tone or formatting issues under adversarial input | Backlog |
What teams tend to underestimate
- Indirect prompt injection through retrieved documents is more common, and less defended against, than direct injection.
- Tool-call bypass is easier than expected. Agents over-trust their own reasoning about whether to call a tool, so enforce permissions in code, outside the model.
- Cultural and political failures carry higher stakes in India than English-centric red-team libraries assume. Build a local probe set.
- Customer-facing hallucination is rarely about exotic inputs. It comes from confidently worded everyday questions the model has no source for.
What good output looks like
A red-team engagement should produce a categorised finding list with severity and reproducer, a remediation plan, a regression suite that runs on every model or prompt change, and an executive summary an auditor can review. Anything less is a workshop, not a deliverable. Mapping findings to a recognised framework such as the NIST AI Risk Management Framework helps risk teams slot them into existing processes, and where personal data is involved, link each finding to your obligations under India's Digital Personal Data Protection Act, 2023.
A worked example
Consider a hypothetical insurance claims assistant that reads uploaded documents and can look up claim status. A useful probe set would include a claim PDF with hidden instructions to approve the claim; requests for another policyholder's status using a guessed claim number; leading questions about cover the policy does not include; and abusive or politically charged messages in Hindi and English. The fixes for the first two belong in code — treating retrieved content as untrusted data and checking entitlement before every tool call — not in a longer system prompt.
How often to re-run
After every major change to the model, prompts, tools or retrieval corpus. Quarterly at minimum even if nothing changes, because attacker techniques evolve faster than your system. And continuously in CI for known regression cases.
FAQ
Can we red-team a system built on a third-party model?
Yes, and you should. The model provider tests the base model; you are testing your prompts, retrieval, tools and permissions, which exist only in your deployment.
Is automated scanning enough?
No. Automated tools are good for volume and regression; many of the serious findings come from people who understand the domain and think like an attacker.
How we approach this at Velura Labs
Our AI strategy & roadmap and custom LLM applications engagements include red-team passes for regulated deployments; add agentic systems work when the tool-call surface is large. Read our guardrails playbook for the mitigation side and our healthcare compliance piece for a worked example in a regulated sector. Talk to us before your auditor asks for the red-team report — acting early is much cheaper than reacting.
Our clients for this span US tech hubs (San Francisco, Seattle, Austin, New York), European markets (Paris, Milan, Rome), the Middle East (Dubai, Riyadh, Abu Dhabi) and India. Start a conversation from anywhere.