AI application security

1. Map the AI system and its trust boundaries

Start with a data-flow diagram. Include the user interface, orchestration service, model provider or hosted endpoint, system instructions, retrieved content, vector database, tools, agent memory, AI gateway, logs, and downstream applications.

For each component, record what data it receives, whose identity it uses, what it can disclose or change, where secrets reside, and which party operates it. Mark boundaries between users, tenants, services, providers, and tools. Identify whether content is trusted, user-controlled, or externally sourced.

2. LLM security

• Prompt injection: malicious or conflicting instructions in user input or external content try to change the model's behavior. Separate instructions from data where supported, constrain the task, limit tools and accessible data, and validate actions outside the model. Prompt wording alone is not a reliable security boundary.
• Jailbreaking: attempts to bypass intended model behavior or safeguards. Evaluate likely misuse cases and layered controls, but do not assume any prompt filter can guarantee refusal. Restrict harmful capabilities at the application and infrastructure layers.
• System prompt attacks: users may try to elicit hidden instructions or infer internal configuration. Avoid placing credentials, private data, or security-critical secrets in prompts. Assume prompts may be exposed; enforce access controls in services.
• Sensitive information disclosure: outputs may reveal personal data, secrets, training data, or context from another user. Minimize data sent to models, enforce retrieval permissions, redact sensitive values where appropriate, and test cross-user leakage.
• Insecure output handling: generated text may contain unsafe HTML, SQL, shell commands, URLs, or structured fields. Encode for the output context, parse against strict schemas, use parameterized database access, and require explicit policy checks before actions.
• Excessive agency: an application gives a model more tools, data, or autonomy than the task needs. Reduce capabilities, require confirmation for consequential actions, set limits, and make actions reversible where possible.
• Model supply-chain risks: models, adapters, datasets, plugins, and inference software may be compromised, poisoned, mislabeled, or unexpectedly changed. Track provenance and versions, use trusted sources, scan artifacts, review licenses, and evaluate updates before deployment.

Model behavior can vary with updates, context, and sampling. Define expected behavior through evaluations and product controls rather than relying on a single demonstration.

3. RAG security

Retrieval-augmented generation (RAG) searches a corpus and supplies relevant documents to a model. Retrieval can expose information even when the model itself is behaving as designed; the application must preserve document permissions end to end.

• Document poisoning: attacker-controlled or compromised documents influence answers or attempt to redirect actions. Authenticate sources, review ingestion, track provenance, and quarantine suspicious content.
• Retrieval manipulation: irrelevant or crafted content is made more likely to rank highly. Use robust ingestion and ranking, filter by trusted metadata, monitor retrieval quality, and test adversarially with representative corpora.
• Data leakage: retrieved passages may contain confidential data that appears in answers, logs, or caches. Minimize indexed data, classify it, apply retention rules, and test prompts and output paths for leakage.
• Access-control failures: authorization must be checked for each document at retrieval time, not only when a user signs in. Enforce tenant and document ACL filters in a trusted service and deny by default if policy evaluation fails.
• Cross-user data exposure: shared indexes, caches, conversation history, or asynchronous jobs can mix tenants. Partition or rigorously filter by tenant, bind cache keys to identity and permissions, and test concurrent sessions.
• Embedding and vector security: embeddings can encode sensitive information, and vector stores may have weak isolation, metadata filtering, or deletion semantics. Restrict direct access, protect backups, validate metadata filters, control exports, and verify deletion across indexes and replicas.

Do not treat embeddings as anonymous or harmless by default. Apply data governance to source documents, embeddings, metadata, retrieved chunks, and generated answers.

4. AI agent security

An agent combines a model with tools and a loop that can choose actions. This creates a path from untrusted content to real effects. Define the agent's intended scope, then ensure each tool independently checks authority and validates arguments.

• Tool abuse and unsafe execution: an agent can call a tool with dangerous or malformed arguments. Use narrow typed interfaces, strict validation, allowlisted operations, sandboxing, timeouts, and human approval for high-impact actions. Never pass generated text directly to a shell or privileged interpreter.
• Excessive permissions: broad service accounts let a compromised workflow affect too many resources. Give each agent and tool minimal, task-specific permissions and scope them to the current user and resource.
• Agent hijacking and indirect prompt injection: hostile content in email, documents, web pages, tickets, or tool results can steer later actions. Treat tool results as untrusted data, constrain action selection, and require independent checks before data leaves or state changes.
• Identity propagation: downstream services should know whether an action is performed for a user, a service, or an agent. Propagate a verifiable, least-privilege identity and delegation context; do not silently substitute a powerful shared credential.
• Agent-to-agent trust: one agent's output is not proof of another agent's authorization or correctness. Authenticate agents, define message schemas and provenance, minimize delegated authority, and validate each handoff.

Use a permissioned action broker between the model and tools. The broker can enforce policy, record decisions, cap operations, and require approval without asking the model to police itself.

5. AI infrastructure

• Model APIs and endpoints: authenticate callers, authorize model and feature use, set quotas and payload limits, protect transport, and separate development and production. Treat provider responses and errors as external input.
• Vector databases: isolate tenants, restrict network and administrative access, protect snapshots and backups, validate filters, and monitor bulk export or unusual query patterns.
• MCP: treat Model Context Protocol servers and tools as privileged integrations. Verify server identity and provenance, expose only necessary tools, review requested scopes, validate inputs and outputs, and make user consent and approval meaningful. Keep credentials scoped to the integration and avoid automatically trusting tool descriptions or returned content.
• AI gateways: centralize authentication, model routing, quotas, policy enforcement, logging, and data-loss controls where useful. Ensure the gateway cannot be bypassed through alternate endpoints and avoid collecting more prompt data than needed.
• Secrets: store provider keys and service credentials in managed secret storage, rotate them, scope them narrowly, and prevent them from entering prompts, source code, client bundles, telemetry, or model-visible context.
• Authentication and authorization: authenticate users and services separately, enforce tenant and resource checks in backend services, and pass only scoped credentials to tools. Model responses and prompt instructions must never establish identity or permissions.

6. AI red teaming

AI red teaming is a structured, authorized assessment of how an AI-enabled product behaves under misuse, adversarial inputs, and unexpected combinations of features. Scope the system, test accounts, datasets, tools, provider endpoints, and allowed impact before testing.

1. Define product goals, security and safety boundaries, assets, and prohibited outcomes.
2. Map attack surfaces: prompts, uploaded and retrieved content, tools, APIs, identity, memory, and integrations.
3. Create scenarios for direct and indirect prompt injection, data disclosure, authorization bypass, unsafe output, tool misuse, resource abuse, and supply-chain changes.
4. Test with synthetic data and least-privilege accounts in an isolated environment; include multi-user and tenant-boundary cases.
5. Record reproducible steps, expected control, actual impact, affected versions, and evidence with sensitive content removed.
6. Prioritize and fix findings, then add regression evaluations and verify the mitigations.

Measure both attack success and safe task completion. A system that refuses everything may block attacks while failing its purpose; a system that is helpful without authority boundaries may create severe risk.

7. AI security engineering

Convert red-team findings into controls in the product architecture, not only prompt edits. Keep security properties enforceable even when a model is confused, manipulated, unavailable, or updated.

• Minimize model context and tool capability; separate instructions from untrusted content where the platform supports it.
• Enforce authorization, quotas, and policy in ordinary backend code at every data and action boundary.
• Validate model output with strict schemas and safe rendering; require confirmation for consequential operations.
• Use tenant-aware retrieval and caching, data minimization, retention limits, and protected observability.
• Maintain model, prompt, dataset, tool, and integration inventories with owners and version history.
• Evaluate model and pipeline changes against a versioned suite of security and quality cases before release.
• Monitor abuse, policy failures, anomalous tool use, and service costs while protecting user privacy.
• Prepare incident response for exposed data, compromised keys, poisoned sources, provider changes, and unsafe agent actions.

Document residual risks and operational fallbacks. A graceful fallback should preserve access controls and avoid silently granting broader permissions when a model or policy service is unavailable.

Practice and recap

Free learning path and practice

Learn the AI-security landscape

• OWASP GenAI Security Project and its Top 10 for LLM Applications — a free risk map for LLM-enabled products.
• NIST AI Risk Management Framework — voluntary guidance for governing, mapping, measuring, and managing AI risk.
• MITRE ATLAS — adversary tactics and techniques for AI-enabled systems.
• Cloud Security Alliance AI security resources — current AI governance and security material; MAESTRO (Multi-Agent Environment, Security, Threat, Risk, and Outcome) is a seven-layer threat-modeling approach for agentic systems.

Practice safely

• TryHackMe rooms catalogue and HTB Academy catalogue — search for current AI, LLM, and agent-security labs. These catalogs change quickly; if no dedicated room is available, adapt the guide's RAG and agent exercises to a local demo with synthetic data.
• PortSwigger Web Security Academy and Hacker101 CTF — free, intentionally scoped web-security labs for the application-security foundations that AI products also depend on; these are not dedicated AI test environments.
• Microsoft Learn: AI red teaming — free guidance on planning and performing AI red-team assessments.

Watch AI-security playlists

• IBM Technology: AI Security Explained — accessible security primers from a channel with approximately 1.81M subscribers; use as an introduction rather than a hands-on LLM testing course.
• IBM Technology: AI Agents Explained — concepts to help identify agent tools, permissions, and trust boundaries.

Subscriber counts are approximate and change over time; video view counts vary. For technical testing, pair these high-audience introductions with the OWASP GenAI Security Project and hands-on exercises above.

AI threat frameworks and laws are not interchangeable. The standards and compliance guide maps AI risk frameworks, management standards, and regulation.