An LLM feature is rarely just a model endpoint. It is an application that accepts untrusted input, retrieves data, builds prompts, invokes tools, calls APIs, stores conversation state and renders generated output. The security boundary therefore sits across code, identity, data and workflow—not inside the model alone.

Traditional application testing still matters: authentication, authorization, injection, data exposure, SSRF, insecure integrations and business-logic abuse do not disappear when a model is added. AI-specific behaviour creates additional paths, including direct and indirect prompt injection, unsafe output handling, retrieval poisoning, excessive agency and unbounded consumption.

A useful assessment models what the AI-enabled system is allowed to know, decide and do for each user and tenant. It then tests whether untrusted content can change those permissions or cross a trust boundary. The objective is not to make the model say something surprising; it is to prove whether the surrounding application can produce an unauthorised disclosure or action.

Identify the real AI application before choosing test cases

Start with the complete request path. A user message may pass through moderation, prompt templates, retrieval, memory, orchestration and one or more tools before a response is returned. Each stage has a different trust level and may use a different identity. Missing one transition can make a test look successful while the actual privileged path remains unexamined.

The highest-risk designs often combine broad service credentials with weak separation between instruction and data. A retrieved document, website, email or support ticket may be treated as context even though an attacker can influence it. An agent may then convert that context into a tool call with more authority than the originating user.

  • The model or agent can call tools that read, write, approve, send, purchase, refund or administer.
  • Retrieved documents, web content or user uploads can influence system behaviour.
  • Conversation memory or vector stores contain data from multiple users or tenants.
  • A shared service account performs actions without preserving the initiating user’s authority.
  • Generated output reaches HTML, SQL, code, shell, templates or downstream automation.

Map identities and authority through the agent path

List the human user, application session, model provider, orchestrator, service account, tool credential, data source and downstream API. For every hop, record whose authority is being used and which policy makes the final decision. A non-human identity should not silently inherit more access than the user who initiated the request.

Build test pairs across roles and tenants. The same prompt should be exercised with different ownership, subscription and lifecycle states. When the agent retrieves or acts, compare the result with the policy enforced by the underlying system, not with what the model claims happened.

  • Tool calls preserve user, tenant and purpose context through every service.
  • Service credentials are scoped to the minimum operations and data domains required.
  • Approval gates are enforced by application code rather than natural-language instructions.
  • Logs correlate the user request, retrieved context, model decision and tool outcome.

A valid proof identifies the initiating actor, effective service identity, attempted object or action, policy decision and authoritative final state. A generated confirmation message is not proof that an action occurred.

Test direct and indirect prompt injection as trust-boundary failures

Direct injection begins with attacker-controlled input to the model. Indirect injection enters through content the application retrieves or summarises. In both cases, the important question is whether untrusted content can override a higher-trust instruction or influence an action outside the user’s authority.

Vary the location and encoding of instructions, but avoid treating jailbreak entertainment as the end goal. Attempt bounded outcomes: reveal protected context, select an unauthorised record, change a tool parameter, suppress a required approval, or make the application send data to an untrusted destination.

  • Retrieved content is labelled and handled as data, not privileged instruction.
  • Tool parameters are generated from validated application state and constrained schemas.
  • Sensitive system context is not relied upon as a secret control.
  • High-impact actions require deterministic policy checks and explicit confirmation.

Capture the controlled input, application context, resulting tool request and blocked or completed outcome. Do not retain real confidential prompts when a synthetic equivalent proves the boundary.

Challenge RAG, memory and vector-store isolation

Retrieval-augmented generation introduces ingestion, chunking, embedding, indexing, filtering and ranking decisions. Security failures can occur before the model is called: a poisoned source enters the index, a metadata filter is missing, a tenant identifier is dropped, or deleted content remains retrievable.

Test ingestion authority separately from retrieval authority. Then test direct identifiers, semantic neighbours, paraphrases and cross-session references. Verify deletion and update behaviour against the source of truth rather than accepting a friendly model response.

  • Every indexed item has verified provenance, owner, tenant, classification and lifecycle state.
  • Retrieval filters are enforced server-side before context reaches the model.
  • Conversation memory is isolated and expires according to a documented rule.
  • Deleted, revoked or reclassified content is removed from effective retrieval paths.

Use canary facts or synthetic documents with known ownership. The evidence should show which context was retrieved and why the requesting identity was or was not entitled to receive it.

Treat model output as attacker-influenced input

Generated text can reach browsers, document renderers, databases, APIs and code execution systems. If downstream components trust it, an attacker may convert prompt influence into XSS, injection, SSRF, unsafe deserialisation or command execution. Output moderation does not replace context-specific validation and encoding.

Trace every output sink. Use allowlisted operations and structured schemas for tool calls. Encode for the final rendering context, validate URLs and file paths, and keep code execution isolated with minimal privileges. Failure handling must not expose hidden context or retry an unsafe action with broader permissions.

  • HTML and rich content are contextually encoded or sanitised before rendering.
  • Generated URLs, queries, commands and file operations pass deterministic validation.
  • Tool schemas reject unexpected properties, destinations and privilege changes.
  • Downstream failures cannot cause verbose prompt, secret or stack disclosure.

Demonstrate the exact sink and its authoritative effect. A model producing a suspicious string is a hypothesis; the security finding depends on how the application handles that string.

Constrain agency, cost and repeated action

Agents can loop, retry, fan out and chain tools. An individually harmless operation may become damaging when repeated or combined. Apply budgets for time, tokens, tool calls, data volume and business actions. Make idempotency and final-state checks part of the workflow.

Test cancellation, timeout, partial failure and concurrent requests. Confirm that retries cannot repeat a payment, invitation, deletion or notification. The agent should not obtain new authority merely because a previous step returned an error or ambiguous result.

  • Per-user and per-tenant budgets exist for model and tool consumption.
  • High-impact actions use idempotency, transaction and approval controls.
  • Loops and retries terminate safely and preserve an explainable event trail.
  • Cancellation prevents queued or asynchronous actions from continuing unexpectedly.

Measure both technical and business state after the test. Cost or availability findings need a bounded proof and agreed limits to avoid creating the harm being assessed.

Test the model supply chain and configuration without overclaiming

The application may depend on hosted models, adapters, prompt packages, evaluation data, embedding models, vector databases and orchestration libraries. Inventory versions and trust sources. Separate a component provenance concern from a demonstrated application exploit, and document when the assessment cannot inspect a managed provider’s internals.

Configuration changes can be as important as code changes. A new tool, broader service scope, different retrieval source or modified system instruction can alter the attack surface. Include those changes in release review and trigger targeted retesting when authority or data access changes.

  • Model, adapter, embedding and orchestration dependencies are versioned and reviewed.
  • Prompt and tool configuration changes are controlled like application changes.
  • External model-provider limits and customer responsibilities are explicit.
  • Evaluation cases include security invariants, not only answer quality.

Report what was observed in the deployed application and distinguish it from provider-level assumptions. Avoid claiming that an application pentest certifies the security of a third-party foundation model.

What a WIMD assessment would test across the AI-enabled architecture

WIMD would start with the product workflow, not a generic prompt list. We would map identities, tenants, retrieval sources, memory, tools, downstream APIs and authoritative state. The scope would identify which actions are safe to exercise and which need synthetic data, approval gates or a staging replica.

We would create adversarial hypotheses around prohibited outcomes: cross-tenant retrieval, hidden-context disclosure, tool execution outside user authority, unsafe output reaching a privileged sink, or repeated agent actions changing business state. Prompt variations would support those hypotheses rather than become the measure of depth.

Findings would include the full trust path and durable control point. Retesting would replay the original prompt and action path, then sample alternate users, documents, tools and channels that share the same control.

  1. Map users, service identities, tenants, retrieval stores, tools and downstream APIs.
  2. Write explicit data and action invariants for each critical AI-enabled workflow.
  3. Test direct and indirect prompt influence against deterministic application controls.
  4. Validate output sinks, tool parameters, RAG isolation, memory lifecycle and agent budgets.
  5. Report minimum proof, root cause, affected pattern, remediation and retest evidence.

Define completion before the AI feature reaches production

A release decision should state which AI-enabled workflows were tested, which identities and tenants were represented, which tools and retrieval sources were included, and which unsafe actions were deliberately excluded. Record model and configuration versions because behaviour and attack paths can change without a traditional code deployment.

Security controls belong in the application architecture: least privilege, policy enforcement, provenance, validation, isolation, budgets, approvals and audit trails. Prompt hardening can reduce accidental behaviour, but it should not be the final boundary protecting sensitive data or business actions.

  • Every critical tool action is authorised independently of model output.
  • RAG, memory and vector retrieval preserve user and tenant isolation.
  • Generated output is validated for its actual downstream context.
  • Agent loops, retries, concurrency and cost are bounded and observable.
  • Configuration and model changes have security regression triggers.

Questions buyers and engineering teams ask

Is prompt injection the same as a traditional injection vulnerability?

Both involve untrusted input changing behaviour, but the exploit path differs. In an LLM application, the decisive risk is whether prompt influence crosses an application control and exposes data or triggers an unauthorised action. Traditional output sinks may then create familiar XSS, SQL, SSRF or command-injection consequences.

Can a hidden system prompt protect secrets?

No. System prompts can guide behaviour but should not be treated as a secret store or an authorization control. Keep secrets outside model context and enforce access through deterministic application policies.

What test data should be used for RAG isolation?

Use synthetic canary documents with known owners, tenants, classifications and lifecycle states. They allow testers to prove retrieval and deletion boundaries without exposing real confidential records.

Does an LLM application pentest assess the foundation model itself?

It primarily assesses the deployed application, integrations, configuration and observable model-mediated behaviour within scope. Provider internals may be outside access. The report should state that limitation rather than imply certification of the underlying model.

Useful technical references

Security test the authority around the model

LLM and agent risks become actionable when they are connected to identities, data and business effects. A test that stops at unusual output misses the controls that determine whether an attacker can cause harm.

Build the assessment around trust boundaries and prohibited outcomes. That approach remains useful as models and frameworks change because it tests the application decisions the organisation must continue to enforce.

Discuss an AI-enabled application security assessment with WIMD across prompts, tools, retrieval, identities, APIs and business workflows.