AI assurance · Architecture

Securing RAG and Agentic AI Systems

How retrieval, tools, memory, and authorization reshape application security.

HackStack Security insight

RAG and agents turn model output into application behavior.

Retrieval-augmented generation improves relevance, while agents let models take action. Both also introduce new trust boundaries that traditional application testing can miss.

Retrieval is an authorization problem

Documents should be filtered using the requesting user’s identity and permissions before they reach the model. Similarity search is not access control, and metadata filters must be tested for bypasses.

External content can become instructions

Retrieved pages, support tickets, documents, and emails may contain malicious instructions. Systems should treat retrieved content as untrusted data and restrict its ability to influence tools or system behavior.

Agents need deterministic boundaries

Models should not decide their own permissions. Tool access, parameters, approval requirements, transaction limits, and tenant boundaries must be enforced outside the model.

Memory changes the risk lifecycle

Conversation memory can preserve sensitive data, malicious instructions, or incorrect assumptions across sessions. Teams should define retention, isolation, deletion, and review controls.

Security review questions

  • Can one user retrieve another user’s data?
  • Can a document influence tool execution?
  • Can the agent perform a high-impact action without approval?
  • Can sensitive information persist in memory, logs, or traces?
  • Can the application explain and audit why an action occurred?
Research notes

Questions that turn guidance into a review.

Use these prompts to challenge assumptions, collect evidence, and make the article actionable for engineering and security teams.

Review questions
  1. Is authorization applied before retrieval, after retrieval, or only in the user interface?
  2. Can retrieved content be interpreted as instructions instead of untrusted data?
  3. Can the model select tools, parameters, or transaction values beyond the user’s authority?
  4. How are memory, traces, embeddings, and cached responses isolated and deleted?
Evidence and signals
  • Metadata-filter bypasses and cross-user retrieval results
  • Poisoned document behavior, indirect prompt injection, and unsafe citations
  • Tool-call validation, transaction limits, approval steps, and retry behavior
  • Sensitive data appearing in logs, traces, vector stores, or model output
Primary references

References support the review approach; they do not replace architecture-specific threat modeling or validation.

Start with a focused conversation

Need an independent review of your AI architecture?

Tell us what you are building, changing, or concerned about. We will help you define the right security review.

Request a security review ↗Explore services ↗