The AI Security Checklist

A prioritized, practical checklist for securing language model applications and agents

//Executive Summary

Securing an AI powered application is not fundamentally different from securing any other piece of production software, but it does introduce a distinct set of failure modes that conventional application security checklists do not cover: prompt injection, excessive agency, unbounded tool access, and the challenge of auditing systems whose reasoning is not fully deterministic. OWASP's ranking of prompt injection as the top risk facing deployed language model applications, alongside Gartner's description of it as a critical, unavoidable threat, reflects a genuine architectural property of these systems rather than a simple implementation bug. This paper offers a practical, prioritized checklist covering the specific controls that matter most for organizations deploying language model applications and autonomous agents today.

//Table of Contents

• ▸Introduction
• ▸Background
• ▸Core Concepts
• ▸Technical Deep Dive
• ▸Practical Applications
• ▸Challenges
• ▸Best Practices
• ▸Future Outlook
• ▸Key Takeaways
• ▸Conclusion
• ▸References

//Introduction

Security teams evaluating an AI powered application often default to applying a conventional application security checklist, and while much of that checklist remains relevant, authentication, encryption, input validation, dependency management, it misses the risks specific to language model systems. This paper is organized around the additional controls that matter specifically because a language model, and any agent built on top of one, behaves differently from conventional deterministic software.

//Background

The risks specific to AI systems became concrete rather than theoretical through 2026, as disclosed vulnerabilities in widely used coding and productivity agents demonstrated real world exploitability, several rated at or near the top of the standard severity scale used to prioritize security response. This has pushed security frameworks, including OWASP's Top 10 for Large Language Model Applications and Gartner's threat research, to treat prompt injection and related risks as first tier concerns rather than a niche or emerging category.

//Core Concepts

**Excessive agency.** A system granted broader capabilities or permissions than its task genuinely requires, which turns any successful manipulation of its reasoning into a larger scale problem than it would otherwise be.

**Insecure output handling.** A failure to treat a language model's output as untrusted content before acting on it or displaying it, which can allow injected instructions or malicious content to propagate into downstream systems.

**Data poisoning.** The introduction of malicious or misleading content into training data, retrieval sources, or persistent memory stores that a model relies on, intended to bias or manipulate its future behavior.

**Supply chain risk.** Vulnerabilities introduced through third party models, plugins, tools, or agent frameworks rather than through code an organization wrote itself.

//Technical Deep Dive

The checklist, organized by layer

```mermaid

flowchart TD

A[Access and identity] --> B[Tool and capability scoping]

B --> C[Input and output handling]

C --> D[Monitoring and audit]

D --> E[Incident response readiness]

```

**Access and identity**

• ▸Authenticate every user and system interacting with the AI application using the same rigor applied to any other production system.
• ▸Apply role based access control to any AI feature that can access sensitive data, ensuring the system respects the same permissions the underlying data already has.
• ▸Avoid granting a shared, broadly privileged service account to an AI system when narrower, task specific credentials are feasible.

**Tool and capability scoping**

• ▸Grant every tool an agent can call the minimum privilege necessary for its specific task, reviewed on a recurring schedule, not only at initial deployment.
• ▸Avoid granting write access to production systems, financial systems, or external communication channels without a human approval gate for any action with real world consequences.
• ▸Treat every third party tool, plugin, or MCP server integration as a supply chain dependency requiring the same vetting as any other external code dependency.

**Input and output handling**

• ▸Treat all content a model reads, whether user input, retrieved documents, emails, or web pages, as untrusted, since any of it can carry an injected instruction.
• ▸Apply content filtering to detect likely injection attempts, understanding that filtering reduces but does not eliminate risk given the architectural nature of the vulnerability.
• ▸Treat model output as untrusted before it is used to trigger further actions or displayed in a context where it could be rendered as executable content, such as embedded scripts or links.

**Monitoring and audit**

• ▸Log every tool call, decision, and action an agent takes with enough structure to reconstruct what happened during an incident review.
• ▸Monitor for anomalous patterns in agent behavior, such as unusual tool call sequences or a sudden spike in a specific type of action, which can indicate a successful manipulation attempt.
• ▸Establish baseline behavior for agent systems so that deviations are detectable rather than lost in normal operational noise.

**Incident response readiness**

• ▸Define a clear ownership and escalation path specifically for AI related incidents, distinct from, but connected to, conventional security incident response procedures.
• ▸Maintain the ability to quickly disable or roll back a specific agent's tool access without taking down the entire application, to contain an incident without unnecessary broader disruption.
• ▸Conduct tabletop exercises simulating a prompt injection or data leakage incident specifically, since generic security incident training often does not cover the unique dynamics of these failure modes.

A severity based prioritization framework

| Risk category | Typical severity if unmitigated | Recommended baseline control |

|---|---|---|

| Prompt injection leading to data disclosure | High | Least privilege scoping and output filtering |

| Excessive agency enabling unauthorized actions | High | Human approval gates on consequential actions |

| Supply chain risk from third party tools | Medium to high | Vetting and sandboxing of external integrations |

| Data poisoning through retrieval or memory | Medium | Access controls and content validation on knowledge sources |

| Insecure output handling | Medium | Treating all output as untrusted before downstream use |

Building an internal red team practice

Organizations with mature AI security posture increasingly maintain a dedicated internal red team function specifically for AI systems, distinct from conventional application security red teaming, tasked with actively attempting to manipulate deployed agents through prompt injection, excessive agency exploitation, and other AI specific attack patterns before external adversaries do. This function is most valuable when it operates continuously rather than as a one time pre launch exercise, since both the space of known attack patterns and the specific system's own behavior evolve over time as models, tools, and prompts are updated.

```mermaid

flowchart TD

A[Internal AI red team] --> B[Attempt prompt injection against production and staging systems]

A --> C[Attempt excessive agency exploitation]

A --> D[Attempt data exfiltration through indirect channels]

B --> E[Findings fed back to engineering and security teams]

C --> E

D --> E

E --> F[Prioritized remediation]

F --> A

```

A well run internal red team practice also serves an important secondary function beyond finding vulnerabilities directly: it builds institutional muscle memory for how to respond quickly and effectively when a genuine incident occurs, since the team has already rehearsed the investigative and remediation process repeatedly in a controlled setting rather than encountering it for the first time during an actual live incident.

//Practical Applications

**Customer facing chat and support systems** should prioritize output handling controls and human review gates on any communication sent externally without review, given that these systems are frequently targeted by indirect injection through manipulated support content.

**Internal coding and DevOps agents** should prioritize tool scoping and sandboxed execution environments, given that real world disclosures in 2026 concentrated heavily in this category, including command injection vulnerabilities in agent shell tools.

**Financial and healthcare systems** should apply the most conservative approval gates of any category, given the cost of an error, and should assume regulatory scrutiny of AI decision making will increase rather than decrease over the coming years.

//Challenges

**Checklist fatigue without prioritization.** A long, undifferentiated list of controls without clear prioritization tends to result in partial, inconsistent implementation. Security teams should focus first on the controls addressing the highest severity, most exploitable risks specific to their actual deployment.

**False confidence from partial mitigation.** Teams that implement input filtering alone, without corresponding tool scoping and approval gates, often develop a false sense of security that does not match documented real world success rates for injection attempts against even actively defended systems.

**Cross team ownership gaps.** AI security often falls into a gap between application security teams unfamiliar with model specific risks and machine learning teams unfamiliar with conventional security practice. Closing this gap requires deliberate cross training or a dedicated function bridging both disciplines.

//Best Practices

• ▸Prioritize least privilege tool scoping and human approval gates on consequential actions as the two highest leverage controls available.
• ▸Treat every piece of content a model reads as untrusted input, regardless of its apparent source.
• ▸Build monitoring and audit logging into agent systems from the outset, not as a retrofit after an incident.
• ▸Vet third party tools, plugins, and integrations with the same rigor applied to any other supply chain dependency.
• ▸Run incident response exercises specific to AI failure modes, not only generic security scenarios.
• ▸Assign clear, specific ownership for AI security that bridges application security and machine learning engineering expertise.
• ▸Reassess tool access grants on a recurring schedule, since scope tends to expand gradually unless actively reviewed.

//Future Outlook

**Next two years.** Expect more formalized, independently audited security certifications specific to AI applications and agents, similar to existing security certification frameworks for conventional software, as enterprise buyers increasingly demand documented evidence of security posture rather than general assurances.

**Next five years.** Expect the gap between application security and AI security expertise to narrow considerably as security professionals develop AI specific competency as a standard part of the discipline, rather than a specialized niche.

**Next ten years.** Expect architectural approaches that enforce a structural separation between trusted instructions and untrusted data at the system level, currently an active area of research, to mature into standard practice, meaningfully reducing though not eliminating the underlying risk that prompt injection represents.

//Key Takeaways

• ▸AI security requires a distinct set of controls beyond conventional application security, centered on tool scoping, output handling, and monitoring for anomalous agent behavior.
• ▸Least privilege tool access and human approval gates on consequential actions are the two highest leverage controls currently available.
• ▸Real world disclosed vulnerabilities in 2026 demonstrated that these risks are concrete and exploitable, not theoretical.
• ▸Cross team ownership between application security and machine learning engineering is essential and currently a common organizational gap.
• ▸No single control eliminates AI specific security risk; a layered, prioritized approach is the only responsible posture.

//Conclusion

Securing an AI application is an extension of good security practice, not a replacement for it, applied to a system with a genuinely new set of failure modes. The organizations managing this well are prioritizing the highest leverage controls, tool scoping and approval gates, building monitoring in from the start, and treating this as a distinct but connected discipline rather than either ignoring it or treating it as an entirely separate problem requiring an entirely separate team with no connection to existing security practice.

//References

• ▸OWASP, Top 10 for Large Language Model Applications, owasp.org
• ▸Gartner, Cybersecurity Threat: Prompt Injection and the 2026 to 2027 ThreatScape, gartner.com
• ▸NIST, AI Risk Management Framework, nist.gov
• ▸UK National Cyber Security Centre, guidance on prompt injection and agentic AI risk, ncsc.gov.uk
• ▸Microsoft, security guidance for agentic AI systems, microsoft.com