How do I detect anomalous or compromised behavior in a running AI agent?

Detecting anomalous or compromised behavior in a running AI agent involves establishing a baseline of normal activity and then monitoring for statistically significant deviations. This process is crucial because AI agents have agency and can autonomously make decisions and take actions, making the window between compromise and incident very short.

To detect anomalous behavior, organizations should implement Behavioral Baselining and Anomaly Detection (NIST AI RMF Function: Govern, ISO/IEC 42001 Control: 8.2.2 AI system logging). This involves observing agents over time to understand their normal tool call patterns, data access scopes, and outbound traffic volumes. Alerts should be triggered for statistically significant deviations from these baselines. For instance, an agent that suddenly queries a database it has never accessed before could indicate a legitimate new instruction or a prompt injection. The "Lethal Trifecta" (access to private data, exposure to untrusted content, and external communication) should be prioritized for investigation due to its high-risk profile.

Furthermore, Identity and Credential Correlation (NIST AI RMF Function: Govern, ISO/IEC 42001 Control: 8.2.1 AI system access control) is essential. This phase correlates the agent inventory with the organization's identity infrastructure, such as secrets management systems, OAuth grant logs, and API key issuance records, to determine whose credentials an agent is operating under. Orphaned credentials in shadow agents pose a significant persistence risk.

Finally, comprehensive instrumentation and tamper-evident audit logs (NIST AI RMF Function: Govern, ISO/IEC 42001 Control: 8.2.2 AI system logging) are critical for observability and forensic replay. Telemetry should be produced by construction at every chokepoint, ensuring that every action leaves a trace. This includes distributed tracing with stable trace IDs across all hops, and real-time anomaly detection. It is important to note that traditional security tools like EDR, DLP, and CASB often fail to detect AI agent anomalies due to their lack of semantic understanding of AI agent processes, encrypted LLM traffic, and programmatic API calls outside of browser contexts.