How do I run a post-incident review and corrective action after an AI agent failure?

After an AI agent failure, a post-incident review and corrective action plan should include real-time error detection, trajectory saving for analysis, and mechanisms for deactivation or rollback, all within a structured incident response framework.

Concrete controls for post-incident review and corrective action include:

• Real-time Error Detection: Implement mechanisms like _detect_tool_failure to surface tool failures immediately, allowing for prompt intervention. This aligns with the NIST AI RMF function of NIST-MANAGE-4.1 for incident response and post-deployment monitoring.
• Trajectory Saving: Save every completed conversation as a JSONL entry, separating successful and failed trajectories. This enables post-incident replay, analysis of failure modes, and generation of training data.
• Deactivation and Rollback Procedures: Establish procedures to deactivate, roll back, or retire AI systems that exceed risk tolerances, such as kill-switches or rollback capabilities for agents. This maps to NIST-MANAGE-2.3 for mechanisms to sustain value and retire safely.
• Incident Response Plan: Have an AI/agent incident-response plan in place that covers detection, escalation, containment, communication, and learning. This is a direct control under NIST-MANAGE-4.1.
• Override Audit Logs: Log every human override with the human's identity, the reason given, the prior agent decision, and the override outcome to ensure accountability and improve oversight. This supports the Human Oversight & Override principle.
• Policy Enforcement and Dynamic Intervention: When verification fails, enforcement options include blocking, redacting, transforming, escalating, or quarantining. Dynamic intervention allows for real-time responses without redeployment, such as hot-loading policy bundles or temporarily revoking tool capabilities. This is part of the Verification, enforcement, and dynamic intervention capabilities.