What technical and governance requirements must a high-risk AI system meet under the EU AI Act?

High-risk AI systems under the EU AI Act must meet technical requirements for human oversight and security, and governance requirements for risk management and accountability. These requirements align with controls found in the NIST AI RMF and ISO/IEC 42001.

• Human Oversight: High-risk AI systems must be designed for effective oversight by natural persons, including defined policies for human-AI configurations, override authority, and the boundary of agent autonomy. This includes structured logging of all human decisions in the same audit stream as agent actions to prevent audit blind spots. (NIST-GOVERN-3.2)
• Risk Management and Assessment: Organizations must establish an AI Management System (AIMS) with an AI policy, documented roles, responsibilities, and authorities for AI governance. This includes identifying potential positive and negative impacts to individuals, groups, and society, and conducting AI risk and impact assessments. (ISO/IEC 42001 Cl.5, NIST-MAP-5.1)
• Security and Resilience: AI systems must be evaluated for security and resilience, including adversarial robustness, prompt-injection resistance (OWASP LLM01), and abuse resistance. Policies must address risks from third-party models, datasets, and tools, tracking provenance, licensing, and model-update risk. (NIST-MEASURE-2.7, NIST-GOVERN-6.1)
• Transparency and Accountability: Mechanisms must exist to log decisions and trace AI behavior, ensuring every consequential action has a traceable human accountability path. An inventory of AI/agent systems (models, agents, tools, data flows) must be maintained and kept current. (NIST-MEASURE-2.8, NIST-MAP-1.5)
• Monitoring and Incident Response: Approaches for tracking identified and emergent risks, such as monitoring, logging, and drift detection, are required. Post-deployment monitoring and an AI/agent incident-response plan (detection, escalation, containment, communication, learning) must be in place. (NIST-MEASURE-3.1, NIST-MANAGE-4.1)
• Continual Improvement: The AIMS must undergo continual improvement and corrective action, addressing nonconformities and improving over time. (ISO/IEC 42001 Cl.10)