What is the NIST AI 600-1 Generative AI Profile and how does it apply to LLM agents?

The NIST AI 600-1 Generative AI Profile is a specialized profile of the NIST AI Risk Management Framework (AI RMF 1.0) that identifies risks particularly relevant to Large Language Model (LLM) and agent stacks. It applies to LLM agents by highlighting specific risk areas that need to be addressed, especially for customer-facing deployments or those handling confidential or regulated data.

The Generative AI Profile identifies several key risks for LLM agents:

• Confabulation / hallucination: This refers to the generation of ungrounded or fabricated output by the agent. Controls like weak-retrieval or no-fabrication guards are relevant here.
• Information security: This encompasses risks such as prompt injection (OWASP LLM01), data exfiltration (OWASP LLM02), and insecure tool use (OWASP LLM06).
• Data privacy: This risk involves the leakage of sensitive or Personally Identifiable Information (PII) from training or context data (OWASP LLM02). Architectures should support per-tenant data residency, configurable redaction at ingestion, and retention aligned with regulatory regimes.
• Dangerous / harmful content & CBRN uplift: This addresses the need for guardrails and refusal coverage to prevent the generation of harmful content.
• Value-chain & component integration: This focuses on the provenance of third-party models and tools (NIST-GOVERN-6.1, OWASP LLM03/LLM05). Policies should address risks from third-party models, datasets, and tools, tracking provenance, licensing, and model-update risks.

For managing these risks, practical controls include:

• Incident Response & Post-Deployment Monitoring: An AI/agent incident-response plan should be in place for detection, escalation, containment, communication, and learning (NIST-MANAGE-4.1).
• Mechanisms to Sustain Value & Retire Safely: Procedures for deactivating, rolling back, or safely retiring AI systems that exceed risk tolerances, such as kill-switches for agents, are crucial (NIST-MANAGE-2.3).
• Risk Response Prioritization: Responses to the highest-priority AI risks must be planned, tracked, and resourced (NIST-MANAGE-1.3).
• Observability and Logging: Comprehensive instrumentation by default, tamper-evident audit logs, and out-of-band shipping to a Security Information and Event Management (SIEM) system are essential to prevent log tampering and ensure forensic replay capabilities (NIST-MEASURE-2.8).
• Third-Party Risk Management: Policies should address risks from third-party models, datasets, and tools, including tracking provenance, licensing, and model-update risks (NIST-GOVERN-6.1).