What goes in a model card and what AI documentation do standards require?

Model cards typically include details about the model's architecture, capabilities, and performance benchmarks, especially for complex agentic workflows. Standards require documentation of the AI system's context, purpose, and potential impacts, as well as an inventory of AI systems and their resources.

A model card should contain:

• Model Architecture and Capabilities: Information about the model's design, such as its base model, thinking levels for quality, cost, and latency control, supported input types (e.g., text, images, audio, video), and context window size.
• Performance Benchmarks: Specific benchmark results, including both general and agentic/real-world benchmarks, to demonstrate the model's performance across various tasks. This can include metrics like pass rates on coding tasks, agentic workflow performance, and reasoning capabilities.
• Cost and Latency Information: Details on pricing per token and how different configurations might affect cost and latency, especially for models designed for varying task complexities.
• Safety and Risk Information: While not explicitly called out as "model card" content in the provided sources, system cards for advanced models include extensive safety evaluations, such as red-teaming results for cybersecurity and biological risks, and discussions on alignment and safety innovations.

Standards require the following AI documentation:

• System Context & Intended Purpose: Documentation of the intended purpose, deployment setting, and operating context of each AI/agent system (NIST AI RMF MAP-1.1).
• AI System Inventory: Maintenance of an inventory of AI/agent systems, including models, agents, tools, and data flows (NIST AI RMF MAP-1.5).
• Resources for AI Systems: Identification and documentation of resources such as data, tooling, compute, and human competence, including the model/tool components in use (ISO/IEC 42001 A.4).
• AI Impact Assessment: Processes to assess the impacts of AI systems on individuals, groups, and society across the lifecycle (ISO/IEC 42001 A.5, cross-maps NIST AI RMF MAP-5.1).
• Information for Interested Parties: Availability of information about AI systems, including their capabilities, limitations, and intended use, to relevant interested parties to support transparency (ISO/IEC 42001 A.8).