What is OWASP LLM09 misinformation and how do I reduce hallucinations and overreliance in production?

OWASP LLM09 Misinformation refers to instances where an AI system generates false or unsupported information, leading users to over-rely on it (confabulation). To reduce hallucinations and overreliance in production, implement robust verification pipelines and human oversight.

• Grounding and Citation Requirements: Require the LLM to provide grounding or citations for its outputs and to refuse to answer when evidence is weak. This aligns with the principle of NIST AI RMF Govern function by ensuring accountability for information provided.
• Multi-stage Verification Pipelines: Implement progressive filtering funnels that combine multiple verification methods. This can include multi-LLM verification, cross-checking findings across different models, and hybrid approaches that integrate traditional static analysis tools (SAST) to validate LLM hypotheses. Dynamic testing and sandboxed exploit verification can also be used to confirm the reality and exploitability of findings.
• Human Oversight for Consequential Decisions: Ensure human oversight for decisions that have significant consequences. This is a requirement in major AI governance regimes, including the EU AI Act, NIST AI Risk Management Framework, and ISO 42001.
• Clear AI Disclaimers: Provide clear disclaimers to users about the AI's capabilities and limitations to manage expectations and prevent overreliance.
• Memory Management and Evaluation: For agents with memory, implement explicit retention and deletion rules for sensitive information and evaluate memory as a product surface. This involves testing to ensure the dream (memory) preserves true preferences, rejects one-off notes, updates stale facts, and avoids promoting sensitive data.
• Reduce Effective Dimension of Prompt Interface: Standardize formats, restrict API parameters, bound context lengths, and use function-calling schemas to reduce the complexity of the prompt space, making it more tractable for red-teaming and monitoring. This helps in controlling the model's behavior and reducing the likelihood of unexpected outputs.