Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model-Generated Medical Information: Empirical Study.

چکیده اصلی

BACKGROUND: Despite high reported accuracy on clinical and evidence appraisal tasks, AI-generated medical information may lack explicit support from source documents. This creates challenges for digital health practitioners regarding transparency, auditability, and trust when AI systems are used for evidence synthesis, guideline development, and clinical knowledge management. Large language models (LLMs) can generate fluent and seemingly correct outputs, but existing evaluations often rely on agreement with human judgments and do not directly assess whether AI-generated content is grounded in underlying evidence. OBJECTIVE: This study measures evidence support and hallucination in AI-generated medical information by assessing the extent to which LLM-generated risk-of-bias assessments are supported by source clinical trial reports. METHODS: We evaluated 3 LLMs (GPT-5, OpenAI o3-mini, and GPT-3.5) on risk-of-bias (RoB 2) assessment using all 97 randomized controlled trials for which the source Cochrane systematic review provided complete human RoB 2 annotations and full-text reports were accessible, constituting the complete reference set. No train-validation split was applied; all 97 studies were used for evaluation. Model outputs were constrained to structured RoB 2 signaling questions and domain-level judgments. For each generated claim, relevant text passages were retrieved from trial reports using the Okapi BM25 (Best Matching 25) algorithm. A verification step assigned evidence verdicts (supported, contradicted, not found, or out of scope) with verbatim quotations. We quantified evidence support rates and conservative and strict hallucination rates. Task performance was evaluated using exact and binary accuracy, sensitivity, specificity, F1-score, Youden J, and agreement with human reviewers using Cohen κ and Fleiss κ. RESULTS: Binary accuracy of AI-generated risk-of-bias judgments was high across domains (90%-98%), whereas exact accuracy was substantially lower (42%-71%), reflecting frequent disagreements in severity classification despite correct directional classification. GPT-5 achieved the strongest overall performance, including perfect binary accuracy for overall risk-of-bias conclusions and the highest agreement with human reviewers (quadratic κ up to 0.81). However, evidence support rates across models ranged from only 60% to 65%, with conservative hallucination rates of 34%-37%. GPT-5 showed the highest mean evidence support (64.3%) and the lowest strict hallucination rate (35.7%). Mean top-1 BM25 retrieval scores were similar across models (approximately 30-31), suggesting that differences in hallucination were not primarily attributable to differences in retrieval strength. CONCLUSIONS: AI-generated medical information can achieve high decision-level accuracy while still lacking documentary support in a substantial proportion of outputs. Measuring evidence support and hallucination reveals important limitations that are not captured by agreement metrics alone. Retrieval-based evidence verification provides a reproducible and transparent approach for evaluating the reliability of AI-generated medical information, with direct relevance to digital health practice, evidence-based medicine, and medical informatics.

ترجمه فارسی چکیده

پیشینه: با وجود دقت بالا گزارش‌شده در تسک‌های بالینی و ارزیابی شواهد، اطلاعات پزشکی تولید شده توسط هوش مصنوعی ممکن است فاقد پشتیبانی صریح از سند منبع باشد. این امر چالش‌هایی را برای متخصصان سلامت دیجیتال در مورد شفافیت، قابل ردیابی بودن و اعتماد ایجاد می‌کند. هدف: این مطالعه میزان پشتیبانی شواهد و توهم در اطلاعات پزشکی تولید شده توسط هوش مصنوعی را با ارزیابی میزان پشتیبانی ارزیابی‌های ریسک سوگیری (RoB 2) توسط گزارش‌های بالینی تصادفی شده اندازه‌گیری می‌کند. روش‌ها: ما ۳ مدل زبانی بزرگ (GPT-5، OpenAI o3-mini و GPT-3.5) را بر روی ارزیابی ریسک سوگیری (RoB 2) با استفاده از تمام ۹۷ مطالعه کنترل تصادفی ارزیابی کردیم. نتایج: دقت باینری ارزیابی‌های ریسک سوگیری تولید شده توسط هوش مصنوعی در تمام حوزه‌ها بالا (۹۰٪ تا ۹۸٪) بود، در حالی که دقت دقیق به طور قابل توجهی پایین‌تر (۴۲٪ تا ۷۱٪) بود. GPT-5 بالاترین عملکرد کلی را داشت، اما نرخ‌های پشتیبانی شواهد بین مدل‌ها تنها بین ۶۰٪ تا ۶۵٪ بود. نتیجه‌گیری: اطلاعات پزشکی تولید شده توسط هوش مصنوعی می‌تواند دقت سطح تصمیم‌گیری بالا را به دست آورد، اما همچنان فاقد پشتیبانی سندی در بخش قابل توجهی از خروجی‌هاست.

متن کامل اصلی