Large Language Models for World Health Organization-Uppsala Monitoring Centre Drug-Adverse Event Causality Assessment Using Food and Drug Administration Adverse Event Reporting System Cases: Comparative Performance Study.

چکیده اصلی

BACKGROUND: Causality assessment is central to pharmacovigilance but remains resource-intensive and subjective. The applicability of large language models (LLMs) to formal World Health Organization-Uppsala Monitoring Centre (WHO-UMC) drug-adverse event causality assessment has not been well established. OBJECTIVE: This study aims to evaluate the performance of LLMs in WHO-UMC causality assessment. METHODS: A curated set of 55 cases derived from the US Food and Drug Administration Adverse Event Reporting System, comprising 337 drug-level assessments, was constructed. Cases involving 2 to 11 suspected drugs were stratified by drug count, and 5 cases were sampled from each stratum. To ensure representation of rare but clinically important categories, 5 additional cases containing at least 1 "Certain" drug-adverse event pair were included. Case data were reorganized into a standardized semistructured format that preserved key elements required for WHO-UMC causality assessment. Domain experts conducted a pilot evaluation to align interpretation criteria prior to independently assessing the final dataset, yielding an interexpert agreement (Fleiss κ) of 0.762 across 337 drug-level assessments. Multiple prompting strategies, including standard prompting, chain-of-thought (CoT), CoT with self-consistency, few-shot, reasoning and acting, and tree-of-thought prompting, were applied across multiple LLMs, including GPT-5.4 and its mini variant and Gemini 2.5 Flash and Pro, via their respective application programming interfaces. Agreement with expert assessments was quantified using Cohen κ, weighted κ, and accuracy metrics. Internal consistency across repeated inferences was evaluated using Fleiss κ. RESULTS: Performance varied across models and prompting strategies. Cohen κ ranged from 0.368 to 0.641, weighted κ ranged from 0.641 to 0.821, accuracy ranged from 0.583 to 0.804, and balanced accuracy ranged from 0.513 to 0.735. Fleiss κ ranged from 0.730 to 0.915, corresponding to substantial to almost perfect agreement. The highest Cohen κ was observed for Gemini 2.5 Flash with CoT prompting (0.641). Gemini 2.5 Flash with CoT-self-consistency prompting showed a Cohen κ of 0.640 and achieved the highest observed point estimates for weighted κ (0.821), accuracy (0.804), and Fleiss κ (0.915), although the gains over other prompting strategies were modest. Category-level performance for this model showed higher performance for "Certain" (F1-score=0.793), "Probable/Likely" (F1-score=0.794), and "Unlikely" (F1-score=0.898), whereas performance for "Possible" remained substantially lower (F1-score=0.293), reflecting the difficulty of intermediate causality assessment. CONCLUSIONS: LLMs demonstrated moderate to substantial agreement in WHO-UMC causality assessment, indicating meaningful but still limited performance relative to expert judgment. Although LLMs are not suitable for independent decision-making, they may serve as supportive tools in pharmacovigilance workflows, particularly for preliminary case triage. Further studies using larger and more diverse datasets and evaluating performance on raw narrative reports are warranted.

متن کامل اصلی