The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement From Clinical Decision-Making Quality.

چکیده اصلی

BACKGROUND: Large language models (LLMs) have been proposed as decision-support tools in assisted reproductive technology (ART), but it remains unclear whether posttraining alignment strategies translate into clinically acceptable decision support. Outcome-based benchmarks may reward token-level correctness while overlooking the reasoning quality that clinicians rely on. OBJECTIVE: This study aimed to evaluate whether 4 mainstream alignment paradigms for medical LLMs (supervised fine-tuning [SFT], direct preference optimization [DPO], group relative policy optimization [GRPO], and in-context learning [ICL]) produce comparable algorithmic and clinical alignment (SFT vs GRPO) when used for infertility diagnosis and treatment planning. METHODS: This retrospective single-center study used 8201 deidentified electronic health records from West China Second University Hospital, collected between January 2020 and December 2022 (mean age 31.79, SD 4.63 y). All 4 strategies were built on a shared open-source biomedical backbone. Evaluation comprised the following 2 tiers: (1) automatic field-level metrics (accuracy, macro-F1, and mean absolute error [MAE]) on 5 structured decision fields (infertility type, initial diagnosis, ART strategy, controlled ovarian stimulation [COS] regimen, and gonadotropin starting dose); and (2) blinded independent expert review by 2 reproductive medicine specialists on 100 paired cases across 4 clinical dimensions (reasoning capability, diagnostic accuracy, treatment feasibility, and hallucination). Automatic field-level evaluation included all 4 strategies, whereas blinded expert review was restricted to the SFT versus GRPO contrast. RESULTS: GRPO achieved the highest average automatic performance (eg, infertility type accuracy=92.57%, COS regimen accuracy=62.36%, ART strategy accuracy=76.49%, and gonadotropin dose MAE=44.94). However, in blinded expert review, the conservative SFT baseline showed directionally higher expert ratings than GRPO on reasoning capability and treatment feasibility; diagnostic-accuracy differences were not significant. In the 3-way best-response comparison including the original physician-charted plan, the SFT baseline was selected as the best response in 51.2% (102.3/200) of cases compared with 26.2% (52.4/200) for GRPO, and 22.6% (45.3/200) for the charted plan. This result reflects preference within the standardized review format, not evidence that model-generated decisions are clinically superior to physician decision-making. Hallucination rates were 15% (GRPO) and 18.5% (SFT), indicating that higher automatic performance did not eliminate clinically unsupported content and that neither model is ready for clinical deployment. Subgroup analyses showed GRPO improved F1 in in vitro fertilization (IVF) and preimplantation genetic testing (PGT) but decreased F1 in intracytoplasmic sperm injection (ICSI) cases, where male-factor information was largely captured only in unstructured fields. CONCLUSIONS: Outcome-based metrics alone are insufficient proxies for clinical utility in ART decision support. Algorithmic improvement and clinical alignment suggest a possible decoupling; a phenomenon we term the alignment paradox. However, because the clinical review was based on 2 reproductive medicine specialists with marginal interrater agreement, the clinical-alignment findings should be interpreted as exploratory. External multicenter validation is required before clinical deployment.

ترجمه فارسی چکیده

پیشینه: مدل‌های زبانی بزرگ (LLM) به عنوان ابزارهای پشتیبانی تصمیم در تکنولوژی کمک‌باروری (ART) پیشنهاد شده‌اند، اما مشخص نیست که آیا استراتژی‌های همسویی پس از آموزش به تصمیم‌گیری بالینی قابل قبول منجر می‌شوند یا خیر. هدف: این مطالعه ارزیابی می‌کند که آیا چهار رویکرد همسویی اصلی (SFT، DPO، GRPO، ICL) در تشخیص و برنامه‌ریزی درمان ناباروری، همسویی الگوریتمی و بالینی قابل مقایسه‌ای ایجاد می‌کنند یا خیر. روش‌ها: این مطالعه بازنگری تک‌مرکزی از ۸۲۰۱ پرونده پزشکی الکترونیکی (EHR) از بیمارستان دوم دانشگاه غرب چین (ژانویه ۲۰۲۰ تا دسامبر ۲۰۲۲) استفاده کرد. ارزیابی شامل دو لایه بود: (۱) معیارهای خودکار در ۵ حوزه تصمیم‌گیری ساختاریافته و (۲) بررسی مستقل تخصصی توسط دو متخصص پزشکی باروری بر روی ۱۰۰ مورد جفت‌شده. نتایج: GRPO بالاترین عملکرد خودکار را داشت، اما در بررسی تخصصی، SFT امتیازات بالاتری در توانایی استدلال و قابلیت اجرای درمان داشت. SFT در ۵۱.۲٪ موارد به عنوان بهترین پاسخ انتخاب شد، در حالی که GRPO در ۲۶.۲٪. نتیجه‌گیری: معیارهای مبتنی بر نتیجه به تنهایی برای کاربرد بالینی کافی نیستند و پدیده‌ای به نام «تناقض همسویی» مشاهده شد. نتایج بالینی باید به عنوان اکتشافی تفسیر شوند و نیاز به تأیید چندمرکزی خارجی دارد.

متن کامل اصلی