Evaluating the Reliability, Accuracy, and Readability of ChatGPT-3.5 and GPT-4 in Providing Patient Education Related to Glenohumeral Joint Osteoarthritis.

چکیده اصلی

OBJECTIVES: Chatbots have been increasingly recognized as modern tools that provide patients with reliable health-related information. This study aimed to evaluate and compare ChatGPT-3.5 and GPT-4's ability to answer glenohumeral osteoarthritis-related questions. METHODS: Fifteen questions were derived from the 2020 AAOS Clinical Practice Guidelines for the Surgical Management of Glenohumeral Joint Osteoarthritis. Questions were categorized into three groups: risk factors, implant/intraoperative considerations, and pain/functional outcomes. ChatGPT-3.5 and GPT-4 were prompted with these questions, and responses were evaluated by four fellowship-trained shoulder and elbow surgeons. Each response was rated on a scale (scores:1-5) based on relevance, accuracy, clarity, completeness, and evidence-based support. Data was analyzed descriptively and statistically to compare the scores between ChatGPT-3.5 and GPT-4. RESULTS: Average score for ChatGPT-3.5 was 19.7/25, with "Risk Factor" prompts achieving the highest mean score. GPT-4 averaged 18.7/25, with "Functional Outcomes" prompts scoring highest. However, there were no statistically significant differences between different prompt themes for GPT-3.5 and GPT-4. "Clarity" category received the highest score for GPT-3.5, while "Relevance" was highest for GPT-4. Both models scored lowest on "Evidence-based" prompts. On the Flesch- Kincaid scale, GPT-3.5 responses had a significantly higher score of 18.3 compared to GPT-4's 15.4, indicating a more difficult reading level in GPT-3.5's responses. CONCLUSION: Both ChatGPT-3.5 and GPT-4 performed adequately in providing well-informed medical responses to patient queries about glenohumeral osteoarthritis. Future chatbot versions should focus on providing evidence-based content through systematic and reliable reviews of literature, in an accessible readable manner.

ترجمه فارسی چکیده

هدف این مطالعه ارزیابی و مقایسه توانایی چت‌بات‌های ChatGPT-3.5 و GPT-4 در پاسخگویی به سوالات مربوط به آرتروز مفصل شانه بود. پرسش‌ها از دستورالعمل‌های بالینی ۲۰۲۰ AAOS برای مدیریت جراحی این بیماری استخراج شدند. پاسخ‌ها توسط چهار جراح متخصص شانه و مچ دست ارزیابی شدند. میانگین نمره ChatGPT-3.5 ۱۹.۷ از ۲۵ بود و نمره بالاتر در دسته‌بندی عوامل خطر بود. GPT-4 میانگین ۱۸.۷ از ۲۵ داشت و نمره بالاتر در دسته‌بندی نتایج عملکردی بود. تفاوت‌های معنادار آماری بین تم‌های مختلف پرسش‌ها مشاهده نشد. ChatGPT-3.5 در دسته‌بندی «وضوح» نمره بالاتری داشت و GPT-4 در دسته‌بندی «مرتبط بودن». هر دو مدل در پاسخ‌های مبتنی بر شواهد نمرات پایینی گرفتند. بر اساس مقیاس Flesch-Kincaid، پاسخ‌های ChatGPT-3.5 سطح خواندن سخت‌تری داشتند.

متن کامل اصلی