Development and evaluation of a large language model-based, retrieval-augmented generation application for query response in early oncology clinical trials.

چکیده اصلی

BACKGROUND: Early-phase oncology trials involve complex protocols and extensive documents, making timely resolution of study queries challenging. We developed the Study Document Assistant (SDA), a retrieval-augmented generation (RAG) system that integrates semantic search with large language models to generate context-specific answers from clinical trial documentation. MATERIALS AND METHODS: SDA processes study documents and delivers responses through a secure, conversational interface. A two-arm experiment was conducted in which sponsor clinical study team members answered protocol-related queries either with or without SDA. Questions were stratified by difficulty and assessed by blinded subject matter experts (SMEs) and artificial intelligence (AI) models. The primary endpoint was mean response time. Main secondary endpoints included response time as a time-to-event outcome and overall accuracy. SDA user satisfaction was also assessed. Data were analyzed using Welch t-test for the primary endpoint, Kaplan-Meier method and log-rank test for time-to-event analysis, Mann-Whitney U test for accuracy, and descriptive statistics for satisfaction. RESULTS: In the experiment (N = 10 per arm), SDA users achieved a mean response time of 14.9 [95% confidence interval (CI) 9.3-20.6] versus 26.5 (95% CI 16.8-36.1) minutes in controls, corresponding to a 43.8% reduction (95% CI 11.9% to 64%, P = 0.0347). A similar reduction for median time to response was observed. Accuracy, measured as the median SME score (scale 1-4), was ∼3 ("Mostly Correct") for both arms, with no statistically significant difference. Evaluation by AI models showed similar results. CONCLUSIONS: Our findings demonstrate that SDA accelerates query resolution without compromising answer quality, highlighting the potential of RAG-based digital assistants to streamline clinical trial operations and enhance research efficiency.

متن کامل اصلی