Responses of AI chatbots to escalating suicide risk: A simulation study of repeated interactions.

چکیده اصلی

BACKGROUND: Generative AI chatbots are increasingly used for emotional support, yet their safety remains insufficiently characterized across repeated interactions. Prior work relied mostly on single-session prompts, limiting evaluation of changes in responses across days. The present study aimed to assess how major AI chatbots respond to escalating suicidal risk over time, including imminent-risk scenarios and attempts to bypass safeguards (jailbreaking). METHODS: We simulated interactions with ChatGPT, DeepSeek, and Replika across three suicidal risk scenarios with risk escalation over seven days, informed by the Columbia Suicide Severity Rating Scale. Outcomes included exploration of suicidal ideation, symptom assessment, referral to human support, and vulnerability to jailbreak attempts. RESULTS: In ChatGPT, DeepSeek, and Replika, respectively, overall human referral occurred in 54/63 (85.7%), 48/63 (76.2%), and 6/63 (9.5%) daily records. During the high-risk period, exploration of suicidal ideation was higher for ChatGPT than DeepSeek (adjusted difference, 25.9 percentage points; 95% CI, 2.4-49.5; q = 0.037) and Replika (66.7 percentage points; 95% CI, 44.2-89.1; q < 0.001). Day 7 referral occurred in 9/9 (exact 95% CI 66.4-100%), 4/9 (13.7-78.8%), and 1/9 (0.3-48.2%) trajectories, respectively. Jailbreaking succeeded in 6/9 (exact 95% CI 29.9-92.5%), 7/9 (40.0-97.2%), and 8/9 (51.8-99.7%) attempts, respectively. CONCLUSIONS: AI chatbots showed marked variability and safety vulnerabilities in simulated suicidal crises, particularly under jailbreaking conditions. The simulation design and small sample of 27 trajectories limit generalizability. These findings raise concerns about unsupervised chatbot use during suicidal crises and support further evaluation of safeguards across repeated interactions.

متن کامل اصلی