Los Angeles — New research presented today at the American Psychiatric Association’s Annual Meeting compared an AI therapist and a human therapist based on their delivery of text-based cognitive behavioral therapy (CBT), finding that human therapists excelled over the chatbot.

In the study, 75 mental health professionals and trainees completed a cross-sectional survey in which participants gauged two text-based CBT transcripts, one from AI and one from a human therapist, using the Cognitive Therapy Rating Scale. Participants provided qualitative feedback on the transcripts and evaluated each one using a standardized scale. Participants gauged the quality of elements of CBT such as agenda-setting (listing tasks to be completed in the therapy session and ensuring all agenda items are completed) and guided discovery (helping the patient assess data from their own life to learn about themself). The therapist and bot evaluated identical clinical scenarios to provide consistency.

Twenty-nine percent of the survey participants rated human therapists as highly effective, whereas less than 10% of participants gave the AI therapist the same rating. More than half (52%) of participants scored the human therapist’s agenda-setting skills highest, whereas 28% did the same for the AI therapist. One in four (24%) participants gave the human therapists a high score in guided discovery, but only 12% scored the AI therapist similarly on the same element.

Despite receiving similar ratings to human therapists in understanding patients’ internal reality, the AI therapist was viewed as more rigid and impersonal. The researchers conclude that AI-based therapy is not appropriate for standalone use, although it may serve as an adjunct to therapy provided by humans.

“While our study found that AI-driven text-based cognitive behavioral therapy can apply basic therapy structures, human therapists outperformed ChatGPT-3.5 in key domains such as agenda-setting, eliciting feedback, and applying CBT techniques,” said Esha Aneja of the research team. “These findings underscore that although ChatGPT-3.5 shows potential, it currently lacks the nuanced empathy and therapeutic alliance that characterize effective human therapy.”

The study authors included Sebastian Acevedo, M.D., M.P.H.; Douglas Opler, M.D.; and Eric Jarmon, M.D., of Rutgers New Jersey Medical School, and Esha Aneja, MS4, of California Northstate University College of Medicine.

American Psychiatric Association

The American Psychiatric Association, founded in 1844, is the oldest medical association in the country. The APA is also the largest psychiatric association in the world with more than 39,200 physician members specializing in the diagnosis, treatment, prevention, and research of mental illnesses. APA’s vision is to ensure access to quality psychiatric diagnosis and treatment. For more information, please visit www.psychiatry.org.