Socrates and psychotherapists in artificial intelligence: evaluating state-of-the-art AIs’ ability in mastering mental health therapeutic technique

(Word count: 820, 3-minute read)

Have you ever tried to ‘talk’ to ChatGPT, the latest generation of language-producing artificial intelligence? Are you seeking comfort and wondering if AIs could help you improve your mental health? Are you a therapist wondering about the future landscape of your profession? Or are you a developer who wants to build a mental health application using generative artificial intelligence? The idea of AI therapist might sound like a page from a sci-fi novel, but in fact, it’s becoming our reality. 

We have found that ChatGPT4 could perform the Socratic method as effectively as a human therapist.

Now, you might be wondering, “What exactly is the Socratic method?” It is a helpful questioning technique often used in psychotherapy. It involves therapists asking thought-provoking questions. These questions, a.k.a. Socratic questions, guide patients to self-evaluate their unhealthy and irrational thoughts. These thoughts are often at the root of mental disorders. In therapy, this is how Socratic questions work: someone might blame themselves for being in an abusive relationship. Or they might think themselves a “complete failure” for failing just one exam. In this case, the therapists might ask, “Do you think you choose to be in an abusive relationship?” or “Could one exam determine your success?”. These questions can help the client take a step back and objectively examine reality and irrational beliefs.

But how has therapy been integrated into language-producing AIs?  In fact, a few AI therapy tools have been available since 2017 (examples are Woebot, and Tess). However, their effectiveness in improving mental health is questionable (you can learn more from this news report or delve into the nuances with this academic paper). Also, some tools respond inappropriately to abuse victims, drug abusers, and eating disorder patients. 

Recently though, technical breakthroughs in AI have made them better at language tasks. This leads us to ask: could AI produce high-quality and appropriate Socratic questions? 

Our Study: Pitting AI Against Human Therapists

To understand the potential of AI in therapy, we conducted a study. We provided six certified psychotherapists with 60 written descriptions of unhealthy thoughts. Each description was paired with Socratic questions generated by three different sources. The sources are: other human therapists, ChatGPT4 (the most advanced language AI so far), and SocrAItes (an AI trained by our research team based on OPT, a model developed by MetaAI). 

The therapists were asked to rate the quality and appropriateness of each question without knowing its source. The quality rating is about how helpful the question is in exploring unhealthy and irrational thoughts. Also, the appropriateness rating considers whether the question is insensitive. In both ratings, 0 means ‘extremely bad’ and 6 means ‘extremely good”.

Results: A Surprising Victor

The results show that SocrAItes was rated significantly worse in both appropriateness and quality ratings. ChatGPT4, on the other hand, had the best output, which was rated significantly superior to therapists in both ratings.

Implications: What Does This Mean for Mental Health Care?

The study showed that:

• SocrAItes failed to perform as well as human therapists or ChatGPT4 in quality and appropriateness ratings
• ChatGPT4 outperformed even the therapists in both ratings for the questions produced

The results hint that: 

• Using more advanced models might be more efficient for building AI therapy tools
• ChatGPT4 has the potential to be used to improve mental health using the Socratic method

However, these results do not mean that ChatGPT4 can fully replace therapists. Although ChatGPT4 is better at performing Socratic questions, it was only tested on one of the many tools therapists use. More research is needed to understand how ChatGPT4 uses a wide range of tools, such as goal-setting and problem-solving.

Lastly, ChatGPT4 must be tested in a real therapy context to understand its efficacy in mental health treatments. Previous AI mental health applications have high drop-out rates (see this review). This is because they do not offer the guidance and motivation that therapists provide. It is still unclear how the efficacy and adherence of ChatGPT4 will be. 

Key Take-Aways

• AI, particularly ChatGPT4, has the potential to conduct psychotherapy
• More research is needed to fully test ChatGPT4 in the context of real therapy

Let me know what you think!

If you are a patient seeking mental health care, would you want to ‘talk’ to an AI instead of a human being? What are some of your considerations?

If you are a therapist, are you worried about being replaced by AI? More importantly, what is the therapy fundamentally offering, and can AIs offer that to the same quality?

If you are a developer of mental health applications, what do you think are the biggest technical challenges in building a therapeutic AI? 

Please leave your thoughts on AI mental health applications below. I will make sure to reply before ChatGPT4 does that for me. 

References

1. Abd-Alrazaq, A. A., Alajlani, M., Alalwan, A. A., Bewick, B. M., Gardner, P., & Househ, M. (2019). An overview of the features of chatbots in mental health: A scoping review. International Journal of Medical Informatics, 132, 103978. Retrieved from https://www.sciencedirect.com/science/article/abs/pii/S1386505619307166
2. Browne, G. (2022). The Problem With Mental Health Bots. Retrieved from https://www.wired.com/story/mental-health-chatbots/
3. Cherry, K. (2022). What Is Cognitive Behavioral Therapy (CBT)? Retrieved from https://www.verywellmind.com/what-is-cognitive-behavior-therapy-2795747#toc-cognitive-behavioral-therapy-techniques
4. Clark, G. I., & Egan, S. J. (2018). Clarifying the role of the Socratic method in CBT: A survey of expert opinion. International Journal of Cognitive Therapy, 11, 184-199. Retrieved from  https://link.springer.com/article/10.1007/s10608-015-9707-3
5. Gual-Montolio, P., Jaén, I., Martínez-Borba, V., Castilla, D., & Suso-Ribera, C. (2022). Using Artificial Intelligence to Enhance Ongoing Psychological Interventions for Emotional Problems in Real-or Close to Real-Time: A Systematic Review. International journal of environmental research and public health, 19(13), 7737. Retrieved from https://pubmed.ncbi.nlm.nih.gov/35805395/
6. OpenAI. (2023). ChatGPT4 (Version May 12 2023). Retrieved from https://chat.openai.com/?model=gpt-4
7. Rosenthal, E. (2023). Analysis: Chatbots for mental health care are booming, but there’s little proof that they help. Retrieved from https://edition.cnn.com/2023/05/19/health/chatbots-mental-health-partner-kff-health-news-analysis/index.html
8. Simon, J. (2021). Large Language Models: A New Moore’s Law. In: HuggingFace. Retrieved from https://huggingface.co/blog/large-language-models.

Credits

Cover picture from Shutterstock: www.shutterstock.com/image-vector/robot-psychologist-man-reception-cyborg-psychotherapist-1142553431

In-text meme created by the author using www.imgflip.com