Can generative AI infer thinking style from language? Evaluating the utility of AI as a psychological text analysis tool

doi: 10.3758/s13428-024-02344-0.

Online ahead of print.

Affiliations

Item in Clipboard

David M Markowitz.

Behav Res Methods.

2024.

Abstract

Generative AI, short for Generative Artificial Intelligence, a class of artificial intelligence systems, is not currently the choice technology for text analysis, but prior work suggests it may have some utility to assess dynamics like emotion. The current work builds upon this empirical foundation to consider how analytic thinking scores from a large language model chatbot, ChatGPT, were linked to analytic thinking scores from dictionary-based tools like Linguistic Inquiry and Word Count (LIWC). Using over 16,000 texts from four samples and tested against three prompts and two large language models (GPT-3.5, GPT-4), the evidence suggests there were small associations between ChatGPT and LIWC analytic thinking scores (meta-analytic effect sizes: .058 < rs < .304; ps < .001). When given the formula to calculate the LIWC analytic thinking index, ChatGPT performed incorrect mathematical operations in 22% of the cases, suggesting basic word and number processing may be unreliable with large language models. Researchers should be cautious when using AI for text analysis.

Keywords:

Analytic thinking; Generative AI; LIWC; Large language models; Text analysis.

PubMed Disclaimer

References

1. Alkaissi, H., McFarlane, S. I., Alkaissi, H., & McFarlane, S. I. (2023). Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus, 15(2). https://doi.org/10.7759/cureus.35179
1. arXiv. (2023). arXiv Dataset. https://www.kaggle.com/datasets/Cornell-University/arxiv
1. Blackburn, K. G., Yilmaz, G., & Boyd, R. L. (2018). Food for thought: Exploring how people think and talk about food online. Appetite, 123, 390–401. https://doi.org/10.1016/j.appet.2018.01.022
  
  –
  
  DOI
  
  –
  
  PubMed
1. Borji, A. (2023). A categorical archive of ChatGPT failures (arXiv:2302.03494). arXiv. https://doi.org/10.48550/arXiv.2302.03494
  
  –
  
  DOI
1. Boyd, R. L., Ashokkumar, A., Seraj, S., & Pennebaker, J. W. (2022). The development and psychometric properties of LIWC-22. University of Texas at Austin https://www.liwc.app/