A new study published in arXiv provides some of the first empirical evidence that large language models like ChatGPT can be used to improve the persuasiveness of messages in human communication. The research focused specifically on whether consumers were using ChatGPT to write more effective complaints to financial companies.
Analyzing a database of over 780,000 consumer complaints filed with the Consumer Financial Protection Bureau (CFPB), the authors found that the number of complaints likely written with assistance from AI surged after the release of ChatGPT in late 2022. Using an AI detection tool called Winston AI, they identified which complaints had a high probability of being written with ChatGPT.
The study then compared the linguistic features between complaints deemed as “Likely-AI” versus those by “Likely-Humans.” Key findings showed Likely-AI complaints were more coherent and polite, conveyed more negative sentiment, and were less readable.
Financial firms were over 5% more likely to provide relief for Likely-AI complaints, even after controlling for the complaint sub-issues and linguistic qualities.
To test whether using ChatGPT could directly cause an increase in complaint persuasiveness, the researchers conducted an online experiment. They had participants evaluate financial complaints either in their original form or after being edited by ChatGPT to enhance qualities like clarity, coherence, and professionalism.
The results revealed that hypothetical compensation likelihood was higher for ChatGPT-edited complaints compared to original ones.
This points in the direction that complaints edited by ChatGPT are more effective in achieving compensation than complaints in their original, unedited form.
This early research highlights ChatGPT’s ability to improve the persuasiveness of human communication in real-world settings like consumer finance. The authors say that large language models could have transformative impacts on communication outcomes ranging far beyond just writing more compelling complaints.
However, more studies are still needed to fully understand the breadth of these potential influences on human behavior.
