an Evaluation in NLP Concepts

Download a PDF of the paper titled Large Language Models on Wikipedia-Style Survey Generation: an Evaluation in NLP Concepts, by Fan Gao and 9 other authors

Download PDF
HTML (experimental)

Abstract:Educational materials such as survey articles in specialized fields like computer science traditionally require tremendous expert inputs and are therefore expensive to create and update. Recently, Large Language Models (LLMs) have achieved significant success across various general tasks. However, their effectiveness and limitations in the education domain are yet to be fully explored. In this work, we examine the proficiency of LLMs in generating succinct survey articles specific to the niche field of NLP in computer science, focusing on a curated list of 99 topics. Automated benchmarks reveal that GPT-4 surpasses its predecessors like GPT-3.5, PaLM2, and LLaMa2 in comparison to the established ground truth. We compare both human and GPT-based evaluation scores and provide in-depth analysis. While our findings suggest that GPT-created surveys are more contemporary and accessible than human-authored ones, certain limitations were observed. Notably, GPT-4, despite often delivering outstanding content, occasionally exhibited lapses like missing details or factual errors. At last, we compared the rating behavior between humans and GPT-4 and found systematic bias in using GPT evaluation.

Submission history

From: Irene Li [view email]
Mon, 21 Aug 2023 01:32:45 UTC (622 KB)
Wed, 6 Sep 2023 00:03:11 UTC (796 KB)
Thu, 22 Feb 2024 02:54:19 UTC (1,336 KB)

Source link

Leave a Reply

Your email address will not be published. Required fields are marked *