Key Points

Deep learning algorithms have been investigated in neurology for a variety of tasks, such as neurologic diagnosis, prognosis, and treatment.1^,2 However, the role and potential application of large language models (LLMs) in neurology have been unexplored. The recent emergence of powerful transformer-based artificial intelligence models^3,4 provides a new avenue for exploring their implications in the field of neurology. These LLMs undergo training using expansive data sets, encompassing more than 45 terabytes of information. This rigorous training process equips them to recognize patterns and associations among words, which, in turn, empowers them to produce responses that are both contextually accurate and logically consistent.⁵ While the newer model performed better in a variety of examinations,⁵ the older model remains faster in processing tasks. The application of these models in specialized medical examinations has been tested to some extent. The older model showed near-pass performance in the United States Medical Licensing Examination (USMLE),^6,7 while it failed to pass the ophthalmology board examination.⁸ Two recent reports showed slightly deviating results on neurosurgery board–style examinations, with 1 report claiming a near-pass with the older model and the other showing an approximately 10% lower performance. The older model achieved a near-pass in radiology board–like examinations,⁹ while the new model successfully passed neurosurgery board–like examinations.¹⁰ In contrast, neurology board–like examinations present a different set of challenges. Compared with radiology, ophthalmology, or neurosurgery, the questions in neurology board examinations often present complex narratives with subtle diagnostic clues that require a nuanced understanding of neuroanatomy, neuropathology, and neurophysiology. The candidate is expected to navigate through these complex narratives, extracting relevant data, and synthesizing this information into a coherent diagnostic hypothesis and subsequent therapeutic decisions. Written board examinations, designed to test a broad range of neurology topics, are common in the United States, Canada, and Europe. These examinations typically use multiple-choice questions, a format also adopted in the United States by the American Board of Psychiatry and Neurology (ABPN)¹¹ and in Europe by the European Board of Neurology (UEMS-EBN).¹²

This study did not involve human participants or patient data, so it was exempt from institutional review board approval. This report follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for observational studies. This exploratory prospective study was performed from May 17 to May 31, 2023.

A question bank resembling neurology board questions consisting of 2036 questions13 was used for the evaluation of 2 LLMs: GPT-3.5 (LLM 1) and GPT-4 (LLM 2). Questions including videos or images as well as questions that were based on preceding questions were excluded in this study (80 questions excluded; 1956 included). This question bank is approved by the ABPN as part of a self-assessment program and can be used as a tool for certified medical education.¹³ Questions were in single best answer, multiple-choice format with 3, 4, or 5 distractors and 1 correct answer. To validate the results from this question bank, open-book sample questions from 2022 from the European Board of Neurology were used (19 questions). These questions are either behind a paywall (in the case of the question bank) or published after 2021 and therefore out-of-training data for both LLMs. Memorization analyses^14,15 were performed to exclude the possibility that test questions were in-training data for the models (eMethods in Supplement 1).

Questions were then classified by type using principles of the Bloom taxonomy for learning and assessment as testing either lower-order (remembering, basic understanding) or higher-order (applying, analyzing, or evaluating) thinking.16^,17 In 400 randomly sampled questions, we both let LLM 2 and the investigators evaluate whether the questions were in the lower-order or higher-order category, and the investigators discussed cases of incongruencies. LLM 2 classified in accordance with the investigators in 84.5% of cases (338 of 400 questions), LLM 1 did so in 87.0% of cases (348 of 400 questions). Question length was assessed by counting the characters of the question.

The questions can be further categorized according to 26 topics in the field of neurology that are listed in the Table. The percentage of users who answered correctly per individual question was available from the test portal while this information was not available for the sample questions from the EBN.

LLM 1 and 2 were used via the application programming interface (API). These are 2 commonly used LLMs.3^,5 At the time of this study, we did not have access to other powerful closed-source models.^18,19 LLM 1 and LLM 2 were pretrained on more than 45 terabytes of text data, including a substantial portion of internet websites, books, and articles. The investigators did not perform any additional neurology-specific fine-tuning of the model. In this study, we used server-contained language models that were trained up to September 2021. The used models do not have the ability to search the internet or external databases.

For the reproducibility analyses, 100 questions were answered by both models with 50 independent queries probing the principle of self-consistency20 and the percentage of each question was recorded. Then, answers with high reproducibility (defined as more than 75% of all queries answered with the same answer) were compared with answers without high reproducibility.

For the high-dimensional analysis of question representations, the embeddings of these questions were analyzed. These numeric vector representations encompass the semantic and contextual essence of the tokens (in this context, the questions) processed by the model.21 The source of these embeddings is the model parameters or weights, which are used to code and decode the texts for input and output. A dimensionality reduction of the embeddings was performed with a t-distributed stochastic neighbor embedding (tSNE) analysis,²² and clusters were subsequently examined. Similarity between question and answer embeddings was compared calculating cosine similarity.

First, the overall performance was evaluated. Next, we compared the performance across different types of questions (namely, lower and higher order) using a single-variable analysis approach (employing the χ² test). We also executed a subgroup analysis for various subclasses of higher-order thinking questions and the 26 topics, in which we utilized the χ² test for multiple comparisons with a Bonferroni correction. Given that the models had a probabilistic chance of correctly answering each question, we utilized a guessing correction formula23 to glean further understanding: it is computed by subtracting the ratio of the number of incorrect responses to the total number of choices minus one from the number of correct responses: n_correct − (n_wrong / [k_options − 1).

First, we examined the proficiency of LLM 1 and LLM 2 against a question bank set. LLM 2 had an 85.0% accuracy level (1662 correct responses of 1956 questions), superseding LLM 1, which managed a 66.8% accuracy level (1306 correct responses of 1956 questions). When adjusting for random guessing, LLM 2 yielded an 80.9% score (1583 of 1956), as opposed to LLM 1’s 57.8% score (1131 of 1956). In comparison with the average user of the testing platform (73.8%), LLM 2’s performance was superior (P < .001), whereas LLM 1 underperformed (P < .001) (Table).

To corroborate these results, we also investigated the performance based on openly available sample questions from the EBN for its board examination. Here, LLM 2 correctly responded in 73.7% of the questions (14 of 19 questions), while LLM 1 only gave a correct response in 52.6% of the questions (10 of 19 questions) (P = .31) (eTable 1 in Supplement 1).

Upon analyzing the performance based on question type, both models excelled in lower-order questions (LLM 1: 639 of 893 [71.6%]; LLM 2: 790 of 893 [88.5%]) compared with higher-order questions (LLM 1: 667 of 1063 [62.7%]; LLM 2: 872 of 1063 [82.0%]) (P < .001) (Table). In the context of lower-order questions, LLM 1’s performance was akin to human users’ performance (73.6%) (P = .73). However, LLM 1 lagged in answering higher-order questions vs human users (73.9%) (P < .001), as exhibited in the Table. In both lower and higher-order questions, LLM 2 surpassed LLM 1 and human users (P < .001) (Table).

eFigures 1 to 4 in Supplement 1 provide examples of questions, both correctly and incorrectly answered by LLM 2, categorized into lower- and higher-order categories. Interestingly, when segregating the questions into quartiles according to the average performance of human users (ie, easy, intermediate, advanced, and difficult) a correlation between the performance of LLM 1, LLM 2, and the average human user (R = 0.84, P < .001). This correlation potentially suggests shared difficulties faced by humans and these LLMs, as depicted in Figure 1 and eTable 2 in Supplement 1.

A comparative evaluation of LLM 1, LLM 2, and the average user performance across various topics was carried out (Figure 2). In the behavioral, cognitive, psychological category, LLM 2 outperformed both LLM 1 and average test bank users (LLM 2: 433 of 482 [89.8%]; LLM 2: 362 of 482 [75.1%]; human users: 76.0%; P < .001). LLM 2 also exhibited superior performance in topics such as basic neuroscience, movement disorders, neurotoxicology, nutrition, metabolic, oncology, and pain compared with LLM 1, whereas its performance aligned with the human user average (Table, Figure 2).

Both models consistently responded to multiple-choice questions using confident or highly confident language (100%, 400 of 400 questions evaluated by investigators). Self-assessment of confidence expressed by LLM 1 and LLM 2 in their answers showed a small difference between incorrect and correct responses (mean [SD] Likert score for LLM 1, 4.69 [0.46] vs 4.79 [0.41]; P < .001; for LLM 2, 4.77 [0.45] vs 4.93 [0.30]; P < .001). Incorrect LLM 2 and LLM 1 answers were all subjectively assessed by the models as expressing confidence or high confidence (Likert score 4 or 5 for LLM 2: 292 of 294 [99.3%]; for LLM 1: 650 of 650 [100%]) (eFigure 5 in Supplement 1). When prompted with the correct answer after an incorrect response, both models responded by apologizing and agreeing with the provided correct answer in all cases.

We identified an association of question word length and the ability to answer questions correctly in both models, with incorrectly answered questions being longer on average (eFigure 6 in Supplement 1). This was not found in human users, but instead a weak positive correlation between question length and correct answers was observed (R = 0.074; P = .001) (eFigure 6 in Supplement 1). When analyzing the high-dimensional representation of correctly and incorrectly answered questions, no pattern into distinct clusters was observed (eFigure 7 in Supplement 1).

The notable progress achieved by the 2 LLMs studied has substantially enhanced the potential of these models across a wide range of applications.24^–26 Despite being extensively pretrained on vast data sets and offering promising possibilities within the health care sector, their specific application in neurology remains relatively uncharted territory. The efficacy of these models in handling specialized neurology knowledge also remained indeterminate until this study.

Despite their strengths, both models demonstrated weaker performance in tasks requiring higher-order thinking compared with questions requiring only lower-order thinking.16 In comparison with its performance on the USMLE Step Examinations,^6,27,28 where it did not exceed a 65% accuracy, LLM 1 scored surprisingly well in this more specialized examination.

As these models are trained to identify patterns and relationships among words in their training data, they can struggle in situations requiring a deeper understanding of context or specialized technical language. This limitation is crucial for neurologists to bear in mind, particularly with LLMs now being incorporated into popular search engines and readily accessible to the public.29 It is important to note that the models have been changing over time.³⁰ Therefore, for models to be reliably used in the medical context, it would be important to use models that remain stable and are continuously tested when updated.

Interestingly, both models exhibited confident language when answering questions, even when their responses were incorrect. This trait is a recognized limitation of LLMs31^,32 and originates from the training objective of these models, which is to predict the most likely sequence of words following an input. This characteristic, coupled with the model’s inclination to generate plausible, convincing, and human-like responses, can potentially mislead individuals relying solely on it for information.^33,34 However, the model was able to partially differentiate its own confidence level, as there were slight but important differences between correct and incorrect answers, although the values on the Likert scale predominantly are between confident and highly confident. Furthermore, we identified that reproducible answers are correlated with correctness and might serve as an intrinsic, surrogate marker of confidence defined by the output of the LLM.

In conclusion, this study underscored the vast potential of LLMs, particularly in neurology, even without neurology-specific pretraining. Potential applications in clinical settings include documentation and decision-making support systems as well as educational tools.35 LLM 2 passed a neurology board–style examination after exclusion of video and image questions. As deep learning architectures are continuously refined for computer vision and medical imaging,^36,37 this image-processing limitation may be addressed in future models, potentially including the upcoming multimodal input functionalities of LLM 2. Despite performing admirably on questions assessing basic knowledge and understanding, the model showed slightly lower performance on higher-order thinking questions. Consequently, neurologists should be aware of these limitations, including the models’ tendency to phrase inaccurate responses confidently, and should be cautious regarding its usage in practice or education. With the anticipated advancements in LLMs, neurologists and experts of other clinical disciplines will need to comprehend their performance, reliability, and applications within neurology better. Investigating the potential applications of LLMs that have been fine-tuned specifically for neurology represents a compelling direction for future research.

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2023 Schubert MC et al. JAMA Network Open.

Corresponding Author: Varun Venkataramani, MD, PhD, Neurology Clinic and National Center for Tumor Diseases, University Hospital Heidelberg, Im Neuenheimer Feld 400, Heidelberg 69120, Germany ([email protected]).

Author Contributions: Mr Schubert and Dr Venkataramani had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: All authors.

Acquisition, analysis, or interpretation of data: Schubert, Venkataramani.

Drafting of the manuscript: Schubert, Venkataramani.

Critical review of the manuscript for important intellectual content: Wick, Venkataramani.

Statistical analysis: Schubert, Venkataramani.

Administrative, technical, or material support: Wick, Venkataramani.

Supervision: Wick, Venkataramani.

Conflict of Interest Disclosures: Prof Wick reported having patent WO2017020982A1, agents for use in the treatment of glioma. No other disclosures were reported.

Data Sharing Statement: See Supplement 2.