Evaluating the performance of large language models in haematopoietic stem cell transplantation decision-making

doi: 10.1111/bjh.19200.

Online ahead of print.

Ivan Civettini^{1

2}, Arianna Zappaterra^{1

2

3}, Bianca Maria Granelli^{1

2}, Giovanni Rindone^{1

2}, Andrea Aroldi², Stefano Bonfanti^{1

2}, Federica Colombo^{1

2}, Marilena Fedele², Giovanni Grillo³, Matteo Parma², Paola Perfetti², Elisabetta Terruzzi², Carlo Gambacorti-Passerini^{1

2}, Daniele Ramazzotti¹, Fabrizio Cavalca²

Affiliations

Item in Clipboard

Ivan Civettini et al.

Br J Haematol.

2023.

doi: 10.1111/bjh.19200.

Online ahead of print.

Authors

Affiliations

¹ Department of Medicine and Surgery, University of Milano-Bicocca, Monza, Italy.
² Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy.
³ Department of Haematology and Bone Marrow Transplantation Unit, ASST Grande Ospedale Metropolitano Niguarda, Milan, Italy.

Item in Clipboard

Abstract

In a first-of-its-kind study, we assessed the capabilities of large language models (LLMs) in making complex decisions in haematopoietic stem cell transplantation. The evaluation was conducted not only for Generative Pre-trained Transformer 4 (GPT-4) but also conducted on other artificial intelligence models: PaLm 2 and Llama-2. Using detailed haematological histories that include both clinical, molecular and donor data, we conducted a triple-blind survey to compare LLMs to haematology residents. We found that residents significantly outperformed LLMs (p = 0.02), particularly in transplant eligibility assessment (p = 0.01). Our triple-blind methodology aimed to mitigate potential biases in evaluating LLMs and revealed both their promise and limitations in deciphering complex haematological clinical scenarios.

Keywords:

GPT; HSC transplantation; artificial intelligence; interrater agreement; transplant.

PubMed Disclaimer

References

REFERENCES

1. Thirunavukarasu AJ, Hassan R, Mahmood S, Sanghera R, Barzangi K, El Mukashfi M, et al. Trialling a large language model (ChatGPT) in general practice with the applied knowledge test: observational study demonstrating opportunities and limitations in primary care. JMIR Med Educ. 2023;9:e46599.
1. Hoch CC, Wollenberg B, Lüers J-C, Knoedler S, Knoedler L, Frank K, et al. ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. Eur Arch Otorhinolaryngol. 2023;280(9):4271-4278.
1. Jin D, Pan E, Oufattole N, Weng W-H, Fang H, Szolovits P. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. arxiv.org Cornell University 2020.
1. Jin Q, Dhingra B, Liu Z, Cohen WW, Lu X. PubMedQA: a dataset for biomedical research question answering. arXiv. 2019.
1. Hendrycks D, Burns C, Basart S, Zou A, Mazeika M, Song D, et al. Measuring massive multitask language understanding. arXiv. 2020.