Artificial general intelligence for radiation oncology

doi: 10.1016/j.metrad.2023.100045.

Epub 2023 Nov 24.

Chenbin Liu¹, Zhengliang Liu², Jason Holmes³, Lu Zhang⁴, Lian Zhang³, Yuzhen Ding³, Peng Shu², Zihao Wu², Haixing Dai², Yiwei Li², Dinggang Shen^{5

6

7}, Ninghao Liu², Quanzheng Li⁸, Xiang Li⁸, Dajiang Zhu⁴, Tianming Liu², Wei Liu³

Affiliations

¹ Department of Radiation Oncology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital & Shenzhen Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Shenzhen, Guangdong, China.
² School of Computing, University of Georgia, USA.
³ Department of Radiation Oncology, Mayo Clinic, USA.
⁴ Department of Computer Science and Engineering, The University of Texas at Arlington, USA.
⁵ School of Biomedical Engineering, ShanghaiTech University, China.
⁶ Shanghai United Imaging Intelligence Co., Ltd, China.
⁷ Shanghai Clinical Research and Trial Center, China.
⁸ Department of Radiology, Massachusetts General Hospital and Harvard Medical School, USA.

Item in Clipboard

Chenbin Liu et al.

Meta Radiol.

2023 Nov.

doi: 10.1016/j.metrad.2023.100045.

Epub 2023 Nov 24.

Authors

Affiliations

¹ Department of Radiation Oncology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital & Shenzhen Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Shenzhen, Guangdong, China.
² School of Computing, University of Georgia, USA.
³ Department of Radiation Oncology, Mayo Clinic, USA.
⁴ Department of Computer Science and Engineering, The University of Texas at Arlington, USA.
⁵ School of Biomedical Engineering, ShanghaiTech University, China.
⁶ Shanghai United Imaging Intelligence Co., Ltd, China.
⁷ Shanghai Clinical Research and Trial Center, China.
⁸ Department of Radiology, Massachusetts General Hospital and Harvard Medical School, USA.

Item in Clipboard

Abstract

The emergence of artificial general intelligence (AGI) is transforming radiation oncology. As prominent vanguards of AGI, large language models (LLMs) such as GPT-4 and PaLM 2 can process extensive texts and large vision models (LVMs) such as the Segment Anything Model (SAM) can process extensive imaging data to enhance the efficiency and precision of radiation therapy. This paper explores full-spectrum applications of AGI across radiation oncology including initial consultation, simulation, treatment planning, treatment delivery, treatment verification, and patient follow-up. The fusion of vision data with LLMs also creates powerful multimodal models that elucidate nuanced clinical patterns. Together, AGI promises to catalyze a shift towards data-driven, personalized radiation therapy. However, these models should complement human expertise and care. This paper provides an overview of how AGI can transform radiation oncology to elevate the standard of patient care in radiation oncology, with the key insight being AGI’s ability to exploit multimodal clinical data at scale.

Keywords:

AGI; Large foundation model; Medical imaging; Radiation oncology; SAM.

PubMed Disclaimer

Conflict of interest statement

Declaration of interests The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. The Author Tianming Liu is the Editor in Chief of the journal, the authors Wei Li, Xiang Li, Dajiang Zhu and Dinggang Shen are the Editorial Board Members of the journal, but were not involved in the peer review procedure. This paper was handled by another Editor Board member.

Figures

**Fig. 1.**

Large Foundational Models for Radiation Oncology. The left half of the revised figure represents the existing challenge at hand, while the right half portrays the proposed solution provided by the large foundational models in radiation oncology.

**Fig. 2.**

The accuracy of GPT-4 in re-labeling structure names according to the TG-263 report.

**Fig. 3.**

Large Multimodal Foundation Models for Radiation Oncology. Here we use a brain tumor case as an example and incorporate a visual representation of all data sources.

**Fig. 4.**

AI based treatment planning workflow. a) KBP method b) Dose Prediction method. The dotted box means an improved enhancement for the workflow.

**Fig. 5.**

Synthetic CT generation using deep learning.

**Fig. 6.**

AGI for Image registration.

References

1. Subir Nag, David Beyer, Jay Friedland, Peter Grimm, Ravinder Nath. American brachyther- apy society (abs) recommendations for transperineal permanent brachytherapy of prostate cancer. Int J Radiat Oncol Biol Phys. 1999;44(4):789–799.
  
  –
  
  PubMed
1. Lott JS, Smith Ivan H. Cobalt-60 beam therapy in carcinoma of the esophagus. Radiology. 1958;71(3):321–326.
  
  –
  
  PubMed
1. Siebers JV, Gardner JK, Gordon JJ, Wang S, Ververs JD. Quantification of exit fluence variations and implications for exit fluence-based dose reconstruction based. Int J Radiat Oncol Biol Phys. 2008;72(1):S552.
1. Radhe Mohan, David Grosshans. Proton therapy–present and future. Adv Drug Deliv Rev. 2017;109:26–44.
  
  –
  
  PMC
  
  –
  
  PubMed
1. Schild Steven E, Rule William G, Ashman Jonathan B, Vora Sujay A, et al. Proton beam therapy for locally advanced lung cancer: a review. World J Clin Oncol. 2014; 5(4):568.
  
  –
  
  PMC
  
  –
  
  PubMed