Study objectives and endpoints

The primary objective of this study is to assess the prognostic accuracy and the clinical impact of the RISK^INDEX prediction model. Secondary objectives of this study are to assess: (1) the number and type of changes in medical treatment after presentation of the RISK^INDEX to the ED physician; and (2) the prognostic accuracy of the RISK^INDEX as compared to that of the physician’s clinical intuition and that of other clinical prediction models (e.g. MEWS, SOFA, RISE UP score).

Trial design and setting

The MARS-ED study is designed as an investigator-initiated, open-label, randomized, non-inferiority, pilot clinical trial. Adult patients presenting to the ED who are primarily assessed and treated by an internal medicine specialist will be randomized in a 1:1 ratio to the control group or the intervention group. This study is not blinded, since the physician needs to be informed of the RISK^INDEX in order to assess the size of the clinical impact of the RISK^INDEX.

The study will be conducted in the ED of the MUMC+, which is a secondary/tertiary care medical center in the Netherlands, with 5,500 patients visiting the ED for assessment by an internal medicine specialist each year. In contrast to many other countries where patients can visit the ED without referral by a health care professional (open access ED), nearly all ED patients in the Netherlands after referred after an initial triage process by a general practitioner, a medical specialist or ambulance.

This study protocol is designed in accordance with the Standard Protocol Items: Recommendations for Interventional Trials – Artificial Intelligence (SPIRIT-AI) and the Consorted Standards of Reporting Trials (CONSORT) guidelines (Supplementary Table 1) [17, 18]. This study has been approved by the medical ethical committee (METC) of the MUMC + on June 21st 2022 (METC 21–068) and is registered at clinicaltrials.gov (NCT05497830). The study will be conducted according to the principles of the Declaration of Helsinki (version 2013, July 9th 2018) and in accordance with the standards of Good Clinical Practice. The study is expected to last from September 2022 to September 2024.

Recruitment and selection of eligible patients

All adult ED patients (18 years or older) who are assessed and treated by internal medicine specialist or their residents and who meet the inclusion criteria will be asked informed consent to participate in this study by a member of the research team.

As patients may be unconscious, in a state of delirium or otherwise unable to provide informed consent, a legal representative may make the decision on participation as well. In case of absence of such representative, a deferred consent procedure may take place, in which consent is implied and randomization will take place according to the study protocol and the patient will receive medical care as usual [19]. Within 72 h after randomization, a member of the research team will contact the patient to ask for informed consent. If this is not possible, e.g. because the patient has already been discharged from the hospital before a member of the research team was able to contact the patient, the patient dies before giving informed consent, or the patient (or representative) remains unable (or unavailable) to give informed consent for more than 72 h after randomization, data of this patient will be deleted.

Study intervention and procedure

When a patient enters the ED for assessment and treatment by an internal medicine specialist, the patient will immediately be assessed for eligibility and randomized as soon as informed consent is obtained. Obtaining informed consent is performed by a member of the research team and involves explaining the study intervention and procedure to the patient and handing out a patient information form. The inclusion criteria and exclusion criteria are shown in Table 1.

After inclusion, the patient is allocated to either the intervention group or the control group through randomization. An overview of the patient’s timeline is shown in Supplemental Fig. 1. Then, the patients will receive medical care in the ED as usual. After complete assessment of the patient, in both the intervention group and control group, the physicians will be asked questions regarding their clinical intuition (Table 2). In the intervention group, the RISK^INDEX will be presented to the physician together with the average mortality risk for ED patients with the same age and sex, and subsequently questions will be asked concerning agreement of their clinical intuition with the RISK^INDEX and whether the presented RISK^INDEX leads to changes in the treatment plan (Table 2). If the patient is admitted to the hospital, the attending physician on the medical ward will be asked the questions regarding clinical intuition on the first day of admission. A member of the research team immediately records the answers to the questions of all questionnaires in an electronic case record form, which was specially designed for this study.

The RISK^INDEX

The RISK^INDEX was developed using ML technology at the MUMC + and aims to predict all-cause 31-day mortality for adults ED patients [16]. The score is calibrated ranging from 0 to 100, where a high RISK^INDEX indicates a high risk of mortality, and a low RISK^INDEX indicates a low risk of mortality.

The RISK^INDEX is based on age, sex and the results of at least four routine laboratory tests in the ED performed within the first two hours of the ED visit. All laboratory values are used to calculate the RISK^INDEX with the exception of tests that are prevalent less than 0.01%. The most commonly ordered laboratory values are creatinine, complete blood count (CBC), sodium, potassium, C-reactive protein (CRP), blood urea nitrogen (BUN), glucose, alanine transaminase (ALAT), aspartate aminotransferase (ASAT), creatine kinase (CK), and platelets. A detailed prescription regarding the RISK^INDEX is shown in Supplemental data.

Allocation and blinding

Patients will randomly be allocated in a 1:1 ratio to the intervention group or the control group (Supplemental Fig. 1). This randomization is automatically performed by a computer system using block randomization of 100 patients. Electronic randomization is performed automatically using an online program. The allocation is presented within 2 h after presentation of the patient to the ED. The study is an open-label study that is not blinded for physicians.

Follow-up

After the patients’ visit to the ED, the follow-up regarding 31-day mortality and other endpoints will be checked by reviewing the medical records. In the Netherlands, all deaths are registered by the municipal administration office, and these data are linked to the medical records.

Drop-out/withdrawal

Patients can leave the study at any time for any reason without any consequences if they wish to do so. The investigator can decide to withdraw a patient from the study for urgent medical reasons. If a patients withdraws from the study, the data up until that moment will be used in study analysis. Data of patients who are unwilling to participate in the study after a deferred consent procedure took place will be deleted and will not be used in the study analysis. The patient will not be replaced, as the study design takes less than 10% drop-out into account. Considering the short duration of the study follow-up, the expected drop-out rate is low.

Adverse event monitoring

The presentation of the RISK^INDEX to the ED physician and the possibly resulting change in the medical treatment plan is considered to have a low risk of undesirable outcomes. The 31-day mortality is expected to remain around the usual average, which is approximately 10%. Possible complications will be closely monitored by the researchers and will be followed until they have abated, or until a stable situation has been reached.

Data collection methods

The researcher will collect the results of laboratory tests, which are already routinely measured in the ED for internal medicine patients. The laboratory data, sex and age are used to determine the RISK^INDEX. In addition, the physician will provide answers on the questions regarding their clinical intuition.

Patient characteristics will be retrieved from medical records and include: demographics (age, sex), comorbidities, mode of transportation to the ED (ambulance or own transport), reason for ED visit (according to the international classification of diseases (ICD-) 10 system) [23], date and time of the ED visit, and triage category. The following vital signs will be retrieved: heart rate (HR), blood pressure (BP), respiratory rate (RR), oxygen saturation, temperature, and Glasgow coma scale (GCS). Furthermore, we will collect data on hospital admission, admission to intensive care unit (ICU), treatment restrictions and all-cause mortality within 31 days after the ED visit. The data regarding patient characteristics, vital signs and outcomes will be retrieved manually. To ensure the quality of these data (i.e. to check whether the data are correct and complete), we will perform a double-check by another member of the research team and/or the study monitor. The retrieval of data regarding the results of laboratory tests and the results of the RISK^INDEX will be automated.

Data will be stored anonymously in an online electronic case record form in CASTOR, which is available for researchers in all participating study centers. Personal data will be handled in compliance with the EU General Data Protection Regulation and the Dutch Act on Implementation of the General Data Protection Regulation, as well as the “Algemene Verordening Gegevensbescherming)” (law on the protection of general data) [24, 25].

The patient identification code list will be stored digitally and encrypted with a strong password. The patient data and documents will be stored for 15 years. Data may be used for other studies which are in line with the current study, as approved by the METC. Data monitoring will be performed by the Clinical Trial Center Maastricht (CTCM).

Sample size

This pilot trial has an explorative aim, providing future trials with estimates of effect sizes and potential clinical impact of the RISK^INDEX. This study aims to provide robust estimates for the likely recruitment and retention rates and give an indication of the potential variability in the proposed outcome measures, which will in turn be used to inform the power calculation for a future definitive randomized controlled trial (RCT). We calculated a required sample size of 784 patients to detect a 2% difference in the number of changes in medical treatment after presentation of the RISK^INDEX between the control group and the intervention group with a power of 0.8.

Yearly, approximately 5,500 patients are treated by an internal medicine specialist at the ED of MUMC+. Considering fluctuations in the number of patients presenting to the ED and moments of crowding, there may be moments where there will not be enough time to include patients and/or to complete the questionnaires. Therefore, we expect a total sample size of 1300 patients during the inclusion period.

Statistical analysis

All analyses will be performed using the SPSS software (IBM Corp. Released 2021. IBM SPSS Statistics for Windows, version 28.0. Armonk, NY: IBM Corp).

The prognostic accuracy of the RISK^INDEX and the physician’s clinical intuition to predict adverse outcomes will be analyzed using an area under the receiver operating characteristics curve (AUC). To assess the clinical impact of the RISK^INDEX, the number of changes in medical treatment based on the RISK^INDEX will be calculated. The prognostic accuracy of the RISK^INDEX will also be compared to that of the physician’s clinical intuition and to that of other clinical prediction models. Sub-analyses will be performed on the ED physicians’ characteristics, i.e. level of experience and function in relation to the accuracy of their clinical intuition. Summary estimates of effects will be presented along with their 95% confidence intervals (CI). Differences between the intervention and control groups will be presented in the form of an unadjusted mean difference for continuous outcomes, and an odds ratio for binary outcomes. Exploratory analysis using ANCOVA for continuous variables and logistic regression for binary outcomes will consider adjustment for the stratification variables in assessment of the treatment effects. Baseline characteristics will be summarized for all patients, and separately for the intervention group and control group. Feasibility and process evaluation data, such as practice recruitment rate, implementation and uptake of and adherence to the intervention, and follow-up rates will be summarized and presented as percentages.