Participants
This was a retrospective study, registered at chictr.org (ChiCTR2300067597). This retrospective study was carried out in the ICU of the Affiliated Hospital of Xuzhou Medical University and was approved by the ethics committee (approved number: XYFY2022-KL464). Due to the retrospective and observational nature of the study, informed consent was waived. The Affiliated Hospital of Xuzhou Medical University, located in Xuzhou City, Jiangsu Province, China, is a tertiary hospital. The medical institution has two hospitals, the east and west hospitals, with 4150 beds.
Patients with ARF who received HFNO were screened for enrollment. Inclusion criteria: (1) diagnosed as ARF (defined as oxygenation index ≤ 300 mmHg, the oxygenation index is the percentage of arterial partial pressure of oxygen divided by the concentration of inspired oxygen) and given HFNO; (2) the age between 18 and 89 years old. Exclusion criteria: (1) ICU stay < 24 h; (2) multiple admissions to ICU; (3) Patients with incomplete clinical data.
Clinical data characteristics
The characteristics of clinical data in the study are shown in Table 1. The clinical data included: (1) baseline characteristics and comorbidities; (2) vital signs, Glasgow Coma Scale (GCS) score, clinical variables on the first day of HFNO; (3) treatment measures (use of sedation, vasopressors, albumin, diuretics and glucocorticoids); (4) primary outcome. HFNO failure was defined as either application of invasive mechanical ventilation or switching to the other study treatment modality.
Development of machine learning models
The outcome-related feature screening process was carried out using least absolute shrinkage and selection operator (LASSO). For the development of the models, the most relevant features chosen are employed. Non-zero characteristic indicators selected through LASSO analysis were put into the multivariate logistic regression analysis to identify the independent risk factors associated with HFNO failure.
We considered seven different types of models: support vector machine (SVM), adaptive boosting (ADABOOST), logistic regression (LR), extreme gradient boosting (XGBOOST), stacking ensemble algorithms (STACK), random forest (RF), and naive bayes (NB); The STACK is algorithms that integrate LR, SVM, NB, and RF. For the selection of hyper-parameters in models, we used five fold cross-validation for the selection of hyper-parameters, which also helped to effectively prevent the model’s over-fitting.
Model validation
For validation of prediction model, we divided the data randomly into a training set and validation set according to a 70–30 split, and then used the resampling method for the internal validation of the prediction model in training set. Finally, we performed the validation again in the validation set. We provide additional technical information on the methods and parameter settings in the Supplementary material Table 1.
Model performance and explainability
To evaluate our models, we considered three predictive metrics: area under receiver operating characteristic (AUROC) curve, Brier score and area under precision recall curve (AUPRC). AUROC is bounded between 0.5 and 1.0, with higher values being better. The Brier score is the mean squared difference between the predicted probability of HFNO failure and the actual outcome (0 or 1 where 1 indicates failure of HFNO). The Brier score is bounded between 0 and 1, with lower values being better. We additionally compared the models by plotting their receiver operator characteristic (ROC) curves, precision recall (PR) curves and calibration curve.
We applied the Shapley (SHAP) value to explain features in the training set. The SHAP summary, combining feature importance with feature effects, was visualized with dot plots to present the distribution of SHAP The position on the y-axis was determined by the feature and that on the x-axis by the SHAP value. The features are ranked by importance. Moreover, partial dependence plots (PDPs) were created to visualize the average change in probability of HFNO failure for all values of a predictor while keeping all other predictors constant17.
Sample size and statistical analysis
Pmsampsize package (https://search.r-project.org/CRAN/refmans/pmsampsize/html/pmsampsize.html) in R software computes the minimum sample size required for the development of a new multivariable prediction model using the criteria proposed by Riley et al.18. Riley et al. lay out a series of criteria the sample size should meet. These aim to minimize the over-fitting and to ensure precise estimation of key parameters in the prediction model. Following the parameters set in the pmsampsize package, we set the c-statistic to 0.80, the potential prediction parameter to 8, and the target event incidence to be 14.1%. Minimum sample size required for new model development based on the above parameters inputs was 459, with 65 events. The sample size in the training set satisfies the minimum sample size requirement for the development of a new multivariable prediction model.
The Kolmogorov–Smirnov test was used to test the normal distribution for measurement data. Normally distributed data were expressed as means ± standard deviation, and the skewed distributed data was reported as medians with interquartile (25th–75th) percentiles. The two groups were compared using student t-test or Mann–Whitney U tests. Numeric data were expressed as a percentage (%), using χ2 or Fisher’s exact probability tests. R software was used for all analyses (R Foundation for Statistical Computing, Vienna, Austria).
Ethics statement
The study was approved by the Ethics Committee of the Affiliated Hospital of Xuzhou Medical University (approved number: XYFY2022-KL464).The procedures were followed in accordance with the ethical standards of the Ethics Committee of the Affiliated Hospital of Xuzhou Medical University on human experimentation and with the Helsinki Declaration of 1975. Due to the retrospective and observational nature of the study, informed consent was waived by the Ethics Committee of the Affiliated Hospital of Xuzhou Medical University.
