Leakage-Free, Explainable Ensemble Machine Learning for COVID-19 Clinical Risk Stratification: A Two-Task Study of Intensive-Care Prediction and Blood-Marker Diagnosis

Abstract

Many of the clinical machine-learning struggles on COVID-19 have two common flaws. Models are often described to perform well, because data is leaked into them, and they are "black boxes" designed that a clinician cannot be requested to examine. We do both at once by executing a single, clean pipeline on two distinct clinical tasks on distinct, public datasets both at once.
The first task foretells admission to intensive care. It is based on 385 patients from a Brazilian tertiary hospital, and we only accept measurements the first two hours after arrival, exclude the patients who are already in the ICU during this period and set a very strict definition of what we consider to be a transfer, which is from the outside to the inside of the hospital. The second was a diagnostic task, which involved identifying SARS-CoV-2 infection from a common blood test called a complete blood count (CBC) in a separate sample of 5,644 individuals.
Six ensembles were tested under the same conditions on the same data (Random Forest, Extremely Randomised Trees, Gradient Boosting, AdaBoost, LightGBM, XGBoost), using stratified splitting, 5-fold cross validation, and metrics that take into account class imbalance. We applied Permutation Importance and SHAP to open up the models that we obtained. Additionally, held-out estimates per class are reported along with 95% Wilson score confidence intervals, and inter-rater agreement style (Cohen's kappa) is compared to the Landis-Koch scale, which includes an explicit measure of statistical uncertainty that is not expressible from raw point estimates alone.
Boosting-family models, AdaBoost and Gradient Boosting, performed better than both bagging-family models on both tasks, out-performing the Random Forest and Extra-Trees methods. In the ICU problem the best models obtained very high sensitivities, based primarily on age percentile, neutrophil and lymphocyte counts, and renal/cardiovascular markers. AUC values ranged from 0.74 to 0.80 for diagnosis, and the eosinophil, leukocyte, and platelet counts contributed most to the AUC. It should be emphasized that features revealed by permutation importance are similar to those reported in the clinical literature.
The bigger depiction: When the leakage is known, when a post-hoc explainability report is provided, when a formal statistical-uncertainty report is available, and when the implementation is complexity-aware, a prediction has a firmer and more clinically usable, and more reviewable, footing than can be established by a headline accuracy figure. But, with small sample sizes and the unevenness of the data that remains in each class, it is important to be cautious before generalizing these results.