Meso Scale Discovery

Sep 10, 2023

Abstract Sepsis is a time dependent condition. Screening tools based on clinical parameters have been shown to increase the identification of sepsis. The aim of current study was to evaluate the additional predictive value of immunological molecular markers to our previously developed prehospital screening tools. This is a prospective cohort study of 551 adult patients with suspected infection in the ambulance setting of Stockholm, Sweden between 2017 and 2018. Initially, 74 molecules and 15 genes related to inflammation were evaluated in a screening cohort of 46 patients with outcome sepsis and 50 patients with outcome infection no sepsis. Next, 12 selected molecules, as potentially synergistic predictors, were evaluated in combination with our previously developed screening tools based on clinical parameters in a prediction cohort (n = 455). Seven different algorithms with nested cross-validation were used in the machine learning of the prediction models. Model performances were compared using posterior distributions of average area under the receiver operating characteristic (ROC) curve (AUC) and difference in AUCs. Model variable importance was assessed by permutation of variable values, scoring loss of classification as metric and with model-specific weights when applicable. When comparing the screening tools with and without added molecular variables, and their interactions, the molecules per se did not increase the predictive values. Prediction models based on the molecular variables alone showed a performance in terms of AUCs between 0.65 and 0.70. Among the molecular variables, IL-1Ra, IL-17A, CCL19, CX3CL1 and TNF were significantly higher in septic patients compared to the infection non-sepsis group. Combing immunological molecular markers with clinical parameters did not increase the predictive values of the screening tools, most likely due to the high multicollinearity of temperature and some of the markers. A group of sepsis patients was consistently miss-classified in our prediction models, due to milder symptoms as well as lower expression levels of the investigated immune mediators. This indicates a need of stratifying septic patients with a priori knowledge of certain clinical and molecular parameters in order to improve prediction for early sepsis diagnosis. Trial registration: NCT03249597. Registered 15 August 2017. Introduction Sepsis is defined as a life-threatening organ dysfunction due to a dysregulated host response to infection 1 . Despite advances in medical care, the mortality of sepsis ranges from 10 to 40% 1 , 2 , 3 . In Sweden, sepsis affects approximately 70,000–80,000 people annually 4 , 5 , while the corresponding number globally is almost 50 million 6 . For this reason, WHO has called for a global action on sepsis 7 and early diagnosis is one crucial aspect to consider for improved care of the septic patient. Timely treatment is shown to reduce mortality and improve outcomes in patients with sepsis and septic shock 8 , 9 , 10 ; early treatment requires early identification. Since more than half the patients with severe sepsis are transported to hospital by ambulance 11 , 12 , identification during this first physical contact with health care should improve patient outcome. This is supported by studies demonstrating that the time to treatment is reduced when the septic patient is identified in the prehospital setting 11 , 13 . Identification of sepsis in the prehospital setting is currently based on clinical judgment, which is proven inadequate 14 . Identification can be increased when using screening tools, however, to date, there are few screening tools available 15 , 16 , 17 , and few have been developed for use in the ambulance. We have previously, in the prospective study Predict Sepsis, developed a set of three Predict Sepsis screening tools based on symptoms and/or vital signs in the prehospital setting 18 . However, as one third of the patients with severe infection exhibit normal vital signs, screening tools based mainly on vital signs present a problem 19 . Furthermore, parameters reflecting the underlying pathophysiology are not included in this type of screening tools. Immune dysregulation in sepsis is currently a field of intense research including both excessive inflammation and immunosuppressive reactions to the underlying infection 20 . A large number of markers for diagnostic and prognostic purposes have been studied, including immune, vascular, organ, coagulation, and cellular markers but few have been found to increase sepsis identification 21 , 22 . One likely reason is that these biomarkers have typically been studied as single markers in isolation, i.e., not taking complex pathophysiological interactions into consideration 23 , 24 . In the current study, the aim was to evaluate the additional predictive values of immunological molecular markers to our previously developed Predict Sepsis screening tools. Materials and methods Study design This current study is part of the Predict Sepsis study, which is a prospective cohort study in the ambulance setting, with patient inclusion between 2017 and 2018, in Stockholm County, Sweden (for details see Wallgren et al. 18 ). The study received approval from the Stockholm Regional Ethical Review Board (reference number 2016/2001–31/2 and 2018/2202). Written informed consent was obtained from all participants. The study was registered at ClinicalTrials.gov, identifier: NCT03249597. The outline of the current study is illustrated in Fig. 1 . Figure 1 Study population The study included a total of 551 adult, non-trauma patients assessed to have a new onset infection according to clinical judgment made by the ambulance personnel. The inclusion and exclusion criteria have been published elsewhere 18 . A selection of candidate molecular markers reflecting immune responses were performed in a smaller group of consecutively included patients, i.e., the screening cohort (n = 96). The selected candidate molecular markers from the screening cohort were analyzed in the remaining patients (n = 455), i.e., the prediction cohort, and used as predictors in combination with the available clinical variables measured in the ambulance in the final prediction modeling (details of cohorts, see Table 2 ). Blood sampling Blood was drawn in EDTA-tubes in the ambulance, and at arrival to the hospital bound emergency departments (EDs), tubes were centrifuged, aliquoted, and frozen in − 70 °C in biobank. Furthermore, blood was drawn directly into PAXgene tubes (PreAnalytix, GmbH, Hombrechtikon, Switzerland), with immediate stabilization of intracellular RNA, before being frozen in − 70 °C at arrival to the ED. Quantification of circulating inflammatory mediators Initially, a total of 71 circulating proteins were analyzed within the screening cohort using U-PLEX Biomarker Group 1 kits (Meso Scale Discovery, Rockville, MD) detected by electrochemiluminescence in Meso QuickPlex SQ 120 (Meso Scale Discovery), according to the manufacturer's instructions. Three additional proteins, CXCL6, HGF, and TGF-α were measured by Human Magnetic Luminex Assay (R&D systems, Inc. Minneapolis, MN), according to the manufacturer's instructions. The samples were analyzed on a Luminex®200™ instrument (Invitrogen, Merelbeke, Belgium), and the data were collected using the xPONENT 3.1™ software (Luminex Corporation, Austin, TX). Later in the prediction cohort, nine selected mediators, i.e., CCL24, CX3CL1, CCL27, CCL11, IL-17AF, IL-17A, IL-1Ra, TNF, and CCL19, were analyzed using customized U-PLEX kits (Meso Scale Discovery, Rockville, MD). All values were expressed as pg/mL deduced from the standard curve, using a 5-parameter logistic algorithm. Values below the detection limit were given half the value of the detection limit. All samples were run in duplicates and a coefficient of variation (CV) below 20% was considered acceptable. In Supplementary Table 1 , the average CVs and detection limits of the nine proteins analyzed in the prediction cohort are listed. RNA extraction and cDNA extraction All samples were arranged in random order prior to RNA extraction. For the screening cohort, RNA extraction was done using PAXgene Blood RNA Kit (Cat. No. 172021754, Qiagen, GmbH, Hilden, Germany), according to the manufacturer's instructions. RNA quality and concentration was measured with NanoDrop 2000 (Thermo Fisher Scientific, MA, USA) and 2100 Bionalyzer (Agilent, CA, USA). The A260/A280 ratios were above 1.7 and the RNA integrity number (RIN) values were above 7. cDNA was synthesized using the High-Capacity cDNA Reverse Transcription Kit (Applied Biosystems, CA, USA) in a LifePro Thermal Cycler (Bioer, Hangzhou, P.R. China), using 200 ng RNA per 20 μL reaction. Gene expression was performed in a Quantstudio 7 Flex Real-Time PCR system (Applied Biosystems, CA, USA), using TaqMan Gene Expression Assays and TaqMan Fast Universal PCR Master Mix (Applied Biosystems, CA, USA) in a 20 μL reaction, according to the manufacturer’s instructions. For the prediction cohort, the samples together with a negative extraction control (consisting of RNase-free water) were extracted using the QIAsymphony extraction robot (Qiagen GmbH, Hilden, Germany). RNA was eluted in a total volume of 80 μL and immediately denatured at 65 °C for 10 min using a thermal cycler (T-100, BioRad, CA, UAS). Sample concentration and purity were determined by spectrophotometry on the Lunatic instrument (Unchained Labs, CA, USA) and RNA integrity was analyzed on capillary gel electrophoresis, Fragment Analyzer (Agilent, CA, USA) using RNA Standard Sensitivity Fragment Analyzer kit (Cat. No. DNF-471, Agilent, CA, USA). The A260/A280 ratios were above 1.5 except for 21 samples. These samples did not turn out as outliers in neither univariate nor multivariate analyses, thus were not excluded. None of the negative control samples (ENTCs) showed cross-contamination. All samples were reversed transcribed into cDNA using the TATAA GrandScript cDNA Synthesis Kit (Cat. No. A103, TATAA Biocenter AB, Gothenburg, Sweden). The reverse transcription was performed using 450 ng RNA per 20 μL reaction. Real-time PCR In the screening cohort, a total of 15 genes encoding inflammatory mediators, inflammasome components, and transcription factors (PYCARD, CASP1, NLRP3, IL1B, IL18, TNF, IL6, IL10, IL1RN, HLA-DRA, HIF1A, SPI1, EPAS1, SIRT1, NFKBIA) were analyzed using qPCR. Gene expression was performed in a Quantstudio 7 Flex Real-Time PCR system (Applied Biosystems, CA, USA), using TaqMan Gene Expression Assays and TaqMan Fast Universal PCR Master Mix (Applied Biosystems, CA, USA), according to the manufacturer’s instructions (TaqMan assay IDs are listed in Supplementary Table 2 ). HPRT1 was used as a reference gene, determined by NormFinder R package (MOMA, Aarhus University Hospital, Denmark) for normalization among a total of three candidate reference genes. All samples of the study were analyzed in duplicates, and the mean quantity values were used in further data analysis. The accepted CV of technical sample replicates was ≤ 15%. Samples with a CV > 15% for each specific assay were re-analyzed. Cycle threshold (CT) cut-off value was set to 35 and all reactions had an efficiency between 90 and 110%. In all cases, gene expression levels were obtained from a six-point serially four-fold diluted calibration curve. The calibration curve was developed from cDNA of PBMCs stimulated by 1 μg/mL LPS. In the prediction cohort, three genes, EPAS1, HIF1A, and NLRP3 were analyzed using assays designed and validated by TATAA Biocenter AB. qPCR was performed with TATAA SYBR®GrandMaster Mix Low Rox (Cat. No. TA01, TATAA Biocenter AB, Gothenburg, Sweden) in 10 μL reaction volume. Human ValidPrimeTM (Cat. No. A105P10, TATAA Biocenter AB, Gothenburg, Sweden) was used to monitor and correct for contaminating gDNA 25 . An inter-plate calibrator (Cat. No. IPC250S, TATAA Biocenter, Gothenburg, Sweden) was run on each plate to be able to correct for inter-run differences. All samples were run in duplicates in 384-well plate format using QuantStudio™ 7 Pro Real-Time PCR system (384-well, ThermoFisher Scientific). The pipetting was performed by a pipetting robot OT-2 (Opentrons, NY, USA). qPCR raw data were pre-processed and analyzed with GenEx software v.7 (MultiD Analyses AB, Gothenburg, Sweden). The limit of quantification of the assays were determined using standard dilution series for which the relative standard deviation of a replicate was < 35%. The accepted standard deviation of technical sample replicates was ≤ 0.5, whereas the accepted standard deviation of the IPC (Inter Plate Calibrator) replicates was ≤ 0.2. Samples with a standard deviation > 0.5 for each specific assay were re-analyzed. Three reference genes, beta-glucuronidase (GUSB), peptidyl-propyl isomerase A, cyclophilinA (PPIA), and ubiquitin C (UBC) were selected from a list of 12 reference gene candidates using the geNorm and NormFinder functions in GenEx software v.7 (MultiD Analyses AB). The relative gene expression was calculated using the delta CT method. Statistical modeling and data analysis In the data analysis pipeline, clinical and molecular variables were assessed with regard to their differences between sepsis and non-sepsis cases and regarding their quality as predictors, using univariate and multi-variable models. For all models, prediction performance was measured in a nested cross-validation approach as AUCs for the hold-out testing set. Finally, variable importance measures, as eligible for the different analysis methods, were applied. This data analysis pipeline was first run on data from the screening cohort, followed by a consensus variable selection for further analysis in the prediction cohort. Then, the pipeline was run again, this time only involving the selected molecular variables, on the data from the prediction cohort, with prediction performances and variables importance reported as before. Selection of molecular variables in the screening cohort Based on screening cohort data, molecular variables as synergistic predictors with the clinical parameters were selected through a stepwise process of (i) univariate analyses, (ii) multivariate analyses and (iii) literature review. From this process, a weighted curation was performed for the final selection of molecular variables, which were used for further analysis in the prediction cohort. The univariate variable selection of the most relevant molecular variables was performed by fitting individual mixed effect models (lmerTest package in R; mixed effects model with the sepsis/non-sepsis as fixed effect and sex as random effect) of the 74 inflammatory mediators as well as the expression levels of 15 genes, to differentiate between non-septic and septic patients, followed by the false discovery rate (FDR) estimation for multiple comparisons. Molecular variables with fold-changes (FC) above the thresholds, set to FC ˃ 1.2 for proteins and FC > 2.0 for mRNAs, and a Benjamini–Hochberg FDR ˂ 0.05, were selected as candidates for analysis in the prediction cohort. The multivariate variable selection of the most relevant molecular variables was performed by machine learning implemented with a nested cross-validation workflow assessing variables as classifiers of non-septic and septic patients. A set of seven different machine learning algorithms were trained in parallel and tested to evaluate different algorithms with regards to classification performance on 7 different variable sets. The variable sets were; (a) all molecular variables, (b–d) previously reported Predict Sepsis screening tools 1, 2 and 3 (using un-categorized original values of the clinical parameters presented by previous study 18 , summary of parameters in the screening tools see Table 1 ), and (e–g) combining Predict Sepsis screening tools 1, 2 and 3 with the molecular variables. In addition, two-way interaction between all variables were created by multiplying variables for evaluation of interaction effects. Table 1 Parameters of the predict sepsis screening tools 18 . Identification of molecular candidates for sepsis prediction in the screening cohort An inflammatory/immune panel of 74 proteins and 15 genes was analyzed in the screening cohort. Elevated levels of IL-17AF and IL-17A were observed in the septic patients group compared to the non-sepsis group in univariate comparisons (Fig. 2 ). In the multivariate analysis of the molecular markers, no separation between sepsis and non-sepsis patients was revealed using the unsupervised analyses with principal component analysis (PCA) (Supplementary Fig. 1 A). However, with a supervised dimensional reduction approach using partial least squares (PLS), partial separation was observed (Supplementary Fig. 1 B) between sepsis and non-sepsis patients, and this separation was further evaluated with supervised machine learning for sepsis prediction and molecular marker candidate selection. The results from machine learning demonstrated a moderate power of the molecular markers to separate sepsis from non-sepsis with averaged AUCs of nests between 0.57 and 0.67 for the different algorithms (Fig. 3 A). The posteriors for mean AUCs showed in general a rather wide distribution for the molecule models indicating an intra-variability effect of resampling nests in the screening cohort. Evaluation of the added value of the molecular variables to the previously reported Predict Sepsis screening tools 18 showed that in general, the molecular variables did not increase the performance of the screening tools in the screening cohort (results from Lasso regression and XGBoosted trees are shown in Fig. 3 B). Figure 2 Levels of molecular markers in the screening cohort. Comparison of the levels of 74 proteins and 15 genes measured in the screening cohort. (A) Volcano plot of the univariate comparison between sepsis and non-sepsis patients; the threshold set to fold-change ˃ 1.2 for proteins and > 2.0 for mRNAs, and a Benjamini–Hochberg adjusted p-value ˂ 0.05. (B,C) Box-violin plots with individual values of the levels of IL-17A and IL-17AF for sepsis and non-sepsis patients, with median and interquartile range (IQR). The variable importance of the molecular variables in the nested cross-validations was evaluated by both model-agnostic importance by permutation (Fig. 4 A) and by model-specific weights (Fig. 4 B,C). None of these paired variable interactions showed higher importance than the original variables per se and they are therefore omitted from the figures. Thirty-three inflammatory mediators and eight genes, with higher variable importance listed in Fig. 4 , together with IL-17A and IL-17AF in Fig. 2 , were further considered in the selection of molecular markers candidates. The literature evaluation gave final weights to the selection of 12 molecular markers (9 proteins and 3 genes) to be evaluated in the prediction cohort; namely CCL24, CX3CL1, CCL27, CCL11, IL-17AF, IL-17A, IL-1Ra, TNF, CCL19, and genes, including EPAS1, HIF1A, and NLRP3. Figure 4 Variables with highest importance in sepsis classification models of the screening cohort. Molecules were ranked by their variable importance values from all classification models based on the molecular parameters of the screening cohort. (A) Model agnostic variable importance by permutation from nested cross-validations of seven different algorithms trained on all proteins and gene expressions. Model specific variable importance weights: (B) Coefficients for Lasso regression, and (C) Gini index node impurity for XGBoost. Molecules labeled with mRNA in parenthesis refers to the gene expression data. Boxplots presented with median and interquartile range (IQR). Differential expressions of the selected molecular markers between sepsis and non-sepsis patients in the prediction cohort Among the 12 selected molecular markers, the univariate analysis results show that levels of IL-1Ra, IL-17A, CCL19, CX3CL1, and TNF were significantly higher in plasma from the sepsis patients compared to non-sepsis patients in the prediction cohort, whereas levels of IL-17AF were higher in non-sepsis patients (Fig. 5 ). The multivariate analysis, similar to the screening cohort, showed that the supervised PLS, but not the unsupervised PCA, demonstrated partial separation between sepsis and non-sepsis (Supplementary Fig. 2 ). This separation was further evaluated for prediction of sepsis in the prediction cohort. Figure 5 Expression levels of immune mediators and genes in the prediction cohort. Volcano plot of the univariate comparison between sepsis and non-sepsis patients. The threshold set to fold-change ˃ 1.2 for proteins and > 2.0 for mRNAs, and a Benjamini–Hochberg adjusted p-value ˂ 0.05. Performance of sepsis prediction models with selected variables in the prediction cohort The evaluation of added value of the selected 12 molecular variables to the Predict Sepsis screening tools 18 showed that the molecular variables per se did not further contribute to the predictive performance of the screening tools (Fig. 6 B). Training of models on the 12 selected molecular markers based on the prediction cohort data and with the previously employed different machine learning algorithms showed moderate predictive performance with averaged AUCs between 0.65 and 0.70 (Fig. 6 A). The posteriors for mean AUCs showed a smaller distribution for the molecule models in the prediction cohort compared to the screening cohort indicating less intra-variability effect of resampling nests. Figure 6 Machine learning classification of sepsis and non-sepsis patients in the prediction cohort. Upper sections show the distributions of posteriors for mean area under the curve (AUC) from the nests in nested cross-validation and lower sections show the averaged ROC curve (AUCs within parenthesis in 6A). (A) Distributions of posteriors for mean AUCs and averaged ROC curve from nested cross-validations with seven different algorithms trained on molecular variables alone. (B) Distributions of posteriors for mean AUCs and averaged ROC curve from nested cross-validations trained on screening tools variables with or without molecular variables and their interactions. LR Penalized regularized logistic regressions (LASSO), RF Random forests, XG XGBoosted trees, NN Neural network, NB Naïve Bayes, LGBM lightGBM, ST Stacked model of LR, XG and NN. The model-agnostic variable importance, as obtained by permutation, indicates that IL-1Ra is the most important predictor among the molecular markers (Fig. 7 A). Again, no variable interactions had higher importance than the original variables per se. The evaluation of permuted variable-based importance for models of the Predict Sepsis screening tools with molecular variables show higher importance for many of the clinical variables than for the selected molecular variables (Fig. 7 B). A Pearson correlation matrix showed a high correlation of the screening tools variable “temperature” and several of the selected molecular variables, such as IL-1Ra (Fig. 7 C), capturing much of the informative variation of the molecular markers. Figure 7 Importance of molecular and clinical variables from sepsis screening tools in the prediction cohort. (A) Molecules with highest model-agnostic variable importance by permutation from nested cross-validations of seven different algorithms trained on molecular markers alone. (B) Top-20 model-agnostic variable importance by permutation from nested cross-validations of seven different algorithms trained on all screening tools variables and molecular variables. (C) Pearson correlations between the molecular markers and temperature (*, ** and *** denotes a p-value ˂ 0.05, 0.01 and 0.001 respectively, color denotes correlation coefficient). Molecules labeled with mRNA in parenthesis refers to the gene expression data. Boxplots presented with median and interquartile range (IQR). Evaluation of miss-classified subgroups of patients To explore and understand the underlying inability to fully predict septic patients in the current cohort, an evaluation of the miss-classified patients was performed. Groups of patients who was consistently miss-classified (with probabilities above 0.6 to be classified into the other group in all nested cross-validations) was identified. As demonstrated in Fig. 8 , miss-classified septic patients (n = 26) presented with milder fever, higher GCS and systolic blood pressure as well as lower IL-1Ra and IL-17A levels, while miss-classified non-sepsis patients (n = 33) demonstrated higher temperature, lower GCS, lower systolic blood pressure, and higher level of IL-1Ra and IL-17A, when comparing to the rest of their group respectively. Figure 8 Expression of clinical and molecular parameters for miss-classified patients. Box-violin plots with individual values of the expression levels of clinical and immune parameters among miss-classified patients in all the prediction models from both sepsis and infection non-sepsis groups, with median and interquartile range (IQR). Temp temperature, GSC Glasgow Coma Scale.