AI Tool Predicts Surgical Infections, but Is Highly Site Specific

DALLAS—An artificial intelligence tool developed by researchers in the Netherlands proved quite accurate at predicting postoperative surgical infections in a local hospital, but less so when extended to a large academic center, emphasizing the need to retrain AI data on external data as these tools gain ground in medicine.

Despite multipronged efforts to reduce the incidence of postoperative infections, they still occur in approximately 20% of all patients. On average, these infections are not diagnosed until postoperative day 5, said Siri van der Meijden, a junior data scientist and PhD candidate at Leiden University Medical Center (LUMC), in the Netherlands, who presented her team’s research at the Surgical Infection Society’s 2022 annual meeting.

To be more proactive in the care of postoperative patients, Ms. van der Meijden and her colleagues developed an AI tool to estimate the risk for infection based on preoperative and interoperative variables; the tool is meant to be applied directly after surgery. Patients deemed at high risk could be closely monitored, “and ideally patients with a low risk of infection could be discharged from the hospital earlier,” she said.

The researchers used electronic health record (EHR) data from 73,423 retrospective patient records to build the model, and then validated it on both prospective patient data from LUMC (612 patients) and on external patient data from the Radboud University Medical Center (14,529 patients). The primary end point was the model’s ability to predict the risk for any treated bacterial infection by day 30 after surgery. Infection criteria included registered infections, use of non-prophylactic antibiotics, infection-related second surgery or surgical intervention, and elevated C-reactive protein.

The tool predicted postoperative infections—including surgical site infections, pneumonia, urinary tract infections and other bacterial infections—in 32% of the retrospective patients, 30% of the prospective real-world patients and nearly 48% of the external prospective patients. Its accuracy in the smaller patient sample was 86%, with a mean area under the receiver operating characteristic (AUROC) curve of 92%. In the larger data set, accuracy dropped to 63% and AUROC curve to 68%.

“So, the performance of the model was similar when applied to the site [where it was developed] in a prospective setting, but that performance decreased clearly when we applied the model to another hospital site,” Ms. van der Meijden said.

“From that we concluded that we needed to retrain the model when going to a different site; really, getting a location-specific model in order to get sufficient performance.”

Currently, Ms. van der Meijden and her colleagues are trying to solve that problem. “We’re creating a pipeline where we can ingest data from the hospital and train it locally to get maximum performance. It’s taken a lot of work just to get a flexible pipeline, because the data types for each hospital are quite different.”

Heather Evans, MD, MS, a professor of surgery at the Medical University of South Carolina, in Charleston, who was the discussant for the paper, considered it a representative presentation of ongoing work in AI, particularly in infection control research.

“Doing infection control in the right way with a lot of detail is a huge undertaking. I think this is an amazing effort to develop a manageable, automated way to identify surgical site infections using the EHR. And this is the holy grail—marrying surveillance, which is a very different kind of data ascertainment, and clinical decision making.”

Dr. Evans had some criticism of the analysis. “The rate of postoperative infection was rather high. I question whether there was an overestimation of infection rate in the sample, because the way they were defining infection was by identifying that the patient had been treated for infection, rather than relying on culture or other documentation of clinical symptoms and signs of infection.”

She also noted the challenge, as the researchers encountered, of using an algorithm developed at one site to gauge risk at another. “Your algorithm has to be tweaked based on the data set you’re using and the population you’re serving. This is where we get into situations where AI can discriminate and exclude minority populations.”

But she was delighted by the presence of Ms. van der Meijden at the SIS meeting as a representative of an emerging field of medical data scientists.

“I always welcome when someone like Siri comes to our meeting because they’re thinking outside of the box, a data scientist who really understands the potential of the data they’re looking at and not biased by the data they think is coming or want to see.”