Infection Inspection

Antibiotic resistance poses one of the most urgent challenges to public health worldwide. In this process, bacteria acquire genetic mutations that help them to become resistant to antibiotics. If bacteria become completely resistant to all antibiotics, this treatment will effectively become useless, and simple infections could cause deaths. In fact, this problem is already causing an estimated 1.3 million deaths every year.

One of the biggest challenges is that current testing methods can take up to two days to determine the most effective antibiotic for an infection. Our team’s goal is to create a test that detects whether a patient’s bacteria are resistant to antibiotics within an hour. The test works by taking images of a patient’s bacteria under a microscope and using artificial intelligence (AI) to look for any changes that occur when antibiotics are applied to these samples.

You can learn more about our project in this video: https://www.youtube.com/watch?v=8zKpE26zzck.

We have collected thousands of images of resistant and sensitive bacteria that have been treated with antibiotics. If the bacteria are resistant, we expect them to look like a untreated bacteria, as the antibiotics have had little or no effect on the cell during the short treatment period of 30 minutes. Bacteria that are sensitive to an antibiotic treatment develop changes to their shape, DNA, and cell wall as the antibiotic interferes with their functions of life. The AI model learns to detect these changes by studying images of bacteria that have responded to the antibiotic treatment and images of bacteria that don’t have those changes.

Sometimes, even though our AI model has seen thousands of images of antibiotic-treated bacteria, it still makes mistakes. In a recent research paper (https://doi.org/10.1038/s42003-023-05524-4), we showed that our current model is about 80% accurate at classifying each Escherichia coli (E. coli) cell. Although this leads to very high confidence when determining whether a whole sample is antibiotic sensitive or resistant, we want our diagnostic test to be as robust as possible.

We noticed that there was some variation in the extent to which E. coli cells changed after the antibiotic treatment, even when they were treated with the same concentration of antibiotic and had the same level of antibiotic resistance. In some cases, cells that looked like a resistant cell were actually sensitive, and vice versa. We started the Infection Inspection project to see which bacterial cells were most likely to be misinterpreted by volunteers, so that we could learn what features might also confuse the AI model. Then, we can focus on understanding those types of cells in our future research. We were also curious whether humans could detect more nuanced features than the AI model.

The Infection Inspection workflow showed volunteers a picture of an E. coli cell that we treated with the antibiotic ciprofloxacin, stained, and imaged with our microscope. Because we grew the E. coli from strains collected at the hospital microbiology laboratory, we knew which strains were sensitive or resistant to ciprofloxacin on standard tests.

The cell membrane and the DNA are stained so that you can see the DNA change as the cell is damaged by the antibiotic. When the cell responds to the antibiotic, the DNA tends to become compact and moves to the centre of the cell. You can see more examples in our project’s field guide if you’re curious! Equipped with the field guide, volunteers could classify each image as Resistant, Sensitive, or an Image Processing Error.
We were honoured that more than 5,000 volunteers contributed more than 1 million classifications to our project.

How difficult is it to classify a bacterial cell?

To understand which images were most likely to be misclassified as resistant or sensitive, we needed a way to measure whether a volunteer’s classification matched what we expected. In our dataset, we used E. coli cells from 5 clinical strains with different levels of resistance and treated them with different concentrations of ciprofloxacin for 30 minutes. We decided that if a cell was treated with an antibiotic concentration greater than its level of resistance (called Minimum Inhibitory Concentration), we expected it to look Sensitive. This wouldn’t always be true, but this defined our “Ground Truth.” In contrast, if a cell was treated with an antibiotic concentration less than its level of resistance, we’d expect it to look Resistant. By defining these categories, we could discover when the biology didn’t match these predictions.

As we suspected, there was a lot of variety in how easy a cell was to classify and how well it lined up with our predictions. If we defined accuracy as what fraction of volunteer classifications lined up with our expectation, some cells were classified correctly every single time (accuracy = 1.00 or 20/20 classifications), others most of the time (accuracy = 0.65 or 13/20 classifications), and others almost never (accuracy = 0.20 or 4/20 classifications). This was true for both Resistant and Sensitive cells. We could tell from images of the cells that were rarely classified “correctly” that these cells had unusual or atypical DNA features.

From our analysis of the volunteer classifications, we couldn’t find any relationship between a volunteer’s accuracy and the number of images they classified or the number of days they were active on the project. It seems like most of the difficulty of this task comes from the images themselves, rather than a user’s expertise.

What makes a cell confusing to classify?

We thought that most of the cues that a cell is responding to ciprofloxacin come from changes in the DNA. The results from Infection Inspection confirmed that hypothesis. We measured seven different image features that we thought could be affected by the antibiotic. This included some features to do with the cell membrane, such as its length and shape, and other features that measured the DNA, such as how much space it took up in the cell and how it was clumped. We used a mathematical technique called a Principal Component Analysis to plot all 7 of these features on a graph, where every coloured point is one of our cell images. Cells that have more similar characteristics will be close together on the graph, and more distant coloured points will have larger differences in the features we measured. The labelled black lines represent the cell features that push the points toward different parts of the graph.

When we looked at the cells that were almost always classified correctly (more than 19/20 classifications matched what we expected), the Sensitive cells (blue) mostly clustered far away from the Resistant cells (red). This means that these images in the Resistant cluster (red) looked very different from the ones in the Sensitive cluster (blue).

That wasn’t the case when we looked at cells that were more confusing. If we focused on the images that were classified incorrectly most of the time (less than 10 out of 20 classifications matched our expectation), the clusters were on top of each other and were much more spread out around the graph. Those Resistant (red) and Sensitive (blue) cells were classified incorrectly because their features were in the middle of what we expect a Resistant or Sensitive cell to look like. We can also tell that these cells were more varied in unexpected ways, because they tended to spread out above and below the main clusters. These images were classified “incorrectly” because their features were unusual or somewhere in the middle of what we expect from an E. coli cell treated with ciprofloxacin.

In fact, when we looked at the feature measurements for cells that were most often classified the opposite of what we’d expected, there was no statistically significant difference in the measurements of Sensitive or Resistant cells for three features that measured DNA compaction and heterogeneity. Basically, if you took one of those incorrectly classified cells, and tried to judge whether it was Resistant or Sensitive based on one of these measurements, it would be impossible to say.

What we learned and what comes next

The Infection Inspection project showed us that misclassifications of ciprofloxacin-sensitive and ciprofloxacin-resistant E. coli bacteria are associated with greater diversity in the appearance of the bacterial DNA after antibiotic treatment. It seems like most misclassifications happen when the features don’t line up with what we expect from a Sensitive or Resistant cell, rather than our AI classifier mis-identifying features.

Even though we expect the bacteria in each of our samples to be genetically identical, there are clearly some cells that respond differently than others. This is an area with lots of open questions that we are designing experiments to answer.

Some volunteers started to notice that in some of our images, it looked like a cell was in the process of cell reproduction. This idea could be related to why some of the bacteria in our samples respond differently than others. It’s possible that the stage of the bacterial life cycle at which the cell is exposed to the antibiotic has an impact on the appearance of the DNA. This is a question we’ll continue to explore in our research.

Thank you

Our group is actively looking into the antibiotic response of many bacterial species to different antibiotics, so that we can develop a rapid test for antibiotic resistance. There is a lot to learn here, and we are extremely grateful to the Zooniverse volunteers who participated in this project. Your enthusiasm and curiosity were extraordinary. Thank you for your dedication and engagement. We will be back if we find another research question where we could use your help!

Thanks also to the Zooniverse platform leaders, Helen Spiers, Mary Westwood, and Cliff Johnson for their expertise and contributions to the development of this project.

The full manuscript detailing our results is available with Open Access at Scientific Reports: https://www.nature.com/articles/s41598-024-69341-3

The Infection Inspection Team