Finished! Looks like this project is out of data at the moment!
This project is now complete! Many thanks to everyone who participated. See the results
Results
Thanks everyone!
Many thanks to everyone who contributed their time and energy to this project. Altogether you classified 7959 different images, with top submitters Ianmetcalf, DickieS, gatoreasy1 and anilahc completing over 5000 classifications each!
How often do change points occur in real-life medical datasets?
Your classifications were combined to create 'consensus labels' using a method called density-based clustering with noise, i.e. if at least 5 of you drew a line within 3 pixels of each other, this location was considered to be a change point.
Altogether you found 4477 distinct change points across our 8 hospital datasets, an absolutely huge number, and far more than we expected to find when we started this project. This shows that change points can occur very frequently indeed, and so it is important for all researchers to check their datasets before they use them.
Testing of computer-based methods
One of the main purposes for collecting this data was to see if computer-based methods could do the same job as humans at spotting 'unnatural' looking change points.
A number of existing methods were tested against the consensus labels but none were found to be sufficiently accurate compared to humans. However, we are hopeful that more attention will now be paid to developing change point detection methods now that there is such a rich dataset to test them against.
The data is publicly available to download from https://doi.org/10.5281/zenodo.7331161, and a full description of how the data was generated has been published at https://doi.org/10.1093/gigascience/giad060.
What should researchers do now?
Until computer-based methods get better at identifying change points, researchers will need to continue to inspect their datasets manually. To help with this, we have created a new tool called daiquiri, in the R programming language, which automatically creates appropriate graphs for quick visual inspection in an easy to manage html report. More information on this can be found at https://ropensci.github.io/daiquiri/.
Thanks again!