Bounding Box Bonanza

This project has been used to support the following efforts (this page will be updated as new efforts are hosted in this sandbox space):

2024-present: "Optimizing the Human-Machine System for Citizen Science"

The University of Minnesota and Adler Planetarium Zooniverse teams are conducting research and experiments with the aim of removing classification inefficiencies that are currently a bottleneck for Handwritten Text Recognition (HTR) efforts. We are specifically aiming to gather data that will allow us to improve textual line prediction accuracy on HTR projects, with the eventual goal of reducing the need for a line-drawing step altogether, while still allowing researchers and volunteers to benefit from the clarity and resulting data provided by a line-by-line classification structure.

Project Aims

The workflows developed for this project aim to collect bounding box data from volunteers, to help us train text identification models to increase their accuracy. We are using boxes rather than lines, in an attempt to increase the quality of our line-detection processes. We are also using adjustable-slope boxes to more accurately identify the position of handwritten text, and account for natural slant in handwriting.

Context & Method

More than 75 text transcription projects have launched on Zooniverse, with participation from 10,000s of volunteers. Within this set, there are many HTR projects which include among their goals generating training data for automated transcription methods as well as optimizing volunteer contributions (e.g., by having volunteers review and edit machine transcriptions) in support of scale and speed. Existing machine models for our HTR projects do an initial box-detection of each line of text before performing transcription, but have shown poor performance due to the wide diversity in text orientation, confusion from cursive hand-writing, image contrast and page layouts. Volunteers are thus required to do both the line annotation and the actual transcription, creating a major overhead in project completion. This experiment expands on our previous 'correct-a-machine' efforts, which focused both on line and text prediction. Ironically we are now taking a step back from our previous experiment to just focus on line detection as a method of increasing 1) positive user experience through reduction of 'bad' line detection; and 2) accuracy of subsequent text prediction.

Output

We will update this as our experiment progresses!

Funding

This project was created with support from a 2023 Collaborative Research: Human-Centered Computing award from the National Science Foundation. View full details of the award here.

History of Our Prior Line Identification Work

Our current project is the next step in our ongoing efforts to provide the most efficient and best possible tools for crowdsourced text transcription for the ever-increasing number of transcription projects on the Zooniverse platform. Since 2017, we have made a concerted effort to create new and improved tools for text data collection and analysis. This approach combined machine and human classifications, with the aim of optimizing volunteer effort by making the transcription process more efficient.

Our data science team at the University of Minnesota trained a handwritten text recognition model that predicted the position of lines on an image of a page of handwritten text, and generated transcriptions for the recognized lines. The Adler Zooniverse team built new infrastructure for the platform that supports ingestion of machine-generated data in Project Builder projects. We combined these two efforts to create a Zooniverse project that used the Transcription Task, but with a twist: the 'first pass' annotations + transcriptions were generated by a machine learning algorithm. Beta testers were invited to help us 'correct' the machine, and provide feedback on the user experience.

That project was intended to measure the reliability of our machine-learning model, as well as to help us collect feedback on the user experience for similar 'correct-a-machine' workflows.

The full research effort took place in four parts:

1. Training a machine-learning model for HTR using existing transcription data from the ASM project;
2. Building a data pathway for uploading machine transcription data into the Zooniverse platform;
3. Creating a new workflow on Zooniverse to combine the machine-generated transcriptions with volunteer effort, using existing tools for collaborative text transcription;
4. An experiment to test the ASM HTR model on other, similar datasets from the University of Minnesota's Archives & Special Collections.

Output

The output from this total effort was the new data ingest pathway (step 2 above), and an evaluation of best practices for combining human and machine effort in the production of high-quality transcription data. This workflow helped us 1) better understand the data quality of machine transcriptions; and 2) gather feedback on the user experience for this type of 'correct-a-machine' workflow. The project results are now being used to refine future Zooniverse tools and projects aiming to invite volunteers to help correct machine-generated transcriptions.

Funding

The initial project was created with support from a 2020 Digital Extension Grant from the American Council of Learned Societies.