Project results and updates are also posted on the Deep Lake Explorer News

Update: Manuscript published!

We have published a manuscript titled Deep Lake Explorer: A web application for crowdsourcing the classification of benthic underwater video from the Laurentian Great Lakes on the results of the Deep Lake Explorer project. Check it out!

---

Initial results from Deep Lake Explorer

In July 2019, we posted our first full dataset on Deep Lake Explorer (DLE). This was intended as a pilot study to see if crowdsourced classification is an effective way to classify underwater video collected in the Great Lakes. To evaluate the effectiveness of crowdsourcing as a tool for underwater video classification, we compared the results of the crowdsourcing to expert classification of the same videos. This page presents some of the initial results from this classification.

The project in stats

• 746 video clips
• 531 volunteers participated (Thank you!!!)
• All clips were classified by 10 different volunteers in 2 weeks
• 6% of the volunteers commented at least once on the talk forum.
• Volunteers left 192 tags, comments, or questions on the forum.

Read on for more detail about what we found in this pilot study!

Where were videos collected?

The videos classified were collected in Lake Ontario, Lake Huron, and the Niagara River. The videos in Lakes Ontario and Huron were collected as part of lakewide surveys. You can learn more about those surveys here. The Niagara River videos were collected as part of a pilot study as part of the National Coastal Condition Assessment (NCCA) to assess ecological conditions in Great Lakes connecting channels. Learn more about NCCA here.

Map of sites in Lake Huron, Lake Ontario, and the Niagara River where videos were collected. In Lake Huron, videos were collected within 2 km of the base sites shown.

How were videos collected?

On Lake Ontario and Lake Huron, videos were collected aboard large research vessels, the R/V Lake Explorer II and the R/V Lake Guardian. Two cameras were mounted on a large steel carriage, one in the down-looking direction and one in the oblique-looking direction. Lights were also mounted in each direction. The video carriage was then lowered to the bottom using a winch on the vessel and allowed to rest on the bottom for approximately one minute. The carriage was then raised back to the deck and the videos were downloaded from the cameras.

Left to right: Large vessel video carriage, the R/V Lake Guardian, Deploying the video carriage from the R/V Lake Guardian.

On the Niagara River, videos were collected aboard small boats with a smaller carriage with a similar format. Both downlooking and oblique cameras were mounted on a lightweight steel frame along with lights. The frame was deployed by hand over the side of the boat and lowered to the bottom for about one minute.

Left to right: Small video carriage, small boat used to deploy small video carriage.

How were videos classified on Deep Lake Explorer?

Only oblique-looking videos were classified. The videos were trimmed, separated into 15 second clips, and uploaded to Deep Lake Explorer. Then, we turned the classification over to volunteers on Zooniverse to complete.

After each video clip was classified by 10 different volunteers, the results were reviewed and compared to expert classification. Experts classified the original uncut videos. The final result for each clip analyzed on DLE was determined based on a threshold level of agreement among volunteers:

• 30% volunteer agreement for round goby presence
• 40% agreement for zebra/quagga mussel presence
• 40% volunteer agreement for vegetation presence
• 50% volunteer agreement for substrate type hard or soft

The thresholds were selected based on the thresholds with the highest agreement between the volunteer classification and expert classification. Next, we aggregated the results for each clip into a result for the full video.

Comparison to expert classification

Agreement between crowdsourced classification and expert classification ranged from 77% to 90%, depending on the category). We also did a comparison of two expert analysts, and agreement varied between 90-96%, illustrating that the tasks we asked volunteers to do were challenging – even two experts did not always agree! Lastly, we calculated detection rates for the crowdsourcing classification:

Detection rate = Number of videos in which volunteers and expert classification detected attribute/number of videos in which only experts detected attribute

Table 3. Accuracy of two-expert classification, DLE classification, and detection rates for each attribute classified by both DLE and experts. The expert to expert comparison was based on 51 Lake Ontario videos only, while the DLE-expert comparison was based on 196 videos. Detection rates are given for hard substrate/soft substrate.

	Agreement between two experts	Agreement between DLE and experts	Detection Rate
Round goby presence	96%	89%	78%
Mussel presence	92%	79%	66%
Vegetation presence	94%	90%	93%
Substrate type hard	90%	77%	62%/95%

Agreement between the crowdsourced classification and expert classification was lower than between two experts. We identified several challenges and potential solutions to improve future analyses:

Challenge	Potential Solutions
1. Video segmentation methods. Experts classified full videos, and volunteers classified 15 second clips that only included times when camera was at the bottom. This was a problem in videos such as the example below, where attributes are visible during the time the camera is lifted but not when the camera is at the bottom.	In the future, we could upload all footage and include (like many other Zooniverse projects) an initial “Is anything there?” step in which volunteers first evaluate if the clip contains anything of interest. Then, only videos with “something there” would go on for full classification.
2. Video resolution on Deep Lake Explorer. Video clips had to be resized and compressed for uploading to Deep Lake Explorer to ensure the videos would load fast enough for users with limited internet speeds. Because of this, experts classified higher quality images than volunteers did.	This factor is currently limiting, but as high-speed internet access increases globally, larger video files may be supported on Zooniverse.
3. Rare objects challenging to detect. Within the video frame, round gobies and zebra/quagga mussels are often relatively rare. Rare objects are harder to spot, especially for beginners.	One way to address this is to increase the number of reviews required for each clip, to increase the chance of a few users spotting individuals in the very hard videos. Another approach is that the results of crowd- sourced analysis can be considered minimums. Lastly, we calculated detection rates based on these surveys where we had an expert classification. For future surveys, these detection rates can be used to estimate “true” rates of presence.
4. Analysis complexity. These videos were hard to classify! Some of the challenges were when objects were a little far from the camera, when the water clarity was poor, or when vegetation or mussels covered the bottom making it hard to see fish, mussels or the substrate.	We can strive to improve the workflow to provide more examples to help users respond consistently. We could also add an “I don’t know” option to help distinguish videos where it is impossible to tell what the bottom is. Lastly, video is not a perfect tool, and some videos will ultimately just be too hard for volunteers or experts to agree on.

---

Example of challenge 1:

This is an example of a video in which the video segmenting methods used were not ideal. The camera hovers over an obvious bed of zebra/quagga mussels, but when it lands no mussels are visible. Experts, who watched the entire video, detected the mussels. Because DLE only included footage when the camera was at the bottom, the crowdsourced classification did not detect these mussels.

Waterbody results

The graph below shows the proportion of area in each waterbody with the different attributes we had volunteers and experts identify. Round gobies were most common in Lake Huron and least common in Niagara River. Lake Huron and Niagara River had the largest proportions of videos with hard substrate. Niagara River had the most sites with vegetation present (80% of sites). Mussels were most common in Lake Ontario. This type of information can give an idea of how common these invasive are in each lake, give an indication of how they may be affecting the ecosystem, and help managers track how their populations change over time.

Percent of videos classified on Deep Lake Explorer with given attributes for each waterbody assessed based on DLE classification. Substrate type is presented as percent dominantly hard; the remaining proportion of sites assessed were classified as dominantly soft.

Summary

Overall, we found that the Deep Lake Explorer’s first season was successful. We were impressed by the volunteers’ enthusiasm and thankful for their time and contributions to the project. We learned a lot about how to classify underwater video on Deep Lake Explorer. The videos can be challenging to classify, but we have identified solutions for many of the challenges we encountered. Our team is working on a manuscript with many more details about what we found. We’ll also post a link to the manuscript here once it is published.

Deep Lake Explorer's next analysis is yet to be determined... stay tuned!

Dive Deeper

For more background on underwater video collection and this project, check out the Deep Lake Explorer News. You can also post your questions about the project for the researchers!