Each year, underwater remotely operated vehicles (ROVs) record thousands of hours of video from unexplored ocean habitats, but turning that footage into biodiversity data requires scarce, costly expert annotation. Our underwater work uses BisQue to manage, annotate, and analyze this video at scale, from automated detection of deep-sea invertebrates to habitat studies in Bering Sea submarine canyons.
A deep-sea video dataset and a detector that uses the seafloor as context.
The Dataset for Underwater Substrate and Invertebrate Analysis (DUSIA) is a benchmark suite and growing large-scale dataset for temporally localizing four underwater substrates and for temporally and spatially localizing 59 invertebrate species. Its videos were captured by an ROV following pre-planned transects across the ocean floor near the Channel Islands of California.
Each video is annotated with the start and end times of substrates and with counts of species of interest, and some frames also carry precise bounding boxes for invertebrates. To our knowledge, DUSIA is the first dataset of its kind for deep-sea exploration, with video from a moving camera, that includes substrate annotations and invertebrate species found at depths sunlight does not reach.
Alongside the dataset, the novel context-driven object detector (CDD) uses explicit substrate classification to influence an object detection network, so that it simultaneously predicts a substrate and a species class influenced by that substrate. The work also presents a method for improving training on partially annotated frames and a baseline for automatically counting invertebrate species of interest.
Beyond detection, the module can reconstruct a full 3D model of a species from a single reference frame — for example, generating the fragile pink urchin, with its rigid test and fine spines, from one input image.
The full dataset is about 120 GB. You can watch the videos online in BisQue or download them from your browser, and neither option needs an account.
Open the DUSIA videos in BisQue to browse and play them. No account or login is needed.
BisQue can package the videos for download. At about 120 GB, the full dataset needs plenty of disk space and a stable connection.
MD5 checksums are available for each video file so you can verify your download.
If you use DUSIA in a research publication, please cite the paper below and acknowledge the Marine Applied Research and Exploration (MARE) group, who collected the videos.
R. Austin McEver, Bowen Zhang, Connor Levenson, A S M Iftekhar, and B. S. Manjunath. Context-Driven Detection of Invertebrate Species in Deep-Sea Video. International Journal of Computer Vision, 131(6), 1367–1388, 2023. doi:10.1007/s11263-023-01755-4
@article{McEver2023,
title = {Context-Driven Detection of Invertebrate Species in Deep-Sea Video},
author = {McEver, R. Austin and Zhang, Bowen and Levenson, Connor and Iftekhar, A S M and Manjunath, B. S.},
journal = {International Journal of Computer Vision},
volume = {131},
number = {6},
pages = {1367--1388},
year = {2023},
doi = {10.1007/s11263-023-01755-4}
}
Structure-forming corals and sponges, and their use as fish habitat.
BisQue has been used to manage and analyze 23.3 hours (884 GB) of high-definition video from dives in Bering Sea submarine canyons, to evaluate the density of fishes and structure-forming corals and sponges and to document and describe fishing damage.
Non-overlapping frames were extracted from each video transect at a constant rate of one frame every 30 seconds. An image processing algorithm developed in MATLAB detected laser dots projected onto the seafloor as a scale reference, and BisQue's module system allowed this MATLAB code to be wrapped into an analysis module and parallelized across a compute cluster.
Each frame was also manually annotated with objects of interest such as fishes, sponges, and substrates, and these annotations and other image metadata such as pixel resolution and GPS location were stored in BisQue's flexible metadata store. The annotations were then used to compute the average density of species and co-habitation behavior in different regions of the canyons, resulting in new insights into this ecosystem.
A 3D seafloor model built from ROV video, by Mayachitra, Inc.
MitiLens, a product of Mayachitra, Inc., turns ROV video into a 3D model of the seafloor that can be measured in meters. This showcase follows one 16-minute dive across a rocky slope, 176–189 m deep.
Video and MitiLens by Mayachitra, Inc., Santa Barbara, California. Open the video (68 MB).
MitiLens builds the model from 289 frames of the dive and takes its scale from the vehicle's own depth sensor, so distances can be read directly from a frame. In the video, a sea fan measures 0.31 m from base to blade, with a typical error of 0.06 m at that distance.
The vehicle's laser pair provides a check, because it is never used to set the scale. Its two dots come out 10.7 cm apart in the 3D model and about 11 cm apart from the depth sensor alone across the dive.