by Li Yuan, Chinese Academy of Sciences

Credit: CC0 Public Domain

A research group led by Prof. Jia Fucang from the Shenzhen Institute of Advanced Technology (SIAT) of the Chinese Academy of Sciences has proposed a multi-task fine-grained spatiotemporal model that can effectively identify action triplets in laparoscopic gallbladder removal surgery videos.

The study was published in IEEE Journal of Biomedical and Health Informatics on July 27.

In laparoscopic surgery videos, a single frame may contain multiple triplets composed of surgical instruments, surgical actions, surgical targets. These triplets exhibit temporal dependencies and high similarity within different categories, posing significant challenges to the recognition by deep learning models.

In this study, the researchers developed a novel model for surgical action triplet recognition. This model consists of two key components: a framework capable of handling multiple tasks simultaneously in surgical video, and a loss function designed for multiple similar labels. "Both spatial and temporal features within surgical videos are considered by our framework, a departure from previous methods that predominantly focused on spatial features alone," said Prof. Jia.

The proposed model outperformed existing methods, including Triplet, Attention Triplet, and Rendezvous approaches. Compared to the state-of-the-art Rendezvous method, the model achieved average precision improvements of 4.6%, 4.0%, and 7.8% in instrument, action, and organ recognition tasks, reaching 82.1%, 51.5%, and 45.5%, respectively. In the overall triplets' recognition task, the proposed model also improved by 3.1% in average precision, reaching 35.8%. They further demonstrated the effectiveness of different modules through ablation experiments.

"In future work, we aim to enhance recognition accuracy based on the proposed model framework," said Prof. Jia.

More information: Yuchong Li et al, MT-FiST: A Multi-Task Fine-grained Spatial-Temporal Framework for Surgical Action Triplet Recognition, IEEE Journal of Biomedical and Health Informatics (2023). DOI: 10.1109/JBHI.2023.3299321

Provided by Chinese Academy of Sciences