Accurate and robust feature description and dense point-wise matching based on feature fusion for endoscopic images

Despite the rapid technical advancement of augmented reality (AR) and mixed reality (MR) in minimally invasive surgery (MIS) in recent years, monocular-based 2D/3D reconstruction still remains technically challenging in AR/MR guided surgery navigation nowadays. In principle, soft tissue surface is smooth and watery with sparse texture, specular reflection, and frequent deformation. As a result, we frequently obtain only sparse feature points that give rise to incorrect matching results with conventional image processing methods. To ameliorate, in this paper we enunciate an accurate and robust description and matching method for dense feature points in endoscopic videos. Our new method first extracts contours of the low-rank image sequences based on the adaptive robust principal component analysis (RPCA) decomposition. Then we propose a multi-scale dense geometric feature description approach, which simultaneously extracts dense feature descriptors of the contours in the original Euclidean coordinate space, the accompanying 3D color coordinate space, and the derived curvature-gradient coordinate space. Finally, we devise a new algorithm for both global and local point-wise matching based on feature fusion. For global matching, we employ the fast Fourier transform (FFT) to reduce the dimension of the dense feature descriptors. For local feature point matching, in order to enhance the robustness and accuracy of the matching, we cluster multiple contour points to form “super-point” based on dense feature descriptors and their spatio-temporal continuity. The comprehensive experimental results confirm that our novel approach can overcome the highlight influence, and robustly describe contours from image sequences of soft tissue surfaces. Compared with the state-of-the-art feature point description and matching methods, our analysis framework shows the key advantages of both robustness and accuracy in dense point-wise matching, even when the severe soft tissue deformation occurs. Our new approach is expected to have high potential in 2D/3D reconstruction in endoscopy.