Instance-level 6D object pose estimation aims to estimate the 6D pose of a specific object. This research area mainly includes four types of methods: correspondence-based, template matching, voting-based, and regression-based methods. Correspondent-based methods can be divided into sparse correspondence methods\cite{wang2019densefusion} and dense correspondence methods\cite{Li_Wang_Ji_2019}. These methods establish sparse or dense correspondences between the image or point cloud and the CAD model and use PnP or least squares to solve the 6D pose. Template-based methodsSundermeyer_Marton_Durner_Brucker_Triebel_2018, Li_Lin_Jia_2022 match the most similar reference model to the object to be estimated and directly apply the known pose of the model to the target object. Template-based methods effectively address the challenges posed by texture-less objects. Voting-based methodsLiu_Iwase_Kitani_2021, Tian_Pan_Ang_Lee_2020 estimate a set of predefined 2D or 3D key points through pixel-level or point-level voting schemes and establish correspondences with the CAD model to solve the object’s pose. Regression-based methodsGao_Lauri_Wang_Hu_Zhang_Frintrop_2020, Kleeberger_Huber_2020 directly obtain the object’s pose from the features extracted by a network.

The category-level pose estimation method aims to predict the 6D poses of all instances within a specified category without relying on a specific object 3D model.NOCS\cite{wang2019normalized} creates a standard 3D model shared within the class in normalized object coordinate space, establishes dense correspondences between pixel points in the input RGB image and the standard 3D model in the NOCS space through a network, and employs the Umeyama algorithm\cite{Umeyama_1991} to obtain the object's pose and size. Considering the effect of shape variation of objects within a class, SPD\cite{liu2023prior} proposes a 3D model reconstruction strategy based on object deformation. It utilizes RGB-D data and deforms it based on the class shape a priori to adapt to the specific morphology of different instances. This approach effectively addresses the challenges due to shape differences and improves pose estimation accuracy. To reduce the effect of redundant information, SGPA\cite{chen2021sgpa} introduces a key point selection mechanism to reduce the effect of redundant information on estimation results. This method improves pose estimation accuracy by acquiring pixel-point-based fusion features\cite{wang2019densefusion} and establishing sparse correspondence between key points and points in NOCS space. Selecting key points enables the model to focus on the most representative features, further enhancing robustness. Unlike the above approach of building a standard 3D model in NOCS space, unlike the approach of building a standard 3D model in NOCS space, gCasp\cite{ligenerative} employs a generative network to directly generate a 3D model of an object and establish semantic correspondences between the input point cloud and the generated 3D model to solve the object's pose. This approach responds effectively to shape differences of objects within a class and provides a more flexible and adaptive solution.

For a given RGB-D image, the object's RGB image patch and point cloud are acquired by our designed data processing module and used as inputs to the feature extraction module(Sect. 7). The feature extraction module employs CNN\cite{he2016deep} and 3D-CGN\cite{Lin_Huang_Wang_2020} as encoders for two different inputs. In the decoding phase, step-by-step fusion is performed using a deep cross-modal feature fusion module(DCF), which effectively aggregates semantic features from different modalities(Sect. 8). Next, the DenseFusion\cite{wang2019densefusion} module combines RGB appearance features and point cloud geometric features at each point, resulting in point-wise fused features. A multi-layer perceptron (MLP)\cite{taud2018multilayer} then extracts key points of the point cloud in the camera coordinate system, while the 3D model generation module\cite{hao2020dualsdf} constructs the 3D model in the object coordinate system(Sect. 9). Finally, by aligning the 3D models in the camera and object coordinate systems, gets the 6D pose of the object(Sect. 10).

To address the previous problem of biased pose estimation results due to the data processing module's inability to obtain accurate masks for unseen objects within the category, as shown in Fig. 2(b), we propose a new end-to-end data processing module in conjunction with the SAM large-model\cite{kirillov2023segment} instance segmentation method, which greatly improves the ability to cope with the generalization of objects within the category.

As shown in Fig. 2(b) module, the object detection algorithm\cite{cheng2024yolo} acquires the bounding box of the object denoted as $N\times4$, where $N$ is the number of objects in the image, and 4 denotes the coordinate information of the upper-left and lower-right corners$(x_{left}, x_{right}, y_{top}, y_{bottom})$ of the bounding box. The position encoder\cite{Tancik_Srinivasan_Mildenhall_Fridovich-Keil_Raghavan_Singhal_Ramamoorthi_Barron_Ng_2020} extracts the object position feature as $F_p{\in}\mathbb{R}^{N\times2\times256}$, where 2 denotes the two corner points of the bounding box.

The input RGB image is first filled to size, and the image features extracted by the VIT encoder\cite{dosovitskiy2020image} pre-trained with MAE\cite{He_Chen_Xie_Li_Dollar_Girshick_2022}. Two different semantic information of and are fed into the transformer-based network module for fusion. A self-attention mechanism\cite{vaswani2017attention} is first employed to enhance the capability of global representation. Subsequently, the position features are used as query in the cross-attention module\cite{liu2019cross}, and the image features $F_i$ are used as key and value to fuse the information of two different modalities, position and image. In order to further enhance the focus of image features on position information, the fusion features after MLP processing are used as the key and value of the subsequent cross-attention module.

In contrast, the original image features are used as a query to obtain the final fusion features. The fused features are mapped to by the up-sampling module, where and are the height and width of the input image, respectively. Finally, a series of fully connected layers are used to calculate the probability that each pixel in the object location region belongs to the mask and generate the final mask.

By employing a mask decoder based on the Transformer architecture and training on a large-scale mask dataset, the model can accurately extract the object's mask even in the face of objects with significant variations in texture and shape within the category. This advanced mask extraction method maintains high accuracy in complex and changing scenes, thus providing a more reliable basis for subsequent tasks.

In response to the fact that the previous direct splicing of RGB features corresponding to pixel points with geometric features cannot overcome the interference of missing input modal data and noise in the feature space, we introduce a practical deep cross-modal feature fusion module for RGB-D data in the original backbone network\cite{ligenerative}. The core of this module is to design a new transformer-based cross-modal fusion module called Deep Cross-Modal Feature Fusion Network (DCF).

The RGB features at the i-th layer of the decoder in each feature extraction branch are denoted as , and the point cloud geometric features are taken as the input to the DCF fusion module. For these features with two different semantic structures, we first apply the self-attention mechanism(SA)\cite{vaswani2017attention} to the RGB and geometric features to establish the correlation within each feature, enhancing their global representation ability. Then, the features $F_R$ are unfolded to align their feature space with the geometric features, and both are composed of feature vectors of size . The unfolded feature representation is denoted as , where . A cross-attention mechanism is introduced to reduce the interference from missing modal data and noise. The global feature vector obtained through max pooling is used as the query feature, and the following linear transformations are applied to obtain the query (Q), key (K), and value (V) matrices required for the cross-attention mechanism:

where , and represent the linear transformation matrices for Q, K, and V, respectively. represents the feature obtained from max pooling.

We use the multi-head cross-attention module to align the semantic information of two modalities and obtain network inference features. The multi-head cross-attention mechanism can learn information in parallel from different feature subspaces, thereby capturing a variety of features and dependencies between features in the input data, improving the inference ability of the network model:

where is the cross-attention mechanism, represents the i-th layer in the multi-head cross-attention, denotes the scaling factor and . is the standard normalization function. represents the transformation matrix, and is the number of cross-attention heads. The features and obtained from the cross-attention module are concatenated with the original input features and , respectively, to obtain the semantically aligned features denoted as and . These semantically aligned features are concatenated into a new feature sequence , the input to the Transformer-based module.

Contrary to conventional direct feature concatenation, we utilize a Transformer-based deep network module to perform implicit fusion of cross-modal features. This method enables the reasonable inference of features from missing modalities while minimizing the impact of noise. Following processing through the cross-attention mechanism, the resulting feature representation is labeled as , with L indicating the layer within this mechanism. Feature is subsequently fed into the Transformer-based module , illustrated in Fig. 3, yielding the final implicit fusion feature representation, denoted as $F_{final}$.

where , is the position embeddings. stands for multi-head self-attention, for layer normalization, and for multi-layer perceptron. Ultimately, the feature is divided into two sequences, and . These sequences are then integrated into the original feature branches to serve as the fused features.

In summary, for the two modal features output from the decoding layer of the network, DCF not only establishes the long-term dependency between them, but also implicitly aggregates the key features of the two modal data by inferring the global semantic similarity between appearance and geometric information. This global semantic consistency helps mitigate feature space perturbations caused by missing or noisy modal data. With the help of two independent feature extraction branches and our proposed fusion module, the system is able to implicitly acquire dense point-pair features and pass these features to the DenseFusion\cite{wang2019densefusion} module to generate richer dense fusion features.

For the point-wise fusion features acquired from DCF and DenseFusion, we refer to \cite{ligenerative} by constructing dense correspondences between key points in the camera coordinate system and the 3D model within the generated object coordinate system. To determine the 6D pose of the object, we employ the Umeyama algorithm\cite{Umeyama_1991}. This process ensures accurate alignment and positioning of the 3D model relative to the camera's viewpoint.

To acquire key points in the camera coordinate system, we execute point-wise classification on the fusion features of the input point cloud. The network's loss function is the cross-entropy, as shown in Equation \ref{crossentropy}. In this equation, represents the predicted semantic class label for a point in the point cloud that has been augmented with RGB features, while denotes the ground truth semantic class label.The semantic class label is defined as shown in Equation \ref{class_label}, where denotes the corresponding point of $p_i$ in the NOCS within the training dataset, and represents 3D points selected from 3D models with distinct semantic classes in the training data. NOCS stands for the corresponding points in the Normalized Object Coordinate Space, and denotes the number of semantic classes (totalling 256 classes). By performing point-wise classification, we calculate the average coordinates of input point cloud points that have the same semantic class label and designate these averages as key points in the camera coordinate system.

In contrast to previous methods that establish class-unified 3D models within the NOCS, our method employs a generative network\cite{hao2020dualsdf} to reconstruct the 3D model of the object. As illustrated in Fig. 4, the generated 3D model is composed of 256 semantic primitives, each labeled with distinct semantic class labels, with different colors used to differentiate these labels. By establishing correspondences between the key points in the camera coordinate system and the 3D model in the generated object coordinate system using semantic labels, we can accurately determine the 6D pose of the object.

By leveraging the semantic class correspondence between the key points in the camera coordinate system and the generated 3D model , we apply Equation \ref{estimate_pose} to accurately solve for the 6D pose of the object. In this process, indicates the total number of key points involved in establishing this correspondence.

The proposed method in this paper is evaluated on the NOCS dataset, focusing primarily on the object segmentation within the data processing module and the pose estimation performance of the entire network. The dataset comprises six object categories: bottle, bowl, camera, can, laptop, and mug.

We adopts two evaluation metrics in line with previous work: 3D Intersection over Union (IoU) and pose error. The 3D IoU is used to measure the ratio of the overlapping volume between the predicted 3D bounding box of an object by the network and the ground truth, with commonly used thresholds being 50% and 75%. The pose error measures the rotational and translational errors between the predicted pose by the network and the true pose. Specifically, the rotational error is quantified using angular difference, while the translational error is measured by calculating the displacement distance. Commonly used thresholds for these errors are as follows: , , ,

.

For the data processing stage of acquiring image patches and point clouds of objects, only the existing target detection algorithms need to be fine-tuned to accomplish the task of object detection of the objects in the NOCS dataset. The 2D bounding boxes of the objects are inputted into the SAM network as the prompt information to obtain the mask information. Finally, the image patches and the corresponding point cloud are extracted by the corresponding position of the mask in the original RGB and depth map, and the image block is resized to the specified size while 1024 point cloud points are sampled. For the image feature extraction part, a pre-trained ResNet34\cite{he2016deep} was used as an encoder and combined with a four-level PSPNet\cite{Zhao_Shi_Qi_Wang_Jia_2017} as a decoder. For point cloud geometric feature extraction, 3D-GCN\cite{Lin_Huang_Wang_2020} was chosen as the feature extraction network. Between the layers of the decoder, a deep cross-modal feature fusion module is inserted to effectively fuse the features of both modalities, image and point cloud. The fused features are consistent with the gCasp method to solve the 6D position of the object.

We compares the proposed method with several state-of-the-art algorithms wang2019normalized, tian2020shape, chen2021sgpa, lin2022sar, ligenerative. We compares the proposed method with several state-of-the-art algorithms wang2019normalized, tian2020shape, chen2021sgpa, lin2022sar, ligenerative. NOCS\cite{wang2019normalized} maps pixel points directly to a canonical space through a network; SPD\cite{tian2020shape} and SGPA\cite{chen2021sgpa} utilize shape priors and object deformation predictions to recover the shape of target objects. SAR\cite{lin2022sar} and gCasp\cite{ligenerative} both construct 3D models of objects using generative models and estimate object poses by spatial correspondences between point pairs. Table 1 shows the quantitative results of the CAMERA25 and REAL275 datasets. Overall, the data processing method proposed in this paper achieves superior IoU metrics on both datasets compared to the methods above. By accurately obtaining the target RGB images and point cloud information and combining it with the pose estimation method proposed in this paper, the pose estimation results for target objects outperform other methods.

Figure 5 presents a qualitative comparison of our proposed data processing method against the traditional Mask R-CNN\cite{he2017mask} on the real-world dataset REAL275. The results demonstrate that our method significantly improves the segmentation generalization capability for category-level datasets.

Figure 6 visually compares the pose estimation results on the real dataset between the proposed method in this paper and the benchmark network. It can be seen that through improvements in data processing and feature extraction, there are significant enhancements in both IoU and rotational and translational accuracy.

To validate the effectiveness of our proposed method on the REAL275 dataset, we conducted a series of ablation studies. These experiments aim to evaluate the contribution of each component to the overall performance by progressively removing or modifying key elements of the method.

Effectiveness of the Preprocessing Module. Compared to previous category-level pose estimation methods, the combination of object detection and instance segmentation adopted in our approach demonstrates stronger generalization capabilities in open scenes. Specifically, while maintaining the backbone network for pose estimation unchanged, the data processing method proposed in this paper significantly improves the IoU metric. Detailed results are shown in Table 2.

Effectiveness of the Fusion Module. The choice of fusion strategy has a significant impact on model performance. To validate the effectiveness of the fusion module, we designed four experiments, as shown in the Table 3. We used the traditional Mask R-CNN and our proposed instance segment module as data processing methods during the data processing stage. Building upon gCasp\cite{ligenerative} as the backbone network in the feature fusion stage, we compared dense feature fusion with the cross-modal feature fusion proposed in this paper. The final results demonstrate that using the two-stage approach in the data processing stage and cross-modal feature fusion in the feature fusion stage yields the highest pose estimation accuracy.

begin{table}[h]\caption{Comparison of Pose Estimation Results Using Different Data Processing and Feature Fusion Methods}\label{tab3}\begin{tabular*}{\textwidth}{@{\extracolsep\fill}ccccccccc}\toprule \multicolumn{2}{c}{Data Processing Methods} & \multicolumn{2}{c}{Feature Fusion Methods} & \multicolumn{4}{c}{Results}\\ \hline MaskRCNN & Our & DenseFusion & Our & $5^\circ2cm$ & $5^\circ5cm$ & $10^\circ2cm$ & $10^\circ5cm$\\ \hline \checkmark & & \checkmark & & $48.2$ & $56.1$ & $65.6$ & $78.2$ \\ \checkmark & & & \checkmark & $50.4$ & $57.8$ &$67.4$ & $79.2$ \& \checkmark & \checkmark & & $49.1$ & $57.3$ & $66.2$ & $78.9$ \& \checkmark & & \checkmark & $53.5$ & $60.7$ & $72.7$ & $81.6$ \\\botrule\end{tabular*}\end{table}

Based on the data presented in Tables 2 and 3, we can draw the following conclusions: the data processing module primarily influences the IoU metric in object pose estimation. This module mainly provides information about the location and category of objects. The subsequent backbone network establishes accurate correspondences between the features extracted from pixel points and the 3D points in the 3D model within the object coordinate system to estimate the 6D pose of the object. Therefore, the data processing module and the cross-modal feature fusion module proposed in this paper contribute to significant improvements in object pose estimation, validating the effectiveness of our approach.