A framework for discovering categories using images and 3D shapes
The RaGCD framework uses reliability cues to combine visual appearance and spatial geometry for better object recognition.
Researchers have developed a framework called RaGCD. This system aims to perform Generalized Category Discovery (GCD). GCD involves recognizing known categories while also discovering novel ones from partially labeled data.
In a GCD setting, labeled samples only cover a subset of categories. The unlabeled set contains both known and novel categories. The model must transfer knowledge from labeled examples without limiting the unlabeled data to known labels.
Previous studies on GCD mostly focused on unimodal observations. Some recent research used image-text interactions. Other work looked at industrial scenarios using images and point clouds. However, the link between visual appearance and 3D geometry remains underexplored.
Images and point clouds provide different information about the same object. Images capture color, texture, and appearance-level semantics. Point clouds provide explicit information about spatial configuration and shape.
These two types of data can help solve different ambiguities. For example, reflections or illumination changes may obscure visual appearance. In those cases, geometric evidence might be informative. Conversely, a closed laptop and a hardcover book may look similar in sparse point clouds. Visual appearance can provide additional semantic cues for those objects.
The usefulness of these modalities can vary for different samples. This is called cross-modal reliability mismatch. Images may have ambiguous or degraded appearance cues. Point clouds may suffer from incompleteness, sparsity, or geometric ambiguity.
This mismatch is a challenge for novel categories. In these cases, labeled supervision is not available. This makes it hard to identify misleading evidence or correct learning signals. Unreliable evidence can hurt multimodal fusion and cross-modal knowledge transfer.
RaGCD addresses these issues using two main components. The first is Reliability-aware Modality Allocation (RMA). RMA combines modality-specific features with prediction-derived reliability cues. These cues include predictive uncertainty and cross-modal disagreement.
RMA uses a learned channel-wise allocation. It combines this with an explicit reliability prior. This creates a fused discovery representation. This design allows the model to adjust how much each modality contributes to a sample.
The second component is Reliability-conditioned Alignment (RCA). RCA regulates cross-modal learning at the feature, prediction, and prototype levels. It reweights feature and prediction losses based on sample reliability. It also selects reliable observations for prototype-level alignment.
RMA is a method that decides how much appearance and geometric information should contribute to a fused representation. RCA is a method that determines which training samples should contribute to cross-modal alignment.
Experiments were conducted on ModelNet40, Washington RGB-D, and ShapeNet-ViPC. These datasets include synthetic CAD objects, real RGB-D observations, and partial geometry. RaGCD achieved the highest overall and novel-category clustering accuracy compared to other methods.
Key facts
- RaGCD uses both images and point clouds for category discovery.
- The framework uses Reliability-aware Modality Allocation (RMA) to adjust modality contributions.
- Reliability-conditioned Alignment (RCA) regulates learning at feature, prediction, and prototype levels.
- RaGCD outperformed other methods on ModelNet40, Washington RGB-D, and ShapeNet-ViPC datasets.