Many researchers and developers assume that 2D foundation models like SAM are the ultimate solution for 3D object understanding. The common logic is straightforward: generate masks for every camera view, project them into the 3D space, and let the 3D Gaussian Splatting (3DGS) engine handle the rest. However, this "project-and-pray" approach quickly falls apart in practice. When you actually move the camera within a 3DGS scene, you will often notice that object boundaries flicker and semantic labels shift inconsistently across frames. This multi-view inconsistency is the primary hurdle preventing embodied AI from reliably interacting with the physical world.
The Root of Semantic Fragmentation
The fundamental problem lies in the independent nature of 2D segmentation. Foundation models process each frame in isolation, unaware of the temporal or spatial continuity required for 3D reconstruction. As the camera perspective shifts, changes in lighting, perspective distortion, and occlusions lead to slight variations in the 2D masks. When these inconsistent masks are assigned to the underlying 3D Gaussians, the points receive conflicting semantic signals. One Gaussian might be labeled as a "table" in one frame and "floor" in another.
Technically, this creates a misalignment between the high-fidelity geometry of 3DGS and the noisy semantic labels derived from 2D views. In the context of referring expression segmentation—where a user might ask a robot to "pick up the blue mug"—this noise leads to a fragmented representation where the robot cannot confidently identify the entire volume of the target object. The result is a 3D scene that looks great visually but is semantically broken for any practical task.
Shifting Paradigms: Tracking Before Labeling
To overcome this, we must shift from a "label-then-integrate" workflow to a "track-then-label" strategy. This approach prioritizes the physical continuity of objects over their linguistic labels. The process begins by establishing tracking across multiple views. Instead of asking what an object is, we first ask where it goes as the camera moves. By leveraging the spatial coordinates of 3D Gaussians, we can build a much more stable correspondence between frames than 2D pixels alone could provide.
Once the tracking phase identifies a consistent set of Gaussians that move and appear together as a single entity, we treat them as a unified 3D object. Only after this structural consistency is established do we apply the natural language labeling. By querying the language model against a consolidated 3D entity rather than individual, noisy 2D frames, the system gains a holistic understanding. This effectively filters out the per-view noise and ensures that the entire object responds to a command, regardless of the viewing angle.
Navigating the Trade-offs of Consistency
Implementing a track-then-label system involves clear trade-offs. The primary cost is computational overhead. Maintaining and updating tracks across thousands of Gaussians and hundreds of frames requires significant memory and processing power. Based on qualitative evaluations in complex environments, this method demands more rigorous optimization than simple per-view projection to maintain acceptable processing speeds. There is also the risk of "tracking drift," where an initial error in grouping Gaussians can lead to an entire object being misidentified or merged with its background.
Furthermore, the complexity of the pipeline increases. Developers must manage the interaction between the 3DGS rendering engine, the tracking algorithm, and the language-vision model. This is a significant jump from simply running an off-the-shelf segmenter on a set of images. However, for applications in robotics where a single miscalculation can lead to a physical collision, the investment in semantic stability is non-negotiable.
Verifying 3D Semantic Integrity
How do we know if this approach actually works? The most critical metric is multi-view IoU stability. If you rotate the camera 360 degrees around an object, the segmentation mask should remain nearly identical in its 3D projection. Any significant drop in the intersection-over-union across different viewpoints indicates a failure in maintaining consistency.
Another practical test involves open-world referring queries. Try using ambiguous or highly specific descriptions like "the scratched metal container near the window." A successful system will activate the same set of 3D Gaussians regardless of whether the camera is close to the window or across the room. True 3D intelligence is achieved when the semantic identity of an object is as permanent as its geometry. If your 3D objects are still "shimmering" with label uncertainty, it's time to stop focusing on the labels and start focusing on the tracks.
Reference: arXiv CS.LG (Machine Learning)