Refining fragmented SAM masks into view-consistent object identities for 3D Gaussian Splatting.
Comparison between the ground-truth annotation, the initial SAM mask, and our refined mask on the Figurines scene. In this cluttered roomtop scene with many small objects, the initial SAM mask often produces fragmented or overly fine-grained predictions, whereas our method yields cleaner object-level masks with more consistent identities.
Comparison between the ground-truth annotation, the initial SAM mask, and our refined mask on the Ramen scene. While the initial SAM mask over-segments fine-grained texture patterns such as those on the room, our method suppresses these spurious fragments and recognizes them as part of a single coherent object.
Comparison between the ground-truth annotation, the initial SAM mask, and our refined mask on the Blue Sofa scene. While the initial SAM mask produces fragmented predictions such as sunglasses and piggy-like doll, our method consolidates them into a cleaner and more coherent object-level segmentation.
Comparison between the ground-truth annotation, the initial SAM mask, and our refined mask on the Teatime scene. When looking at the sheep and bear dolls, over-segmented masks are merged into object-level units.
Comparison between the ground-truth annotation, the initial SAM mask, and our refined mask on the room scene. While the initial SAM mask tends to produce unclean boundaries and fragmented parts for neighboring objects, our method refines them into clearer object-level segments with improved consistency.
Comparison between the ground-truth annotation, the initial SAM mask, and our refined mask on the Snacks scene. While the initial SAM mask over-segments fine-grained patterns such as buttons on the calculator or pocky on the snack box, our method successfully merge those fragments and recognizes them as part of a single coherent object.
Comparison between the ground-truth annotation, the initial SAM mask, and our refined mask on the Covered Desk scene. In this cluttered scene with many adjacent objects, the initial SAM mask tends to generate fragmented or inconsistent object parts, whereas our method consolidates them into cleaner object-level masks with improved structural consistency.
2D foundation models like SAM often produce fragmented, view-inconsistent masks, limiting object-level 3D understanding. We refine these masks with semantic, depth, and edge cues, match identities across views, and lift them into a consistent 3D Gaussian feature field. Our method significantly improves cross-view consistency and segmentation quality over existing baselines, while preserving high-fidelity reconstruction.
From multiview images, MCM refines SAM masks with multi-cue evidence, matches identities across views, and lifts them into a view-consistent, object-level 3D Gaussian feature field.
Generates synergistic semantic, geometric, and structural priors from each input view.
Consolidates fragmented SAM masks with a composite score derived from semantic, depth, and edge cues.
Establishes globally consistent object identities across all viewpoints.
Backprojects refined 2D identities into a view-consistent, object-level 3D Gaussian feature field.
MCM scores neighboring mask pairs by semantic, depth, and edge agreement, merging fragments of the same object while strong boundary evidence keeps distinct objects apart.
Semantic and depth agreement push \(S(i,j)\) up; boundary evidence pulls it down — masks merge only when appearance and geometry agree, and stay separated across real object edges.
A framework that reduces SAM's over-segmentation using multiple cues.
Establishes globally consistent object IDs across all viewpoints.
Transfers 2D identities to 3D Gaussians, filtering unreliable assignments via majority voting.
Large gains in object assignment accuracy on LERF, Replica, and real-world scenes.
Our refinement stays faithful to the scene and consistent across viewpoints, shown here against three baselines.
* For clearer visualization, this figure enhances edge visibility and overlays object IDs on the masks.
Unlike the GaussianGrouping baseline, which over-segments and drifts across views, our masks stay clean and consistent as the viewpoint changes.
Comparison on the LERF and Replica datasets.
| Category | Model | PSNR ↑ | SSIM ↑ | LPIPS ↓ | mIoU ↑ | mBIoU ↑ | #Mask ↓ |
|---|---|---|---|---|---|---|---|
| Standard | 3DGS | 28.6 | 0.88 | 0.11 | – | – | – |
| 3D Segmentation | Gau-Group | 28.6 | 0.88 | 0.12 | 0.685 | 0.642 | 9,627 |
| SAGA | 28.2 | 0.87 | 0.15 | 0.712 | 0.658 | 178 | |
| InstanceGaussian | 28.8 | 0.87 | 0.15 | 0.702 | 0.662 | 101 | |
| Feature Field | Feature3DGS | 27.5 | 0.84 | 0.18 | 0.538 | 0.485 | – |
| GARField | 28.1 | 0.87 | 0.16 | 0.710 | 0.615 | – | |
| CF3 | 28.3 | 0.87 | 0.14 | 0.524 | 0.582 | – | |
| Proposed | Ours | 28.6 | 0.89 | 0.13 | 0.728 | 0.677 | 67 |
Bold = best, underlined = second-best.
@inproceedings{park2026consistent,
title={Consistent Scene Understanding in 3D Gaussian Splatting via Multi-Cue Mask Refinement},
author={Park, Hyunjoon and Cho, Donghyeon},
booktitle={International Conference on Pattern Recognition (ICPR)},
year={2026},
url={https://hjpark83.github.io/consisGS.github.io/}
}