Hydra++: Real-Time Hierarchical 3D Scene Graph Construction With Object-Level Shape Estimation

Abstract

3D scene graphs provide a hierarchical abstraction of environments by encoding spatial entities and their relationships. Existing scene graph systems usually model object geometry with partial point clouds or class-level CAD templates, limiting instance-specific shape detail. Hydra++ integrates category-agnostic shape estimation into a hierarchical 3D scene graph pipeline, adds a reprojection-mask consistency check to reject degenerate shape predictions, and supports a hybrid LiDAR-camera configuration for large-scale outdoor operation. Experiments in simulation and real-world outdoor scenes show improved object-level and scene-level reconstruction quality.

Motivation

Scene-level abstractions are useful for reasoning about semantic context, object relations, and navigable structure, but they are often too coarse for manipulation. Mesh-level geometry is directly actionable for interaction and planning, yet it usually lacks the relational context needed for whole-scene understanding. Hydra++ targets this gap by storing completed object-level geometry inside a hierarchical 3D scene graph.

What Hydra++ adds

Pipeline

Hydra++ builds on the Hydra and Khronos 3D scene graph pipeline. It maintains an active-window metric-semantic map, tracks object candidates over time, then invokes a shape estimator when an object track leaves the active window. Accepted object meshes are inserted into the hierarchical scene graph together with places, regions, and spatial relationships.

Outdoor TSDF with Hybrid LiDAR-Camera

In the hybrid configuration, 3D LiDAR supplies metric depth for TSDF scene mapping, while an RGB or RGB-D camera provides image observations for semantic segmentation and object-level shape estimation. At long range, sparse LiDAR rays strike the ground at high incidence angles, causing projective signed distances to extend beyond the standard truncation band.

Standard TSDF integration then leaves voxels behind the ground surface unobserved, so the sign transitions required for zero-level-set extraction can disappear. Hydra++ applies ground-aware adaptive integration only to ground-labeled voxels: measurements within an additional negative band are clamped to the truncation boundary and integrated with a small weight. This controlled negative support restores the sign transitions needed to reconstruct a continuous ground mesh without changing the standard integration band for non-ground geometry.

Standard TSDF integration

Ground-aware adaptive TSDF integration

Reprojection-Mask Consistency Check (RMCC)

The reprojection-mask consistency check (RMCC) samples points from the predicted mesh, reprojects them into the selected image, and compares the reprojected mask with the observed object mask. Predictions with insufficient overlap are rejected, filtering degenerate shapes caused by partial masks, over-segmentation, or poor viewpoints.

Accepted prediction

Rejected prediction

Evaluation

Hydra++ is evaluated on uHumans2 for indoor object-level scene graph quality and on Kimera-Multi for outdoor hybrid sensor operation. Unlike the original Hydra evaluation, which focused on batch and incremental scene graph construction modes, this evaluation explicitly compares predicted object nodes against real 3D object instances by measuring distances in 3D space. Object detections are matched by same-class centroid distance, while matched meshes are evaluated using Chamfer distance, 3D bounding-box IoU, and volumetric IoU.

Reference

SlideSLAM

Hydra

Khronos

Hydra++

Values are reproduced from the current manuscript draft; final camera-ready numbers remain subject to the paper version.

SlideSLAM

Hydra

Khronos

Hydra++

Shape Model Trade-Offs

The framework is estimator-agnostic. CRISP is the default model for real-time operation over known object classes, while SAM3D shows stronger reconstruction on novel out-of-domain instances at much higher inference latency.

uHumans2

Kimera-Multi

Outdoor Hybrid Mapping in MIT Campus

Outdoor reconstruction combines LiDAR-driven metric mapping with camera-driven object shape estimation. The hybrid mode improves scene mesh continuity and recovers object-level geometry for fire hydrants, cars, and benches in the Kimera-Multi sequence.

These outdoor results should be read as a deployment study rather than a complete outdoor benchmark. A more rigorous outdoor evaluation remains important because public datasets rarely provide scene-scale, instance-level mesh ground truth from a mapping perspective, and mask-based checks such as RMCC do not explicitly verify metric depth consistency. Future outdoor benchmarks should quantify both object detection and mesh accuracy under noisy segmentation, heavy occlusion, and geometrically plausible but incorrect shape predictions.

RGB-D only

Hybrid

Hybrid + adaptive

Fire hydrant

Car

Bench

RGB input

Khronos

Hydra++

BibTeX

@inproceedings{Lim2026iros-Hydrapp,
  title        = {{Hydra++}: Real-Time Hierarchical 3D Scene Graph Construction With Object-Level Shape Estimation},
  author       = {Lim, Hyungtae and Hughes, Nathan and Yu, Xihang and Xu, Ruihan and Chang, Yun and Shi, Jingnan and Talak, Rajat and Carlone, Luca},
  booktitle    = {Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year         = {2026}
}