Abstract

A robot tasked with fetching a mug from the kitchen counter must reason about objects, their spatial relationships, and the affordances those relationships create - which surfaces support which objects, which drawers contain which items, and which configurations allow a gripper approach without collision. Occupancy grids and point clouds answer where things are geometrically; they say little about what things are or how they relate. Scene graphs encode this relational structure explicitly, representing objects as nodes and spatial or semantic relationships as edges. Mobile manipulation systems that build and query such graphs are showing improved performance on long-horizon fetch-and-place tasks compared to geometry-only representations.

Scene Graph Construction from Perception

Building a scene graph incrementally from robot sensor data requires three interleaved processes: object detection and segmentation, 3D localization of detected objects, and relationship classification between pairs of objects. Modern pipelines treat the first two as a joint problem, using instance segmentation networks - commonly Mask R-CNN or its successors fine-tuned on indoor object datasets like ScanNet - to simultaneously identify and localize instances in the RGB-D stream.

Relationship classification is the harder step. Spatial predicates such as “on,” “in,” “next to,” and “behind” must be inferred from 3D bounding box geometry, but semantic relationships - “contains,” “supports,” “is-part-of” - require broader scene context. Neural scene graph generation models, including work from the Stanford Vision and Learning Lab, use graph attention networks that propagate context between object nodes to predict edge labels. These run at rates sufficient for incremental update (3-5 Hz) on an NVIDIA Orin-class edge GPU.

Graph Querying for Task Planning

Once a scene graph is available, task planning can be expressed as a sequence of graph queries and updates. Fetching the mug requires finding the node labeled “mug,” identifying the “on” edge that locates it relative to a surface, and computing a collision-free approach path conditioned on the robot’s current base pose in the same graph. Task and Motion Planning (TAMP) systems such as PDDLStream, developed at MIT, integrate symbolic planning over scene-graph predicates with geometric motion planning, bridging the gap between high-level task decomposition and low-level trajectory execution.

Scene graph representations also support natural language grounding. Language instructions like “bring me the red book near the lamp” can be parsed into graph query patterns using the MDETR or OWL-ViT family of grounded detection models, which return object nodes matching both categorical and relational constraints. Research groups at the University of Washington and at Toyota Research Institute have demonstrated this pipeline on assistive manipulation tasks in home-like environments, with instruction completion rates noticeably higher than systems relying on flat object lists.

Maintaining Consistency Under Robot Motion and Scene Change

A persistent challenge is keeping the scene graph consistent as the robot moves and as the environment changes. When the robot base relocates, object positions in the graph must be updated relative to the new base pose using odometry or SLAM estimates. Object removal - someone moves the mug while the robot is navigating - must be detected and the corresponding node invalidated. Change detection from successive RGB-D frames, combined with a temporal decay on unobserved nodes, provides a practical heuristic, though it introduces latency proportional to the observation interval.

Graph neural network approaches to maintaining consistency, where node embeddings encode both geometric state and observation recency, are an active research direction. The key appeal is differentiability: a learned consistency model could, in principle, be fine-tuned on in-deployment experience without manual re-engineering of the update rules.