ACE Journal

Long-Horizon Manipulation with Object-Centric World Models

Who this is for: robotics researchers and ML practitioners already comfortable with model-based RL or structured representation learning. You should know what a world model is (Dreamer or similar), have a basic sense of attention mechanisms, and have run at least one manipulation benchmark. If you are coming from model-free RL and are hitting the sample-efficiency wall on tasks that require more than ten sequential actions, this is the upgrade path to study next.

The problem: an 18-step dishwasher sequence

Picture a robot loading a dishwasher. It needs to pick a bowl from the counter, slot it into the lower rack without knocking the cup next to it, pick the cup, place it in the upper rack, find the lid, match it to a container already in the rack, and close the door. Call that 18 actions. With a dense reward only at completion and a model-free policy, the agent must discover, by trial and error, a 18-step sequence where the chance of stumbling on any particular one by random exploration is vanishingly small.

Standard model-free approaches fail here not because they are architecturally wrong but because credit assignment across 18 steps is intractable without an internal model. And monolithic world models, which represent the entire scene as a single dense latent vector, fail for a different reason: when you add a new object or rearrange existing ones, the model has to generalize to a distribution it has never seen because the scene-level vector entangles every object’s identity, pose, and properties together.

Object-centric world models attack both problems. They give the planner a model that can imagine future states efficiently, and they structure that model so its generalizations are compositional rather than memorized.

Mechanics: object-centric representations, world models, and model-based planning

Object-centric representation learning

The foundational idea is that a scene with N objects should be represented as N independently maintained latent states, called slots, rather than one entangled scene vector. Slot Attention, introduced by Locatello et al. at Google Brain [1], was an early demonstration that a soft attention mechanism could partition a visual input into object-aligned slots without explicit supervision. Each slot competes with the others to explain a portion of the image, producing a decomposition that roughly corresponds to objects.

Figure 1 shows how a slot encoder maps a raw observation into per-object representations.

graph TD
    A[Raw image observation] --> B[CNN feature extractor]
    B --> C[Slot Attention module]
    C --> D[Slot 1: mug position, shape, color]
    C --> E[Slot 2: bowl position, shape, color]
    C --> F[Slot 3: gripper state]
    C --> G[Slot K: ...]
    D --> H[Relational dynamics model]
    E --> H
    F --> H
    G --> H

Figure 1. Slot encoder pipeline from raw image to per-object slots. Look for how each slot specializes to one physical entity; the relational dynamics model downstream can then treat them as independent nodes.

Subsequent work at DeepMind (OCLF, STEVE) and at the Max Planck Institute for Intelligent Systems extended slot-based representations to handle partially occluded objects, varying object counts, and 3D spatial structure inferred from monocular or multi-view inputs [2]. The resulting representations disentangle object identity from pose and support the kind of relational reasoning that long-horizon manipulation requires.

Structured dynamics models

Once scene representations are decomposed per object, the dynamics model can be structured to match. Each object’s next state depends on its own current state, its spatial and contact relations to nearby objects, and the robot’s action. Graph neural networks are a natural architecture for this relational dynamics model, with objects as nodes and contact or proximity relations as edges.

The GNN dynamics step looks like this in pseudocode:

# Pseudocode: one GNN dynamics step over object slots
# slots: (N, d_slot) - current per-object latent states
# action: (d_action,) - robot action at this step
# edges: list of (i, j) pairs representing relations

def dynamics_step(slots, action, edges):
    # Broadcast action to each slot
    node_inputs = concat([slots, action.expand_as(slots)], dim=-1)

    # Message passing over contact/proximity graph
    messages = []
    for (i, j) in edges:
        m_ij = edge_mlp(concat([slots[i], slots[j]]))
        messages[i].append(m_ij)

    aggregated = [sum(messages[i]) for i in range(N)]

    # Update each slot independently
    next_slots = [
        node_mlp(concat([node_inputs[i], aggregated[i]]))
        for i in range(N)
    ]
    return stack(next_slots)  # (N, d_slot)

This is pseudocode illustrating the architectural pattern; real implementations use PyTorch Geometric or JAX with vectorized message passing.

Dreamer V3’s world-model planning loop has been adapted for object-centric settings by several groups, substituting graph-structured latents for the original global recurrent state [3]. The planner imagines trajectories entirely in latent space, using the learned dynamics model rather than the environment, so it can evaluate many candidate action sequences cheaply.

Model-based planning over learned dynamics

Given a slot encoder, a relational dynamics model, and a reward head trained on decoded states, the full planning loop runs as follows. Figure 2 shows the closed-loop structure.

graph LR
    Obs[Observation] --> Enc[Slot encoder]
    Enc --> Slots[Per-object slots]
    Slots --> Plan[Latent planner<br/>e.g. CEM or MCTS]
    Plan --> Act[Execute action]
    Act --> Obs
    Slots --> Dyn[Dynamics model]
    Dyn --> Slots2[Imagined next slots]
    Slots2 --> Rew[Reward head]
    Rew --> Plan

Figure 2. Closed-loop model-based planning. The planner never touches the real environment during the search; it only queries the dynamics model and reward head. The outer loop runs the selected action in the real environment and re-encodes the new observation.

Common planning algorithms at this layer include Cross-Entropy Method (CEM), which samples action sequences, evaluates their imagined returns, and refits the sampling distribution toward high-return sequences, and Monte Carlo Tree Search adapted for continuous action spaces. The object-centric structure helps here because the dynamics model can be queried for subsets of objects, making imagination cheaper for scenes with many objects.

Long-horizon credit assignment

The standard credit assignment problem asks: which of the 18 actions caused the reward at step 18? Hindsight experience replay and goal-conditioned rewards both help at the algorithmic level, but the structural advantage of object-centric world models is that subgoals can be expressed as desired slot configurations. “Move the mug to position X” is a constraint on a single slot, not a constraint on the full scene vector. This lets planners decompose a long task into a sequence of short imagined rollouts, each targeting a slot-level subgoal, which dramatically reduces the credit assignment horizon.

Worked example: a planning and rollout loop sketch

The following pseudocode sketches the object-centric Dreamer-style loop for a tabletop rearrangement task. It is architecture-level pseudocode; it uses real research concepts but is not tied to a specific library.

# Pseudocode: object-centric Dreamer-style planning loop
# References the architectural pattern from Dreamer V3 + GNN dynamics

def run_episode(env, slot_encoder, dynamics_model, reward_head,
                planner, horizon=15, cem_samples=512):
    obs = env.reset()
    done = False
    while not done:
        # 1. Encode current observation into object slots
        slots = slot_encoder(obs)          # (N, d_slot)

        # 2. Plan in latent space using the dynamics model
        best_action = planner.plan(
            slots, dynamics_model, reward_head,
            horizon=horizon, n_samples=cem_samples
        )

        # 3. Execute only the first action (receding-horizon MPC)
        obs, reward, done, info = env.step(best_action)

    return info

In the Ravens tabletop rearrangement benchmark (Google Brain) [4], object-centric Dreamer variants complete longer task chains with fewer environment interactions than scene-level world models, particularly when the test configuration differs from training. OCRTOC, a challenge that specifically targets open-world tabletop rearrangement and counting, is the most structured public evaluation available for this class of methods as of early 2026.

The hard part: compounding error, slot binding failure, and sim-to-real

Compounding model error

Every world model accumulates prediction error at each step. In a 5-step plan that error is bounded and manageable. In an 18-step plan, errors compound. By step 12 or 13 the imagined state may be so far from any real state the model was trained on that reward predictions become meaningless. This is not a flaw specific to object-centric models; it is a universal property of learned dynamics. The common mitigations are receding-horizon MPC (re-encode and replan after each real step, as shown in the pseudocode above), uncertainty-aware dynamics models that widen their predictions when they are out of distribution, and keeping the imagination horizon shorter than the task horizon by decomposing into subgoals.

Object binding failures

The slot attention mechanism is soft and stochastic. Under occlusion, under rapid motion, or when two objects are similar in appearance, slots can bind to the wrong objects, or a single object can split across two slots. These binding errors produce dynamics predictions that are internally consistent but physically incoherent. A mug slot that briefly merged with the bowl slot will predict a combined object that does not exist, and any plan built on that imagined state fails when it is executed. Architectures that condition slot attention on object identity tokens or that propagate slot assignment across frames using recurrent priors (as in STEVE [2]) reduce binding instability but do not eliminate it.

Sim-to-real gap in contact dynamics

The deepest practical barrier separating current research from deployment on physical hardware is contact-dynamics sim-to-real. Most object-centric world models are pre-trained on simulation data at scale, using GPU-accelerated physics engines like IsaacGym or the newer Genesis [5]. These simulators model rigid-body contact reasonably well, but the dynamics of a soft gripper closing on a compliant or wet surface, or the friction of a ceramic bowl sliding on a damp rack, are not faithfully reproduced. The world model learns contact dynamics that are physically plausible in simulation and systematically wrong in the real world, and those errors are exactly the kind of compounding error that destroys long-horizon plans. Physics-informed slot architectures that tie slot dynamics to learned object mass and friction estimates improve transfer, at the cost of requiring contact-rich interaction data during pre-training.

Integration and day-to-day use

For a team building a robot workcell, the practical path into this space today looks roughly like this. First, pre-train the slot encoder on a large corpus of simulation rollouts. Genesis and IsaacGym both support parallelized environments at the scale needed (thousands of simultaneous episodes) for the encoder to see enough object variety. Second, fine-tune the dynamics model on a small number of real-robot rollouts to partially bridge the sim-to-real gap. Third, deploy a receding-horizon MPC planner that re-encodes every real observation before planning, so accumulated slot errors do not compound across more than a single planning horizon.

Figure 3 shows how this fits into a typical robotics development workflow.

flowchart TD
    Sim[Physics simulator<br/>IsaacGym / Genesis] -->|parallel rollouts| Enc[Slot encoder<br/>pre-training]
    Enc --> DynPre[Dynamics model<br/>pre-training]
    DynPre -->|fine-tune on| RealData[Real-robot rollouts<br/>small dataset]
    RealData --> DynFT[Fine-tuned dynamics model]
    DynFT --> MPC[Receding-horizon MPC<br/>re-plan every step]
    MPC --> Robot[Robot action]
    Robot -->|observation| Enc2[Slot encoder<br/>inference]
    Enc2 --> MPC

Figure 3. Development workflow from simulation pre-training to real deployment. The fine-tuning stage on real-robot data is the critical bridge; without it, contact-dynamics errors accumulate and long-horizon plans fail on physical hardware.

The replay frequency for fine-tuning matters: collecting fine-tuning data in the specific workcell the robot will operate in, with the specific objects it will handle, outperforms generic real-robot datasets by a wide margin. Object-centric representations reduce the data requirement compared to scene-level models because the encoder generalizes per-object, but collecting at least a few hundred real-robot contact episodes per object category is still necessary to close the contact gap.

Against flat end-to-end policies

Diffusion policies and behavior-cloning transformers (ACT, OpenVLA) learn manipulation directly from demonstrations without an explicit world model or object-centric structure [6]. On short-horizon tasks with abundant demonstrations, they match or exceed object-centric model-based methods. The tradeoff reverses on long-horizon tasks where demonstrations are expensive: a flat policy must see demonstrations of the full 18-step sequence, while an object-centric planner can reuse sub-sequence knowledge compositionally.

Against classical task-and-motion planning (TAMP)

Classical TAMP uses symbolic predicates (on, grasped, clear) and geometric motion planning to decompose long tasks [7]. The decomposition is principled and interpretable, but it requires human-authored symbolic state descriptions and fails when the predicate set is incomplete or when perception errors propagate into symbolic state. Object-centric world models learn their state representation from data and can represent continuous, non-binarized properties, at the cost of losing the interpretability and the formal completeness guarantees.

Against image-based world models

TD-MPC2 and similar image-based world models plan in a learned latent space without explicit object decomposition [8]. They work well when the task requires attending to one or two objects, but generalization to novel object counts or arrangements is weak because the latent state entangles all scene content. Object-centric models sacrifice some single-scene performance for compositional generalization, which is the right trade for manipulation problems where object counts vary.

Where it breaks

Novel object appearances. Compositional generalization holds when new configurations are combinatorial rearrangements of objects the model has seen. When a novel object has substantially different visual appearance or reflective properties, slot attention fails to bind it correctly, and the dynamics model has no prior for its behavior. This is not a failure that fine-tuning on a few examples easily fixes; the encoder needs to see enough diversity to generalize.

High object count. Slot attention’s complexity is quadratic in the number of slots in the vanilla formulation. Practical implementations cap slots at around 10 to 16 per scene. Tasks involving densely packed environments (a loaded dishwasher rack with 20 items) push past this limit. Hierarchical slot architectures that aggregate nearby objects into group slots help, but the problem is not solved as of early 2026.

Rapid dynamics. Slot attention assumes that between two frames, an object moves continuously enough for the slot to follow it. Fast-moving objects (a thrown ball, a bouncing cup) break the continuity assumption and cause binding failures. Manipulation settings with mostly slow, deliberate motions are a better fit than settings with dynamic throws or fast conveyors.

Transparent and reflective objects. Transparent glass and mirrored surfaces break the visual features that slot attention uses to segment objects. A glass bowl may segment as “background” or merge with objects behind it. This is a known limitation of appearance-based slot methods and requires depth sensors or structured-light data to address reliably.

Deformable objects. The slot dynamics model assumes each slot represents a rigid object with a predictable future state. Deformable objects (cloth, dough, cables) do not have a fixed shape, so the slot representation is under-constrained and the dynamics model predictions are unreliable. Deformable manipulation is a separate sub-field with its own representation techniques.

Reward shaping failures. Object-centric models enable subgoal decomposition, but the reward head still needs to correctly evaluate slot configurations at each subgoal. If the reward function does not distinguish “bowl in the rack but rotated 90 degrees” from “bowl placed correctly,” the planner will exploit the ambiguity. Reward misspecification is amplified at longer horizons because the planner has more steps over which to find and exploit a gap between the intended and specified reward.

Further reading

For object-centric representation learning, start with the Slot Attention paper [1] and then read the STEVE and OCLF papers [2] for video extensions. For the planning side, read the Dreamer V3 paper [3] to understand the base world-model planning loop before the object-centric modifications. The Ravens benchmark paper [4] describes the tabletop evaluation environment. For the sim-to-real problem in contact dynamics, the Genesis simulator repository [5] is the most recent GPU-accelerated option as of this writing. TD-MPC2 [8] is the clearest image-based world model to compare against.

Written with AI assistance, editorially reviewed.

References

[1] Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. “Object-Centric Learning with Slot Attention.” Advances in Neural Information Processing Systems, 2020. https://arxiv.org/abs/2006.15055

[2] Singh, G., Wu, Y., and Ahn, S. “Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos.” Advances in Neural Information Processing Systems, 2022. https://arxiv.org/abs/2205.14065

[3] Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J. “Mastering Diverse Domains through World Models.” arXiv preprint, 2023. https://arxiv.org/abs/2301.04104

[4] Zeng, A., Florence, P., Tompson, J., Welker, S., Chien, J., Attarian, M., Armstrong, T., Krasin, I., Duong, D., Sindhwani, V., and Song, S. “Transporter Networks: Rearranging the Visual World for Robotic Manipulation.” Conference on Robot Learning, 2021. https://arxiv.org/abs/2010.14406

[5] Genesis Physics Engine. Genesis-Embodied-AI team. https://github.com/Genesis-Embodied-AI/Genesis

[6] Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware.” arXiv preprint, 2023. https://arxiv.org/abs/2304.13705

[7] Garrett, C. R., Chitnis, R., Holladay, R., Kim, B., Silver, T., Kaelbling, L. P., and Lozano-Perez, T. “Integrated Task and Motion Planning.” Annual Review of Control, Robotics, and Autonomous Systems, 2021. https://arxiv.org/abs/2010.01083

[8] Hansen, N., Su, H., and Wang, X. “TD-MPC2: Scalable, Robust World Models for Continuous Control.” arXiv preprint, 2023. https://arxiv.org/abs/2310.16828

[9] Kipf, T., van der Pol, E., and Welling, M. “Contrastive Learning of Structured World Models.” International Conference on Learning Representations, 2020. https://arxiv.org/abs/1911.12247

[10] Wu, Z., Dvornik, N., Greff, K., Kipf, T., and Garg, A. “Slotformer: Unsupervised Visual Dynamics Simulation with Object-Centric Models.” International Conference on Learning Representations, 2023. https://arxiv.org/abs/2210.05861