ACE Journal

Foundation Models for Robot Task Planning

Who this is for: robotics and ML engineers integrating LLMs or VLMs into a real robot stack operating in a structured environment, warehouses, labs, logistics floors, where the set of objects, locations, and primitives is known and enumerable. The assumption is that you already have a working perception and motion stack, the robot can pick, place, and navigate on command, and you want a reasoning layer on top that turns instructions into plans without retraining what already works. This is not a survey of end-to-end learned policies. It is the integration pattern for bolting a planner onto a robot you already trust to move.

The instruction that the motion planner cannot read

A modern warehouse manipulator is good at the things it was built for. Tell it to pick a specific object at a known pose and place it at another known pose, and it does that reliably, with a motion planner that respects joint limits and a grasp model tuned over thousands of trials. What it cannot do is read “clear the returns from shelf B and sort them into the right bins” and turn that into the dozen pick-and-place operations it implies. There is a gap between the instruction a human gives and the primitives the robot exposes, and for years that gap was filled by a person writing a task script by hand.

Large language models close that gap by reasoning over the instruction in natural language and emitting a plan in terms of the primitives the robot already has. This is the shift of the last two years: the language model is not learning to control motors, it is learning to sequence calls to controllers that already work. That framing is what makes the pattern shippable. You are not betting your manipulation reliability on a transformer, only your task sequencing, and the parts that have to be safe and fast stay in the classical stack where they belong.

The catch is that a language model left to its own devices produces plans that are fluent and wrong. It references a shelf that does not exist, assumes an object that was never seen, or sequences two actions in an order the robot physically cannot perform. The whole engineering discipline of this article is the set of guardrails that keep a capable but ungrounded reasoner from issuing instructions the robot will dutifully and disastrously try to follow. Three ideas carry most of the weight: affordance-grounded decomposition, vision-language grounding kept in the slow loop, and schema-constrained replanning.

Mechanics: decomposition grounded by affordances

The core pattern is prompt-based decomposition. You give the model the task, the inventory of primitives, and a serialized snapshot of the current world state, and you ask for an ordered plan. The serialization is the part teams underweight. A language model cannot reason about partial completions or recover from a half-finished task unless the prompt tells it what the robot can currently see and is currently holding. So you flatten the robot’s sensor state, object positions, gripper status, recent action history, into the prompt context on every planning call.

A decomposition prompt template, kept deliberately rigid so the model has little room to improvise structure:

You are a task planner for a warehouse manipulator. You output an ordered
plan using ONLY the primitives and objects listed below. You never invent
objects, locations, or primitives that are not in the registry.

PRIMITIVES (the only verbs you may use):
  - navigate(location_id)
  - pick(object_id)
  - place(object_id, location_id)
  - inspect(location_id)

REGISTRY (the only ids that exist):
  objects:           # e.g. ["box_red_01", "box_blue_02"]
  locations:       # e.g. ["shelf_b", "bin_1", "bin_2", "bin_3"]

CURRENT WORLD STATE:
  gripper_holding:      # object_id or "empty"
  robot_at:        
  observed:            # facts confirmed by perception

TASK: 

Respond with a JSON object matching the action schema. If the task cannot be
completed with the available primitives and registry, return an empty plan
with a "blocked_reason" string. Do not guess.

Decomposition alone is not enough, because the model has no idea whether the plan it just wrote is physically executable. That is the job of affordance scoring, the idea SayCan introduced [1]. For each candidate next action, you compute an affordance: a number in [0, 1] estimating the probability the robot can succeed at that action from the current state. The grasp model scores how reachable and graspable an object is; the navigation stack scores whether a location is approachable; a precondition checker scores whether the action even makes sense given what the gripper is holding. You combine the language model’s preference for an action with its affordance, and you pick the action that is both wanted and possible.

The combination is a product, which is the elegant part. An action the model loves but the robot cannot do (high language score, near-zero affordance) drops out, and so does an action the robot could trivially do but that has nothing to do with the task. Only actions that score on both axes survive. Figure 1 shows the full cycle from instruction to execution, including the replan branch triggered when an outcome does not match expectations.

flowchart TB
    Instr[instruction + world state] --> LLM[LLM proposes<br/>candidate next actions]
    LLM --> Cand[candidate actions<br/>+ language scores]
    Cand --> Aff[affordance model<br/>scores feasibility]
    Aff --> Comb[combine:<br/>language x affordance]
    Comb --> Pick[select highest<br/>combined score]
    Pick --> Exec[execute primitive]
    Exec --> Check{outcome as<br/>expected?}
    Check -->|yes| Done{task<br/>complete?}
    Check -->|no| Replan[replan with<br/>updated state]
    Replan --> LLM
    Done -->|no| LLM
    Done -->|yes| End[stop]

Figure 1. Affordance-grounded execution cycle: the LLM proposes candidates, the affordance model filters by feasibility, and unexpected outcomes loop back to replanning rather than halting.

Here is the affordance-scoring step in Python. The language model returns candidate actions with log-probabilities; each candidate is scored for feasibility against the live robot state; the combined score decides what executes next.

from dataclasses import dataclass

@dataclass
class Candidate:
    action: dict          # e.g. {"primitive": "pick", "object_id": "box_red_01"}
    language_score: float # model's preference, normalized to [0, 1]

def affordance(action: dict, state: "RobotState") -> float:
    """Probability the robot can execute `action` from `state`, in [0, 1].

    Each branch delegates to a classical model the team already trusts:
    the grasp planner, the navigation stack, and a precondition checker.
    """
    prim = action["primitive"]
    if prim == "pick":
        if state.gripper_holding is not None:
            return 0.0                       # cannot pick with a full gripper
        return state.grasp_model.success_prob(action["object_id"])
    if prim == "place":
        if state.gripper_holding != action["object_id"]:
            return 0.0                       # not holding the thing to place
        return state.nav_model.reachable_prob(action["location_id"])
    if prim == "navigate":
        return state.nav_model.reachable_prob(action["location_id"])
    if prim == "inspect":
        return state.nav_model.reachable_prob(action["location_id"])
    return 0.0                               # unknown primitive: not affordable

def select_next(candidates: list[Candidate], state: "RobotState") -> dict | None:
    scored = [
        (c.language_score * affordance(c.action, state), c.action)
        for c in candidates
    ]
    scored.sort(key=lambda x: x[0], reverse=True)
    best_score, best_action = scored[0]
    # A near-zero combined score means nothing is both wanted and feasible.
    return best_action if best_score > 0.05 else None

The select_next function returning None is itself a signal: nothing the model proposed is feasible from here, which is a cue to replan or to surface the situation to a human rather than to flail.

Vision-language grounding, kept in the slow loop

Pure language planning breaks the moment object identity depends on what the camera sees. “The partially filled blue bin” or “the box nearest the conveyor edge” cannot be resolved from the registry alone; they require connecting language to pixels. Vision-language models close this gap. You hand a VLM a live camera frame and a spatial question, and it returns an answer you fold back into the planner as a structured fact. PaliGemma [2] and the open-weight InternVL family [3] are the practical workhorses here, capable enough at visual question answering to resolve the kind of scene-grounded query a warehouse task throws at them.

The architectural rule that matters more than the model choice: keep the VLM in the slow planning loop, never in the reactive control loop. A VLM inference that takes on the order of several hundred milliseconds, call it 800ms as an illustrative order of magnitude, is perfectly fine for a task-level decision you make once per plan step. It is catastrophic in a control loop that has to close at hundreds of hertz to keep a grasp stable. So the system runs two loops at two timescales. The slow loop reasons about the task with the LLM and the VLM. The fast loop runs classical perception and control, the grasp servoing, the collision checking, the joint-space tracking, at the rate the hardware demands. Figure 2 shows the two loops and the narrow interface between them.

flowchart LR
    subgraph slow["slow planning loop (~1 Hz, hundreds of ms ok)"]
        direction TB
        LLM2[LLM task planner]
        VLM2[VLM scene grounding]
        Reg2[object / location registry]
        LLM2 <--> VLM2
        LLM2 <--> Reg2
    end
    subgraph fast["fast control loop (100s of Hz)"]
        direction TB
        Perc[classical perception]
        Ctrl[motion control + grasp servo]
        Perc --> Ctrl
    end
    slow -->|"next primitive<br/>(pick, place, navigate)"| fast
    fast -->|"outcome + state<br/>(succeeded / failed)"| slow

Figure 2. Two-loop architecture separating planning-rate reasoning (slow loop, ~1 Hz) from control-rate execution (fast loop, 100s of Hz); note that the interface carries only a single primitive in each direction.

The contract between the loops is narrow on purpose. The slow loop hands down a single primitive to execute; the fast loop reports back whether it succeeded and what the resulting state is. The VLM never touches the servo rate, and the controller never waits on a transformer. Crossing that boundary, letting a planning-rate model into a control-rate decision, is the single most common way teams new to this turn a working robot into a hesitant one.

The grounding pipeline itself is a short hop: a frame from the camera, a templated question to the VLM, a parse of the answer into a typed fact, and an assertion of that fact into the world state the planner reads. Figure 3 traces that pipeline from camera to planner.

flowchart LR
    Cam[camera frame] --> VLM[VLM: which bin<br/>is more than half full?]
    VLM --> Parse[parse answer to<br/>typed fact]
    Parse --> Fact["fact:<br/>bin_2.fill > 0.5"]
    Fact --> WS[(world state)]
    WS --> Plan[task planner]

Figure 3. VLM grounding pipeline converting a live camera frame into a typed fact that the task planner can act on; closed questions keep the answer parseable and registry-constrained.

Two disciplines keep this honest. Ask the VLM closed questions whose answers map onto the registry, “which of bins 1 through 5 is more than half full” rather than “describe the scene,” so the answer is parseable and constrained to ids you can validate. And treat a VLM answer as a fact with a confidence, letting low-confidence answers trigger a re-inspection rather than silently entering a planning decision. A grounding error that flows unchecked produces a perfectly valid plan built on a false premise, which is harder to debug than a plan that was malformed.

The hard part: schema-constrained replanning that kills hallucinations

Foundation models earn their place precisely when the world does not match the plan. An object is missing, an item was dropped, an instruction was ambiguous and the scene resolved it differently than expected. The replanning loop re-queries the LLM with the updated world state and the original goal, and gets a revised sequence. This is the capability classical task scripts lack: a hand-written script that assumed the red box was on shelf B has no recourse when the box is not there, while a planner re-reasons from the actual state.

The danger is exactly proportional to the capability. A model free to generate plans from updated state is also free to hallucinate, and under the pressure of an unexpected situation it will confidently produce an invalid plan: place the object in bin_7 when the map has bins 1 through 5, or pick box_green when no green box was ever observed. Left unchecked, the robot tries to execute the impossible, and you spend an afternoon debugging a motion failure whose root cause was a fluent sentence.

The fix that does not require a fine-tune is to constrain the model output to a formal action schema and validate every plan against the robot’s own registry before any of it executes. The schema makes fields enum-typed: a primitive must be one of the four verbs, an object id must be one of the ids that exist, a location id must be one of the locations in the map. A plan that names anything outside those enums fails validation and is rejected, and the rejection itself becomes a replanning signal, you hand the validation error back to the model and ask it to try again within the registry.

Here is the constrained JSON action schema. It is a JSON Schema document, which most modern model APIs can enforce at decode time as a response format, so the model is structurally prevented from emitting a non-conforming plan in the first place. Even when the API enforces structure, you still validate semantically against the live registry, because the API can guarantee “is a string from this enum” but not “is an object that is actually present right now.”

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "RobotTaskPlan",
  "type": "object",
  "required": ["plan"],
  "additionalProperties": false,
  "properties": {
    "plan": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["primitive"],
        "additionalProperties": false,
        "properties": {
          "primitive": {
            "type": "string",
            "enum": ["navigate", "pick", "place", "inspect"]
          },
          "object_id": {
            "type": "string",
            "enum": ["box_red_01", "box_blue_02", "box_green_03"]
          },
          "location_id": {
            "type": "string",
            "enum": ["shelf_a", "shelf_b", "bin_1", "bin_2", "bin_3"]
          }
        }
      }
    },
    "blocked_reason": { "type": ["string", "null"] }
  }
}

The enums for object_id and location_id are not hand-written, they are generated from the registry at plan time, so the schema always reflects the current world. A new pallet of objects arrives, the registry updates, and the next plan’s schema enumerates the new ids automatically. The validation step that backs this up checks both structure and live presence:

import jsonschema

class PlanValidationError(Exception):
    pass

def build_schema(registry: "Registry") -> dict:
    """Generate the action schema with enums drawn from the live registry."""
    return {
        "type": "object",
        "required": ["plan"],
        "properties": {
            "plan": {
                "type": "array",
                "items": {
                    "type": "object",
                    "required": ["primitive"],
                    "properties": {
                        "primitive": {"enum": ["navigate", "pick", "place", "inspect"]},
                        "object_id": {"enum": registry.object_ids()},
                        "location_id": {"enum": registry.location_ids()},
                    },
                },
            },
            "blocked_reason": {"type": ["string", "null"]},
        },
    }

def validate_plan(plan: dict, registry: "Registry") -> list[dict]:
    """Validate structure against the schema, then re-check every id against
    the live registry. Returns the step list, or raises with a reason the
    model can act on during replanning."""
    schema = build_schema(registry)
    try:
        jsonschema.validate(plan, schema)
    except jsonschema.ValidationError as e:
        raise PlanValidationError(f"schema violation: {e.message}")

    for i, step in enumerate(plan["plan"]):
        obj = step.get("object_id")
        loc = step.get("location_id")
        if obj is not None and not registry.has_object(obj):
            raise PlanValidationError(f"step {i}: unknown object_id '{obj}'")
        if loc is not None and not registry.has_location(loc):
            raise PlanValidationError(f"step {i}: unknown location_id '{loc}'")
    return plan["plan"]

When validate_plan raises, the loop does not give up. It feeds the error string back into the next planning prompt (“your previous plan referenced unknown location_id ‘bin_7’; the only locations are …”), and the model, now told exactly what it got wrong, almost always corrects on the next pass. This closed loop, generate, validate, feed back the error, regenerate, is what makes the hallucination problem manageable without ever touching model weights.

The last piece is unglamorous and the one most worth doing from day one: log every planning call. The input world state, the prompt, the raw model output, the validation result, the affordance scores, and the eventual execution outcome. Each logged call is a labeled example. The failures, the hallucinated bins, the grounding errors, the plans that validated but did not work, are exactly the dataset you want if you ever decide to fine-tune, and they are also your debugging trail when a deployed robot does something surprising. The logging is cheap and the dataset compounds.

Integration: where each model sits in the stack

The integration has a clean shape. The slow loop owns reasoning and grounding: the LLM proposes, the VLM grounds against the camera, the registry constrains, the validator rejects, and a single primitive drops down to the fast loop, which executes and reports an outcome back up. The registry sits between them as the shared source of truth about what exists, and it is the same registry that generates the schema enums, so exactly one place defines the robot’s world and everything else derives from it.

A practical note: run the LLM and VLM as separate services behind the planner, not in the robot’s real-time process, which should never block on a network call to a model. While a plan request is out to a model service, the robot is either holding still in a safe state or finishing its current primitive. This is another expression of the two-loop discipline. The real-time guarantees live entirely in the fast loop, and the slow loop is allowed to be slow because nothing safety-critical waits on it.

This pattern sits in a crowded and fast-moving space, and it is worth being clear about the alternatives.

SayCan [1] is the direct ancestor of the affordance-grounded decomposition described here. Its contribution was the insight that a language model’s “what should I do” should be multiplied by a value function’s “what can I do,” and that product is the through-line of this whole article. The version here is a pragmatic restatement of that idea with a hardened schema layer on top.

Code-as-Policies [4] takes a different route to the same goal. Instead of emitting a plan as data, the model writes executable code that calls the robot’s API directly, using the language model’s strength at programming to express loops, conditionals, and reactive logic that a flat action list cannot. It is more expressive and correspondingly harder to constrain: a generated program can do anything the API allows, so the validation story is murkier than enum-checking a JSON plan. The choice between data-plan and code-plan is largely a choice between constrainability and expressiveness.

Vision-language-action models like RT-2 [5] and the open-weight OpenVLA [6] collapse the whole stack into a single learned policy that maps pixels and instructions directly to actions, no explicit decomposition, no separate affordance model. When they work, they are remarkably general, and they sidestep the hallucinated-plan problem because there is no discrete plan to hallucinate. The cost is that they are end-to-end learned, so they require large robot-data collection to train or adapt, they are harder to inspect and to constrain to a known-safe action set, and a failure is a failure of the whole policy rather than of a legible plan step. The pattern in this article is the right call when you already have a trusted motion stack and want reasoning bolted on; a VLA is the right call when you are willing to learn the whole map from data and want maximum generality.

Classical PDDL and task-and-motion planning [7][8] solve the planning problem with formal symbolic planners and search, no language model at all. They are sound and complete within their domain model: if a PDDL planner returns a plan, that plan is valid by construction, which is exactly the guarantee the schema-validation layer is trying to approximate. What they lack is the language model’s flexibility at parsing a vague instruction and grounding it against a messy visual scene. The most robust systems being built now are hybrids: a language model parses the instruction and proposes structure, and a classical planner or validator enforces soundness. The schema-and-registry approach here is a lightweight step in that direction.

The even-handed summary: foundation-model planning buys flexibility and visual grounding at the cost of guardrails against hallucination, classical planning buys soundness at the cost of brittleness outside the domain model, and end-to-end VLAs buy generality at the cost of data and inspectability. No single approach dominates. The right choice is dictated by which guarantee you most need and which cost you can most afford.

Where it breaks

The pattern is real and shippable in structured environments, and it still fails in recognizable ways. Knowing them up front is most of avoiding them.

Hallucinated plans. The headline failure. A model under pressure invents objects, locations, or impossible orderings. The schema-and-registry validation kills the structural cases, but it cannot catch a plan that is valid on paper and wrong in intent, picking the right kind of object but the wrong instance. Validation is necessary, not sufficient, and a human-meaningful sanity check on high-stakes plans is still worth having.

Latency unsuitable for reactive control. A VLM or LLM call on the order of hundreds of milliseconds is fine for planning and fatal for control. Every time a team lets a model into the fast loop, the robot becomes hesitant or unstable. The two-loop split is not optional architecture, it is the load-bearing constraint, and violating it is the most common self-inflicted wound.

Grounding failures on ambiguous scenes. A VLM asked “which box is nearest the edge” when two boxes are equidistant, or asked about a partially occluded shelf, will still return a confident answer. That answer enters the planner as a fact, and the resulting plan is valid but built on a false premise. Closed questions, confidence thresholds, and re-inspection on low confidence reduce this, but ambiguity in the physical world does not fully go away.

Long-horizon drift. Over a plan of many steps, small errors compound. A slightly-off grounding fact early leads to a place that is slightly wrong, which the next step reasons from, and the error accumulates. Periodic re-grounding, re-querying the scene rather than trusting facts asserted ten steps ago, bounds the drift, at the cost of more model calls.

Safety and validation. A robot that acts on a model’s output is only as safe as the layer between the model and the motors. The schema validation is the first line, but it does not reason about collisions, force limits, or whether a physically valid action is a safe one in context. The classical safety envelope (collision checking, force monitoring, e-stop) has to remain authoritative and independent of the planner. The language model proposes; the safety layer disposes. Never wire a model’s output straight to actuation without a validated, model-independent safety check in between.

Further reading

Start with the SayCan paper [1] for the affordance-grounding idea this article is built on, and Code-as-Policies [4] for the program-synthesis alternative to data-plans. For the vision-language grounding layer, the PaliGemma [2] and InternVL [3] papers cover the open-weight VQA models that make scene grounding practical. To understand the end-to-end alternative, read the RT-2 [5] and OpenVLA [6] papers, which collapse planning and control into a single learned policy. For the classical side, the PDDL reference [7] and a survey of integrated task-and-motion planning [8] establish what formal planning guarantees and where it is brittle. The function-calling and structured-output documentation from major model providers [9][10] is the practical reference for enforcing the action schema at decode time, and the JSON Schema specification [11] defines the constraint language the schema is written in.

Written with AI assistance, editorially reviewed.

References

[1] M. Ahn et al., “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan).” https://arxiv.org/abs/2204.01691

[2] L. Beyer et al., “PaliGemma: A Versatile 3B Vision-Language Model.” https://arxiv.org/abs/2407.07726

[3] Z. Chen et al., “InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks.” https://arxiv.org/abs/2312.14238

[4] J. Liang et al., “Code as Policies: Language Model Programs for Embodied Control.” https://arxiv.org/abs/2209.07753

[5] A. Brohan et al., “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.” https://arxiv.org/abs/2307.15818

[6] M. J. Kim et al., “OpenVLA: An Open-Source Vision-Language-Action Model.” https://arxiv.org/abs/2406.09246

[7] PDDL, “Planning Domain Definition Language (Planning.wiki reference).” https://planning.wiki/ref/pddl

[8] C. R. Garrett et al., “Integrated Task and Motion Planning.” https://arxiv.org/abs/2010.01083

[9] OpenAI, “Structured Outputs.” https://platform.openai.com/docs/guides/structured-outputs

[10] Anthropic, “Tool Use (Function Calling).” https://docs.anthropic.com/en/docs/build-with-claude/tool-use

[11] JSON Schema, “JSON Schema Specification 2020-12.” https://json-schema.org/specification

[12] D. Driess et al., “PaLM-E: An Embodied Multimodal Language Model.” https://arxiv.org/abs/2303.03378