Abstract

Vision-language models (VLMs) such as GPT-4o, Gemini, and LLaVA-family models can describe images fluently and answer questions about visual content. Yet systematic evaluations continue to reveal a consistent failure mode: models produce plausible-sounding language that is not actually anchored to what is in the image. This “grounding failure” manifests as hallucinated objects, wrong spatial relationships, and overconfident attribute claims. Understanding where grounding breaks down, and what training choices affect it, has become one of the more active areas of VLM evaluation research heading into 2026.

What Grounding Failures Look Like

A typical grounding failure is not a random hallucination. Models tend to hallucinate objects that are statistically likely to co-occur with what is actually in the image, a pattern consistent with the language model component over-weighting prior knowledge about the visual context rather than the actual visual tokens. POPE and CHAIR benchmarks quantify this, and both have shown that even state-of-the-art models hallucinate at non-trivial rates on common objects in COCO. Spatial relationship errors are a separate category: models frequently confuse left/right, above/below, and relative size when objects are at similar scales or in unusual orientations. These are not fixable by prompting alone because the errors originate in the visual encoder’s representation of spatial layout.

Training Dynamics that Affect Grounding

The quality of grounding correlates with several training choices. Higher resolution visual encoders, such as the tile-based encoding used in LLaVA-HR and the dynamic resolution schemes in InternVL, reduce object-level hallucination by giving the model finer-grained visual tokens to attend to. The ratio of image-grounded text data to general text in the pretraining and instruction tuning mix matters considerably: models trained on large proportions of pure text data tend to exhibit stronger language priors that override visual evidence. DPO-style fine-tuning on preference data that specifically penalizes hallucinated object mentions, as in RLHF-V and Silkie from Shanghai AI Lab, has demonstrated 10-20 point improvements on POPE without sacrificing general capability.

Calibration and Practical Mitigation

Grounding failures are partly a calibration problem: a well-calibrated VLM should refuse or hedge when its visual encoder has low confidence in a region, rather than filling the gap with a statistically likely answer. Current models are generally over-confident. Practical mitigations available today include chain-of-thought prompting that asks the model to locate an object before describing it, as this forces the model to anchor its language output to a spatial claim that can be self-checked, and ensemble verification approaches where two model runs on the same image are compared for consistency. Neither is a complete fix. The core issue, a visual encoder that produces insufficient signal for fine-grained grounding, will require architectural and data-pipeline changes that are still works in progress across most leading labs.