Abstract

Contrastive pretraining, popularized by CLIP and its successors, has largely been studied in the image-text pairing setting. By mid-2025, a growing body of work is extending the contrastive framework to modality combinations that go well beyond the canonical vision-language pair: audio-video, depth-RGB, point-cloud-text, biosignal-image, and cross-modal sensor fusion for robotics. These extensions surface both novel capabilities and structural challenges not present in the image-text case. This article examines where the contrastive framework generalizes cleanly, where it fractures, and what architectural choices determine success across a broader modality landscape.

Why the Contrastive Framework Generalizes

The core intuition behind contrastive learning, that aligned cross-modal pairs should be pulled together in embedding space while unaligned pairs are pushed apart, is modality-agnostic. What it requires is a pairing signal and a means of sampling negatives. Image-text benefited from massive natural co-occurrence data (alt text, captions, web crawls). For other modality pairs, this is the first design problem: pairing is frequently synthetic, weakly supervised, or expensive to label.

Audio-video contrastive learning, explored by groups at DeepMind and Meta AI, exploits temporal synchrony as a free pairing signal. A video frame and its co-occurring audio segment are a natural positive pair; frames and audio from different temporal positions in the same clip or from different clips serve as negatives. The resulting embeddings support zero-shot audio-visual retrieval and surprisingly strong sound-source localization without pixel-level supervision.

Point-cloud-text contrastive models, such as those building on PointCLIP and ULIP, address 3D shape understanding, an area where labeled datasets are small by vision-language standards. Here the challenge is that natural language descriptions of 3D objects are sparse and the semantic granularity needed for part-level queries exceeds what typical captions provide.

Architectural Considerations for Non-Standard Modalities

Image and text tokenize relatively cleanly: patches and subword tokens map naturally to transformer inputs. Other modalities are less cooperative. Biosignals (EEG, ECG, IMU) are continuous, high-frequency, and variable-length; point clouds are orderless sets; audio spectrogram representations introduce time-frequency tradeoffs that change what “similar” means in embedding space.

A recurring finding is that frozen or lightly adapted encoders borrowed from single-modality pretraining (e.g., using an ImageNet-pretrained ViT for depth maps) underperform encoders pretrained specifically on the target modality, even when the downstream task appears superficially similar. Modality-specific tokenization and local feature extraction matter before global contrastive alignment begins.

Hard Negatives and Semantic Granularity

One well-documented weakness of naive in-batch contrastive objectives is that most negatives are trivially easy, providing little gradient signal after early training. For image-text this was addressed with hard negative mining strategies (sampling semantically confusable negatives) and techniques like SigLIP’s sigmoid loss, which avoids the softmax normalization bottleneck at large batch sizes.

For richer modality pairs, semantic similarity is harder to define. An audio clip of rain and a video of a waterfall may share acoustic texture while being visually unrelated; treating them as a negative pair risks suppressing useful cross-modal structure. Hierarchical contrastive objectives, where pairs are aligned at coarse and fine granularity separately, are an active area of investigation by teams at Berkeley and CMU as of 2025.

Outlook

The core contrastive recipe is robust enough to transfer across modalities, but the practical engineering overhead grows substantially with distance from image-text. Pairing strategy, negative sampling, tokenization, and encoder initialization each require modality-specific attention. The field would benefit from unified benchmarks that allow fair comparison of cross-modal encoders across more than the canonical retrieval and zero-shot classification tasks.