Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions

Ning E^{1,2 *} Changde Du^{2,5 *} Yizhuo Lu^{2,3 *} Huiguang He^{2,3,4,1,5 †}

¹School of Biomedical Engineering, ShanghaiTech University

²Institute of Automation, Chinese Academy of Sciences

³School of Future Technology, University of Chinese Academy of Sciences

⁴School of Artificial Intelligence, University of Chinese Academy of Sciences

⁵Zhongguancun Academy

arXiv Poster Code

CogAlign aligns MLLMs with the latent structure of human cognition by moving beyond outcomes to underlying cognitive processes through linguistic rationales grounded in behavior-derived semantic dimensions, improving behavioral agreement and neural representational alignment.

Scroll for overview

Overview

Human Decision

Start with human similarity judgments

Build on THINGS: 4.70 million odd-one-out judgments spanning 1,854 object concepts, plus a 66-dimensional interpretable embedding (Hebart et al., 2020, Nat Hum Behav).

Cognitive Dimension

Represent objects along 66 dimensions

Adopt an established SPoSE embedding to capture the most explainable variance in similarity judgements and produce multidimensional highly reproducible and meaningful object dimensions that represent various conceptual and perceptual properties of those objects.

Triplet-Level Attribution

Infer the decision dimension

For each triplet, assess the relevance of each semantic dimension by individually pruning it from the embedding space and measuring the change in softmax probabilities. The dimension inducing the maximum change is identified as the most salient semantic dimension.

Reasoning Enhancement

Turn dimension into rationale

Given textual description and most salient semantic dimension, use an LLM to generate chain-of-thought rationale that elucidates how this dimension drives human reasoning and decision-making.

General Capability trade-off

Maintain general performance

CogAlign consistently improves MMMU_val by 3.12 - 4.34, while yielding declines of 0.05 - 3.58 on MMBench_{en, test}, suggesting benchmark-dependent trade-off in general multimodal capabilities rather than uniform capability degradation.

Better Behavior

Improve human consistency

On 453,642 held-out THINGS triplets, CogAlign improves human consistency over both Direct and CoT baselines across Qwen3-VL model scales (4B to 30B-A3B), maintaining stable accuracy range of 56.21 - 56.99.

Neuroscientific Analysis

Toward representational alignment

Searchlight RSA comparing MLLMs internal representation with human brain, revealing improved geometry alignment across four subjects on the independent NSD fMRI dataset.

Natural objects pairwise similarity

Choose the one that is the least similar to the other two.

Behavior-derived semantic embedding

Illustrative dimensions from the behavior-derived embedding, such as animal-related, food-related, and metallic or artificial.

Jackknife-based attribution method

d^∗ = argmax_D |p_full − p_without-d|.

Pipeline for reasoning enhancement

Input: triplet description and semantic dimension. Output: enhanced reasoning.

MMMU and MMBench benchmarks

Qwen3-VL Instruct models across 4B and 8B scales.

Benchmark

Scale

MMMU

Baseline

CogAlign

Human consistency

Qwen3-VL Instruct models across 4B, 8B, and 30B-A3B scales vs. Centaur on THINGS held-out.

Direct

CoT

CogAlign (our)

Noise Ceiling

Centaur 51.96 70B

Noise Ceiling 68.74 human

Robustness 56 ~ 57 CogAlign

Cortical projection visualization

Darker colors indicate better CogAlign alignment with human brain over Baseline.

Beyond Prompting

Explicit semantic guidance falls short.

Human Consistency

Qwen3-VL 4B Instruct · Prompting vs. CogAlign

S-CoT 53.37

d^*-CoT 49.01

FS-CoT 52.70

CogAlign 56.81

+ 3.44 pp over standard CoT.

On Qwen3-VL 4B, both the most salient semantic dimension CoT (d*-CoT) and full-space CoT (FS-CoT) underperform standard CoT (S-CoT), achieving accuracies of 49.01 and 52.70, respectively, compared to 53.37 for S-CoT. CogAlign achieves 56.81, outperforming all three prompting strategies and demonstrating that semantic dimension guidance alone is insufficient.

Supervision Signals Impact

Supervision signals from dimension-grounded rationales matter.

Baseline: CoT generation. Supervision Signals: decision option (Choice-only), dimension labels (CogDim), and dimension-grounded rationales (CogAlign).

Across MLLM Families

Gains across Qwen2.5-VL, Qwen3-VL, and Gemma3 instruction models.

Across five models spanning Qwen2.5-VL, Qwen3-VL, and Gemma3, CogAlign improves human consistency by 0.8 - 4.7 pp over CoT baselines on 453,642 held-out THINGS triplets. On Qwen2.5-VL-7B, behavioral representational similarity improves from 73.3 to 80.9, indicating that CogAlign enhances both agreement and alignment with human representational geometry.

Takeaways

Beyond behavioral mimicry. Convert behavior-derived semantic dimensions into interpretable linguistic rationales, guiding MLLMs to move beyond outcome imitation toward human cognitive processes.

Gains from CogAlign. Improve behavioral consistency and alignment of representational geometry.

Toward cognitive pathway. Introduce a general, data-efficient framework for injecting population-level cognitive priors into MLLMs, facilitating more controllable, consensus-consistent models.

Citation

@misc{e2026aligningmllmslatentstructure,
  title         = {Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions},
  author        = {Ning E and Changde Du and Yizhuo Lu and Huiguang He},
  year          = {2026},
  eprint        = {2610.41951},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.41951}
}

← BackResearch Next →Home