Large vision–language models (LVLMs) are inherently opaque, making it difficult to determine whether their outputs are grounded in visual evidence or driven by language-model priors. Existing concept-based interpretability methods are limited to classification settings, rely on proxy models, or require predefined tokens, and thus do not extend to LVLMs.
We propose Text-Guided Concept Learning (TGCL), a weakly supervised framework for extracting multimodal concept vectors in LVLMs without token supervision. TGCL builds concept-to-image mappings from data and extracts patch-level activations via concept-guided probing. It then formulates concept learning as a contrastive disentanglement problem, isolating concept-specific patches from background patches to produce sparse, stable, and semantically aligned concept vectors.
We conduct experiments on four datasets (ImageNet, MSCOCO, CIFAR-100, DTD) with three recent LVLMs. Across ImageNet and MSCOCO, TGCL consistently outperforms state-of-the-art baselines, with gains of up to:
TGCL moves concept extraction from token-level representations to patch-level activations guided by textual concepts. Concept learning is a linear contrastive disentanglement problem: keeping the transformation linear keeps concept vectors in the model's own activation space, rather than explaining one black box with another.
The LVLM lists the concepts in each image. For each concept, crops (random or SAM proposals) from supporting images form a bag that mixes concept-present and concept-absent patches — no labels needed.
Each crop gets the binary prompt above. Residuals are averaged over the shared concept-token span only (dropping “No”), so the contrast comes from the image, not the token identity.
Per concept, the activations S are factorized as S ≈ VU⊤ with just two bases (concept vs. no-concept). K small independent dictionaries avoid atom interference and scale to large vocabularies.
The positive basis is picked via maximum-activated crops. Each vector is stored with visual (top patches) and textual (output-head tokens) groundings; predictions are attributed by cosine similarity to residual activations.
Gemma-3n-E4B on ImageNet and MSCOCO. Higher sparsity, lower instability and overlap are better.
| Method | Decomp. | ImageNet | MSCOCO | ||||
|---|---|---|---|---|---|---|---|
| Sparsity ↑ | Instability ↓ | Overlap ↓ | Sparsity ↑ | Instability ↓ | Overlap ↓ | ||
| CRAFT | SNMF | 0.42 | 0.02 | 0.69 | 0.42 | 0.00 | 0.81 |
| SAE | 0.39 | 0.55 | 0.02 | 0.37 | 0.81 | 0.04 | |
| SIMPLE | 0.42 | 0.00 | 0.52 | 0.42 | 0.33 | 0.59 | |
| MCD | SNMF | 0.42 | 0.02 | 0.70 | 0.42 | 0.00 | 0.78 |
| SAE | 0.39 | 0.67 | 0.02 | 0.37 | 0.78 | 0.05 | |
| SIMPLE | 0.42 | 0.00 | 0.67 | 0.43 | 0.23 | 0.45 | |
| CoX-LMM | SNMF | 0.43 | 0.06 | 0.27 | 0.43 | 0.00 | 0.32 |
| SAE | 0.42 | 0.80 | 0.02 | 0.38 | 0.86 | 0.03 | |
| SIMPLE | 0.44 | 0.00 | 0.34 | 0.43 | 0.09 | 0.36 | |
| TGCL (ours) | SNMF | 0.61 | 0.00 | 0.01 | 0.63 | 0.01 | 0.02 |
| SAE | 0.36 | 0.96 | 0.78 | 0.42 | 1.00 | 0.04 | |
| SIMPLE | 0.45 | 0.00 | 0.53 | 0.44 | 0.01 | 0.68 | |
Best values in bold, as in the paper. TGCL with SNMF gives the highest sparsity and the lowest instability and overlap.
Gemma-3n-E4B. In the combined setting the model labels each cell of a 2×2 grid of unrelated images, and attribution is scored against those predictions.
| Method | Dataset | Metric | Random | Text-only | Image-only | Combined |
|---|---|---|---|---|---|---|
| TGCL (ours) | ImageNet | CLIPScore | 0.52 | – | 0.62 | 0.67 |
| BERTScore | 0.71 | 0.78 | 0.84 | 0.86 | ||
| MSCOCO | CLIPScore | 0.48 | – | 0.60 | 0.64 | |
| BERTScore | 0.71 | 0.89 | 0.91 | 0.93 | ||
| CoX-LMM | ImageNet | CLIPScore | 0.49 | – | 0.57 | 0.58 |
| BERTScore | 0.71 | 0.74 | 0.75 | 0.82 | ||
| MSCOCO | CLIPScore | 0.51 | – | 0.57 | 0.55 | |
| BERTScore | 0.71 | 0.83 | 0.79 | 0.73 |
Means shown; standard deviations, DTD, CIFAR-100, Qwen2-VL-7B and Qwen2.5-VL-7B are in the paper.


| Method | Ranked order | Random order | Rank spread ↑ | |||||
|---|---|---|---|---|---|---|---|---|
| Ins ↑ | Del ↓ | Gap ↑ | Ins | Del | Gap | |||
| CRAFT | 0.781 | 0.217 | 0.564 | 0.510 | 0.491 | 0.019 | 0.016 | |
| MCD | 0.793 | 0.185 | 0.608 | 0.512 | 0.489 | 0.023 | 0.016 | |
| CoX-LMM | 0.771 | 0.194 | 0.577 | 0.512 | 0.488 | 0.024 | 0.016 | |
| TGCL (ours) | random crops | 0.815 | 0.187 | 0.628 | 0.517 | 0.489 | 0.028 | 0.086 |
| TGCL (ours) | SAM crops | 0.826 | 0.176 | 0.650 | 0.516 | 0.479 | 0.037 | 0.108 |
Concept insertion/deletion AUC on MSCOCO-10 with Gemma-3n-E4B, mean over 500 images; Gap = Ins − Del. TGCL (SAM) is better than every baseline with p < 0.001 (Welch's t-test, Holm-corrected). Standard deviations and effect sizes are in the paper.
Each prediction on a 2×2 grid, with its two most attributed concepts: attribution score, top activating patches, and decoded tokens.





@inproceedings{kadir2026tgcl,
title = {Multimodal Concepts for Interpreting Large
Vision--Language Models},
author = {Kadir, Md Abdul and Bhatti, Omair Shahzad
and Sonntag, Daniel},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}