Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning, but the encoders often underperform in domain-specific visual tasks such as medical image diagnosis or fine-grained classification, where representation errors can cascade through the language model, leading to incorrect responses. Existing adaptation methods modify the continuous feature interface between encoder and language model through projector tuning or other parameter-efficient updates, which still couples the two components and requires re-alignment whenever the encoder changes. We introduce CRAFT (Codebook RegulAted Fine-Tuning), a lightweight method that fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space, achieving domain adaptation without modifying other parts of the model. This decoupled design allows the adapted encoder to seamlessly boost the performance of LVLMs with different language architectures, as long as they share the same codebook. Empirically, CRAFT achieves an average gain of 13.51% across 10 domain-specific benchmarks such as VQARAD and PlantVillage, while preserving the LLM's linguistic capabilities and outperforming peer methods that operate on continuous tokens.
One-Sentence Summary
Introduces CRAFT, a discrete-codebook method for adapting vision encoders in LVLMs to domain-specific tasks without modifying the language model, enabling portable encoder adaptation across architectures.
Key Contributions
CRAFT: codebook-regulated fine-tuning that decouples vision encoder adaptation from the language model
Discrete codebook anchoring for stable, portable visual representations
13.51% average gain across 10 domain-specific benchmarks while preserving linguistic capabilities
Cite This Paper If...
You are working on vision-language adaptation, codebook-anchored visual adaptation, or decoupling visual and language representations. This paper is relevant if you need to adapt a vision encoder for domain-specific tasks without retraining the language model, or if you want portable encoder fine-tuning that transfers across different LLM backbones.
Frequently Asked Questions
What is CRAFT and how does it differ from projector tuning?
CRAFT (Codebook RegulAted Fine-Tuning) fine-tunes the vision encoder using a discrete codebook that maps visual features to a stable token space. Unlike projector tuning, which couples the encoder and language model through continuous features, CRAFT's discrete interface means the encoder can be swapped or updated independently.
Why use a discrete codebook instead of continuous features?
Continuous features tightly couple the encoder and language model, requiring re-alignment whenever either component changes. A discrete codebook provides a stable interface: as long as both components share the same codebook vocabulary, the adapted encoder works with any compatible language model.
What domains does CRAFT improve on?
CRAFT is evaluated on 10 domain-specific benchmarks including medical imaging (VQARAD), agriculture (PlantVillage), and other fine-grained visual tasks, achieving an average gain of 13.51% while preserving the LLM's general linguistic capabilities.
Can CRAFT be applied to different LVLM architectures?
Yes. The decoupled design is a key advantage: an encoder fine-tuned with CRAFT can boost performance across LVLMs with different language architectures, as long as they share the same codebook. This portability is unique among vision-language adaptation methods.
Does CRAFT hurt the language model's general capabilities?
No. Because CRAFT only modifies the vision encoder and leaves the language model untouched, the LLM's linguistic capabilities are fully preserved. This is confirmed empirically across the benchmark suite.
Keywords
codebook anchored visual adaptationdecoupling vision and languagevision-language model adaptationvisual adaptation codebookefficient vision-language adaptation
@article{wu2026decoupling,
title={Decoupling Vision and Language: Codebook Anchored Visual Adaptation},
author={Wu, Jason and Zhao, Tianchen and Liu, Chang and Cai, Jiarui and Zhang, Zheng and Li, Zhuowei and Singh, Aaditya and Xu, Xiang and Srivastava, Mani and Wu, Jonathan},
journal={arXiv preprint arXiv:2602.19449},
year={2026}
}
Recommended Citation Contexts
Decoupling visual and language branches during VLM adaptation, using discrete codebook anchors, preserves cross-modal alignment while enabling task-specific visual adaptation (Zhao et al., 2026).