GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations

1DEEPNOID Inc., Seoul, South Korea  ·  2Sejong University, Seoul, South Korea

*Equal contribution   ‡Equal second-author contribution   †Corresponding author

NeurIPS 2026

Free-text queries highlighting findings on chest X-rays and CT volumes with sparse gate maps

A radiology report supervises the whole image, yet each finding occupies only a small region. GLINT pairs a sparse gate that aligns each sentence with the patches it describes with pretrained SSL encoders kept fine-grained — enabling zero-shot classification, grounding, and segmentation on chest X-ray and 3D CT, and a powerful standalone vision encoder.

TL;DR

Radiology reports supervise the whole image, yet each finding occupies only a small region. GLINT models this sparse correspondence explicitly: Sparsely Gated Alignment lets a sigmoid gate in a separate gate embedding space select the patches relevant to each sentence, and Dense Feature Regularization anchors the encoder's intermediate features to a frozen SSL teacher so those patch features stay fine-grained. Built with DINOv3 (CXR) and V-JEPA 2.1 (CT), GLINT delivers zero-shot classification, grounding and segmentation from free text, with the largest gains exactly where sparse, query-specific localization is required. The GLINT vision encoder is also a strong standalone backbone: fine-tuned downstream, it outperforms both SSL encoders and medical VLMs on classification, report generation and segmentation.

Motivation

Image-report pairs are the natural supervision for radiology vision-language models, but the pairing hides a mismatch in scale: a sentence like "nodule in the left lower lung" describes a few patches, while the loss is applied at the global image-report level. Prior approaches spread alignment weight densely across all patches, which blurs localization and caps zero-shot grounding and segmentation.

GLINT overview: sparse image-report correspondence, sigmoid-gated alignment versus a noisy dense baseline, and the GLINT-CXR / GLINT-CT family
Each sentence describes a small region of the image. GLINT's sigmoid gate produces sparse, query-specific patch scores where dense baselines stay noisy, and the same framework yields GLINT-CXR and GLINT-CT.

Method at a glance

SGA — Sparsely Gated Alignment

Patch and text features are projected into a separate gate embedding space where a sigmoid gate activates only the patches relevant to each query. A sparsemax-pooled gate confidence shift ℓ = ρ + ω·e restores the absolute activation signal, so absent and present findings stay separable without breaking sparsity.

DFR — Dense Feature Regularization

Token-wise cosine distillation anchors intermediate student features (layer 22) to a frozen SSL teacher — DINOv31 ViT-7B (layer 36) for CXR, V-JEPA 2.12 ViT-G (layer 44) for CT — preserving the fine-grained patch features the gate relies on.

GLINT architecture: sparsely gated alignment head over patch tokens plus dense feature regularization from a frozen teacher
The full training loss is ℒ = ℒalign + λ·ℒreg (λ = 8), trained end-to-end on image-report pairs only — no masks, boxes, or region labels.

Zero-shot results

Chest X-ray zero-shot Classification: Macro AUROC  ·  Grounding: Macro Acc. / Acc.  ·  Segmentation: Macro Dice / Dice  ·  best in bold
MethodClassificationGroundingSegmentation
PadChestPadChest20ChestXray14ChestX-Det10CheXpertOpen-IChestX-Det10MS-CXRChestX-DetSIIM
GLoRIA30.5650.5580.6100.6450.7500.5890.367---
KAD40.7500.7350.7890.7350.9050.8070.391---
MedKLIP50.6290.6880.7260.7130.8790.7590.4810.4070.2490.044
CARZero60.8100.8370.8110.7960.9230.8380.5430.7490.3040.081
MedSigLIP70.7360.7470.7420.7020.8590.763----
RadZero80.8410.8710.8040.7870.9000.8470.6220.8440.3320.171
GLINT-CXR0.8630.9170.8250.8530.9210.8810.7290.8920.4340.317
Chest CT zero-shot classification Macro AUROC  ·  best in bold
MethodCT-RATERAD-ChestCT
CT-CLIP90.7310.629
Merlin10†0.7280.644
fVLM11†0.7780.680
ViSD-Boost12†0.7900.694
RadZero3D130.7620.690
COLIPRI140.7850.730
GLINT-CT0.8500.789
Free-text segmentation on ReXGroundingCT Dice  ·  supervised methods shown for reference
MethodSettingDice
SAT15‡Supervised0.131
VoxTell16‡Supervised0.282
GLINT-CTZero-shot0.128

Chest X-ray baseline rows are quoted from RadZero (MedSigLIP evaluated by us with its released weights). CT-CLIP and RadZero3D rows are quoted from the RadZero3D paper and COLIPRI (CRM, short prompts) from the COLIPRI paper; † reported by ViSD-Boost under a protocol excluding two labels; under that protocol GLINT-CT reaches 0.855 / 0.794. ‡ reported by VoxTell. GLINT also leads downstream fine-tuning — classification, report generation and segmentation on both modalities; full tables in the paper.

Models & Code

Model · Chest X-ray

Deepnoid/GLINT-CXR

DINOv3 ViT-L/16 + MPNet, pretrained on MIMIC-CXR.

Model · Chest CT

Deepnoid/GLINT-CT

V-JEPA 2.1 ViT-L/16 + MPNet, pretrained on CT-RATE.

Code

deepnoid-ai/GLINT

Pretraining, zero-shot evaluation, downstream fine-tuning.

References (16)
  1. Siméoni et al. DINOv3. arXiv, 2025.
  2. Mur-Labadia et al. V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning. arXiv, 2026.
  3. Huang et al. GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-Efficient Medical Image Recognition. ICCV, 2021.
  4. Zhang et al. Knowledge-enhanced visual-language pre-training on chest radiology images. Nature Communications, 2023.
  5. Wu et al. MedKLIP: Medical knowledge enhanced language-image pre-training for X-ray diagnosis. ICCV, 2023.
  6. Lai et al. CARZero: Cross-attention alignment for radiology zero-shot classification. CVPR, 2024.
  7. Sellergren et al. MedGemma technical report. arXiv, 2025.
  8. Park et al. RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability. NeurIPS, 2025.
  9. Hamamci et al. Generalist foundation models from a multimodal dataset for 3D computed tomography. Nature Biomedical Engineering, 2026.
  10. Blankemeier et al. Merlin: a computed tomography vision-language foundation model and dataset. Nature, 2026.
  11. Shui et al. Large-scale and fine-grained vision-language pre-training for enhanced CT image understanding. ICLR, 2025.
  12. Cao et al. Boosting vision semantic density with anatomy normality modeling for medical vision-language pre-training. ICCV, 2025.
  13. Park et al. RadZero3D: Bridging Self-Supervised Video Models and Medical Vision-Language Alignment for Zero-Shot Chest CT Interpretation. ICCV, 2025.
  14. Wald et al. Comprehensive language-image pre-training for 3D medical image understanding. arXiv, 2025.
  15. Zhao et al. Large-vocabulary segmentation for medical images with text prompts. npj Digital Medicine, 2025.
  16. Rokuss et al. VoxTell: Free-Text Promptable Universal 3D Medical Image Segmentation. arXiv, 2025.

BibTeX

If you use GLINT, please cite:

@inproceedings{park2026glint,
  title     = {{GLINT}: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations},
  author    = {Park, Jonggwon and Lee, Seongeun and Park, Junhyun and
               Yun, Hannah and Kim, Hyunwoong and Jeong, Sohyun and
               Kang, Hyewon and Yoon, Byungmu and Choi, Kyoyun},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.03180}
}