I am excited to share our recently published work, “Adaptive Gated Mamba for Multi-View Chest X-Ray Report Generation”, developed together with Caner Köse, Saifullah Tumrani, Radu Timofte, and Fayaz Ali Dharejo.

In this work, we explore how automatic chest X-ray report generation can be improved by controlling how visual information flows through a Mamba-based vision encoder and by learning how information from multiple radiographic views should be combined.

Why Chest X-Ray Report Generation?

Chest radiography is one of the most widely used medical imaging techniques for assessing cardiopulmonary abnormalities. However, interpreting X-rays and preparing detailed radiology reports can be time-consuming.

Automatic report generation aims to assist this workflow by producing structured draft reports from chest X-ray images. A major challenge, however, is that clinically relevant findings can be subtle and may appear differently across different projections, such as frontal and lateral views.

The Starting Point: MambaXray-VL

Our work builds on MambaXray-VL, a vision-language framework that uses a Mamba-based visual encoder together with image-text alignment and a language model for radiology report generation.

We identified two areas where the baseline could be improved:

• Mamba updates are added directly to the residual representation, without an explicit mechanism controlling how strongly each contextual feature should contribute.
• Multiple X-ray views are combined using simple mean pooling, which gives every view and token equal importance.

To address these limitations, we introduced two complementary architectural components.

1. Identity-Initialized Adaptive Mamba Gating

Our first contribution is an identity-initialized adaptive gating mechanism inside the Mamba visual encoder.

Instead of directly adding every Mamba update to the residual stream, the adaptive gate learns a token- and feature-wise modulation that determines how strongly contextual information should modify the current representation.

An important part of the design is its initialization. The gate starts as an identity transformation, meaning that at the beginning of fine-tuning the modified model behaves like the original pretrained network.

The model can then gradually learn where stronger or weaker feature updates are useful without immediately disrupting the pretrained representation.

2. Perceiver-Inspired Multi-View Fusion

The second contribution focuses on how frontal and lateral chest X-rays are combined.

The original baseline averages the visual representations of the available views. While simple and efficient, mean pooling assumes that every view and every corresponding token contributes equally.

We replace this fixed aggregation with a Perceiver-inspired latent cross-attention module. Learnable latent queries attend to visual patch tokens from each available X-ray view and selectively integrate their information.

After the views are incorporated, the latent representation is further refined using self-attention and a feed-forward network before being passed to the language model.

This allows the network to learn which visual information is most relevant rather than relying on uniform averaging.

Combining Both Components

The two proposed components operate at different levels of the model.

• Adaptive gating controls the information flow within each encoded X-ray view.
• Latent fusion controls how complementary information from multiple views is combined.

This made them particularly interesting to study together: one improves intra-view representation, while the other improves cross-view integration.

Experimental Setup

We evaluated our approach on the IU X-Ray dataset, using the same experimental setting as the reproduced MambaXray-VL baseline.

We compared four configurations:

• MambaXray-VL baseline with mean fusion
• Adaptive gating only
• Latent multi-view fusion only
• Adaptive gating + latent fusion

Report quality was evaluated using standard natural-language generation metrics including BLEU-1 to BLEU-4, ROUGE-L, and CIDEr.

Results

The strongest performance was achieved when adaptive gating and latent multi-view fusion were combined.

Compared with the reproduced MambaXray-VL baseline, the combined model improved the reported metrics from:

• BLEU-1: 0.479 → 0.502
• BLEU-2: 0.322 → 0.340
• BLEU-3: 0.236 → 0.249
• BLEU-4: 0.179 → 0.190
• CIDEr: 0.508 → 0.532
• ROUGE-L: remained at 0.388

The combined architecture achieved the best reported BLEU-1, BLEU-2, BLEU-3, and BLEU-4 results in our comparison on IU X-Ray, demonstrating that the two components provide complementary improvements.

What the Ablation Study Showed

One of the most interesting observations came from evaluating the components independently.

Adaptive gating improved all BLEU metrics while preserving the baseline ROUGE-L score, whereas the latent fusion module provided a more balanced improvement in multi-view representation quality.

The best overall performance appeared when both components were used together, supporting the idea that improving information flow inside each view and improving information exchange between views are complementary problems.

Beyond Text Generation Metrics

An important part of the study was also understanding the limitations of evaluating medical report generation purely with lexical metrics.

In one qualitative example, the generated report correctly described the main cardiac and pulmonary findings but additionally mentioned degenerative changes of the thoracic spine that were not present in the reference report.

This demonstrates an important limitation of medical language generation systems: a report can be fluent and achieve strong text-generation scores while still introducing an unsupported clinical statement.

For this reason, future work should complement metrics such as BLEU, ROUGE, and CIDEr with more clinically oriented evaluation methods that explicitly measure false-positive and false-negative findings.

Key Takeaway

Our results suggest that two relatively targeted architectural changes can improve multi-view chest X-ray report generation without modifying the complete pretraining pipeline.

By combining adaptive feature-level gating with learnable cross-view fusion, the model can better control both how visual representations are updated and how complementary information from multiple radiographic views is integrated before report generation.

This project also gave us the opportunity to explore Mamba-based vision models, multimodal learning, cross-attention, medical image analysis, and automatic radiology report generation within a single research framework.

Read the full paper on OpenReview