Sparse cross-modal routing for efficient knowledge-intensive multimodal question answering — matching or exceeding dense baselines with up to 6.8× fewer FLOPs and 2.7× less latency.
Knowledge-intensive multimodal question answering (KI-MMQA) sits at the intersection of three expensive primitives: long visual token sequences, dense retrieval over large external corpora, and full cross-modal fusion. Existing systems pay all three costs uniformly per query, even though only a small fraction of visual content and retrieved knowledge is actually relevant to any given question.
We introduce SKIP (Salient Knowledge-Injected Pathways), a unified inference architecture that routes computation along sparse pathways jointly conditioned on the question, the image, and a difficulty estimate. SKIP combines question-guided visual token pruning, region-conditional sparse retrieval, bipartite sparse cross-attention, and speculative knowledge verification with an adaptive budget controller that allocates compute proportional to predicted question difficulty.
We derive an information-bottleneck bound showing that the optimal visual sparsity rate scales as O(1/√N) under realistic question-image mutual-information assumptions, with retained accuracy guarantees. Across five KI-MMQA benchmarks, SKIP matches or exceeds the accuracy of strong dense baselines while using 3.4–6.8× fewer FLOPs and 2.7× less end-to-end latency.
SKIP replaces the dense KI-MMQA pipeline with five components arranged so that each stage's sparsity decision conditions the next. The key commitment: sparsity is applied compositionally—each stage's output is the sparse input to the next, not independently sparsified.
| Model | OK-VQA | A-OKVQA | InfoSeek | Enc-VQA | ViQuAE | TFLOPs | Lat. (ms) |
|---|---|---|---|---|---|---|---|
| BLIP-2 no retrieval | 45.9 | 39.7 | 11.2 | 8.4 | 19.1 | 0.42 | 210 |
| LLaVA-1.5 no retrieval | 50.3 | 44.1 | 13.7 | 10.2 | 22.3 | 0.51 | 240 |
| KAT | 54.4 | 48.2 | 18.6 | 14.7 | 28.9 | 0.93 | 410 |
| RAVQA-2 | 56.8 | 50.6 | 21.3 | 16.4 | 31.7 | 1.18 | 520 |
| ReVeaL | 58.0 | 52.4 | 23.8 | 18.1 | 33.2 | 1.34 | 610 |
| RA-CM3 | 60.5 | 55.3 | 26.4 | 20.9 | 35.6 | 1.71 | 820 |
| SKIP ours | 63.2 | 58.4 | 30.7 | 24.3 | 37.4 | 0.42 | 305 |
| vs. RA-CM3 | +2.7 | +3.1 | +4.3 | +3.4 | +1.8 | 4.07× less | 2.69× faster |
No individual technique recovers the full system effect. The gains compound: each sparse component filters a different source of noise, and their composition reaches 63.2% that no single component approaches.
| Component | Dense (GFLOPs) | SKIP (GFLOPs) | Reduction |
|---|---|---|---|
| Vision encoder | 72.4 | 72.4 | 1.0× |
| QVS scoring | — | 0.5 | +0.3% |
| RCSR retrieval | 48.2 | 19.3 | 2.5× |
| SKV gate | — | 0.8 | negligible |
| DBC controller | — | 0.3 | negligible |
| BSCA (ρᵥL × Δ̄) | 143.7 | 14.6 | 9.8× |
| LLM decoder | 248.4 | 76.4 | 3.3× |
| Total | 512.7 | 184.3 | 2.8× |
We formalize the intuition that mutual information between visual content and answers concentrates on a small set of salient regions, so the optimal retention rate should be sublinear in V.
Under two mild assumptions — piecewise-Lipschitz mutual information and bounded QVS saliency error η ≤ 0.013 — we prove a closed-form bound. At V = 576 (24×24 patch grid), the bound predicts V′ ≈ 51 retained tokens; the empirical optimum is V′ = 64, consistent within a small constant.
An analogous bound applies to retrieval sparsity: K′ ≥ C′√K log(1/ε), with K′ = 4–8 sufficing in practice at K = 16, ε = 0.05.
Sequential staging avoids gradient conflicts between the discrete pruning signal and the downstream cross-attention loss. Without it, DBC requests maximum budget and SKV never fast-paths.
If SKIP is useful in your research, please consider citing our ICML 2026 EMM-QA workshop paper.