CompRank: Efficient LLM Reranking via Token-Level Compression and Decoding-Free Scoring

Paper: CompRank: Efficient LLM Reranking via Token-Level Compression and Decoding-Free Scoring (v1) https://arxiv.org/html/2606.11700v1

CompRank builds a reranking pipeline from three distinct operations. It forms document representations independently, selects token-level key-value states for query access, and sorts documents by attention-derived scores. No ranking answer needs to be decoded.

Independent document blocks use local positions and prevent cross-document attention. A candidate’s representation therefore does not depend on jointly encoding all the other documents in the list. The original animation keeps those complete blocks visible from the start.

Token compression changes the states that query attention can access. It selects retained KV states without changing the original document encodings. A toy sparse selection illustrates this access boundary; the displayed positions are authored examples rather than measured token choices for a benchmark query.

Query attention is then aggregated over retained document tokens into scores. In the teaching example, fixed illustrative masses of 0.28, 0.22 and 0.50 sort Documents C, A and B while retaining their identities. CopyNet-style candidate supervision trains the method’s inference-time attention scoring directly.

The reported Mistral-7B training uses 30-document lists. Evaluation reranks the same BM25 top-100 candidates across seven BEIR datasets. Step-10 retains 10.2% of document tokens and reports an average 39.2 NDCG@10, compared with full-token CompRank’s 39.7.

The latency table evaluates a separate 500-candidate TREC-COVID case. Direct-List takes 114.5 seconds, full-token CompRank 16.22 and Step-10 12.0 in the measured end-to-end setup. Documents are recomputed there; an optimized reusable KV cache remains a design opportunity. The companion film carries that qualification alongside the scores and uses original vectors to explain the mechanism without claiming an actual toy-document model run.

Pinned source versions: Paper, v1: https://arxiv.org/html/2606.11700v1

#InformationRetrieval #Reranking #AIResearch