VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval

01 · Results

On MSR-VTT, VEDJE matches the text-to-video R@1 of the LamRA reproduction with about 49 times fewer online parameters.

Sections 4.1 to 4.3 of the paper, with Figure 1, Tables 1, 3, 9, 12 and 22, and Appendices A.6, A.8 and B.

Online parameters count the text encoding and scoring that run for each query. With a fine-tuned VideoCLIP-XL first stage, VEDJE raises text-to-video R@1 from 56.2 to 59.8 with about 157M online parameters. The LamRA reproduction reaches 59.7 with a 7.6B-parameter decoder alone, about 49 times as many.

Accuracy against online parametersFigure 1 and Table 22 · MSR-VTT text-to-video R@1 · log scale

Papers: CLIP4Clip · X-CLIP · EERCF · Video-ColBERT · LamRA · CaRe-DPO

VEDJE improves R@1 over each matched first stage on MSR-VTT, in both directions.

Each comparison keeps the first stage's candidates and adds VEDJE on top, with a cache built from the corresponding backbone. The gains remain after the first stage is fine-tuned.

R@1 gain over the matched first stageTable 1b · MSR-VTT · R@1 points · longer is better

Papers: VideoCLIP-XL · VideoPrism · PE-Core-B

Text-to-video R@1 also rises on MSVD, DiDeMo and ActivityNet.

With the zero-shot VideoCLIP-XL first stage, text-to-video R@1 rises on each benchmark. The largest gain is on DiDeMo, from 47.7 to 56.6.

Text-to-video R@1 gain on four benchmarksTable 12 · zero-shot VideoCLIP-XL first stage · R@1 points

Papers: MSR-VTT · MSVD · DiDeMo · ActivityNet

Each video is stored once, in 48 KiB.

The default cache keeps four tokens per sampled frame and is 128.5 times smaller than the frame-and-patch features of the same backbone. In the VideoPrism configuration on MSR-VTT, one token per frame shrinks the cache fourfold to 12 KiB, and text-to-video R@1 stays within 0.2 points.

Visual storage per videoTables 3 and 9 · VideoPrism features · log scale

02 · How it works

VEDJE moves video encoding offline and keeps a separate token group for each sampled frame.

Section 3 of the paper, with Figure 2 and Appendices A.1 to A.3.

Finding the right video often means telling apart similar scenes in which different events occur. Joint matching of the query and the video improves retrieval, but processing rich video features for every query is costly. VEDJE encodes each video once, compresses each sampled frame into a few tokens and lets a small joint encoder read them at query time.

The VEDJE pipeline, step by step.

Figure 2 of the paper (VEDJE pipeline), animated with illustrative values.

The three steps in equations.

A

Offline indexing

Section 3.2

U_t = X_t W_{\mathrm{kv}},

Z_t = C_\eta(Q_0;\, U_t) \in \mathbb{R}^{M \times d_\ell}

Z(v) = [Z_1;\, \dots;\, Z_T] \in \mathbb{R}^{TM \times d_\ell},

B_{\mathrm{cache}} = 2\,T M d_\ell \ \text{bytes}

$X_t$ holds the frozen encoder's frame-indexed patch features at sampled frame $t$ , and $W_{\mathrm{kv}}$ projects them to the token width $d_\ell$ . The compressor $C_\eta$ reads them with $M$ learned queries $Q_0$ , shared by all frames. The cache holds $TM$ tokens whatever the patch count. Four tokens per frame take 48 KiB per video, and one token per frame takes 12 KiB.

B

Training only

Sections 3.3 and 3.5

\widehat{\Delta}_t = F_\omega(Z_t),

\Delta_{t,h} = X_{t+h} - X_t

\mathcal{L}_{\mathrm{delta}} = \frac{1}{|\Omega|} \sum_{(t,h) \in \Omega} \big\| \widehat{\Delta}_{t,h} - \Delta_{t,h} \big\|_2^2

\mathcal{L} = \mathcal{L}_{\mathrm{vtm}} + \mathcal{L}_{\mathrm{vtc}} + \mathcal{L}_{\mathrm{mlm}} + \mathcal{L}_{\mathrm{delta}}

\begin{aligned} \mathcal{L} = \; &\mathcal{L}_{\mathrm{vtm}} + \mathcal{L}_{\mathrm{vtc}} \\ &+ \mathcal{L}_{\mathrm{mlm}} + \mathcal{L}_{\mathrm{delta}} \end{aligned}

The head $F_\omega$ predicts from one frame's tokens how its patch features change $h$ sampled frames later, over the valid pairs $\Omega$ with $t + h \le T$ . Its output $\widehat{\Delta}_t$ holds one prediction $\widehat{\Delta}_{t,h}$ for each horizon $h$ . The matching, contrastive and masked-language losses train the reranker in the same pass. The head is discarded after training, so this objective adds no work at query time.

C

Online reranking

Sections 3.4 and 4.7 and Appendix A.1

\tilde{c}(q,v) = c(q,v) + e_\rho\big(\rho(q,v)\big),

s_\theta(q,v) = h_\psi\big(\tilde{c}(q,v)\big)

$c(q,v)$ is the pooled output of the joint encoder over the query tokens and the cache $Z(v)$ , and $\rho(q,v)$ is the first-stage score. The candidates are sorted by $s_\theta$ . Each query reads only its candidates' caches, so this traffic stays fixed as the collection grows.

03 · Use it

Index your own videos and rerank them with the released code.

Code by the authors. The recipe follows Sections 3 and 4.1 and Appendices A.1 to A.3 of the paper.

You need videos with captions and a GPU for training. One command trains VEDJE on any of the four datasets, and two more index your own videos and search them with text. The Colab notebook indexes a clip of your choice and times a query against a typical reranker, on a free CPU runtime.

GitHub: vedje Open the Colab tour

Index. Run the frozen visual encoder once per video and keep its frame-indexed features.
Prepare the first stage. Embed the training videos and captions with the first-stage retriever, and mine hard negatives from its rankings.
Train. Train the compressor, the joint reranker and the training-only heads together.
Rerank. Write each video's cache once. For each query, rerank the first stage's candidates from their caches.

Open source

Rerank your own collection.

Clone the repository and train VEDJE with one command. Two more commands, or a few lines of Python, index your videos once and search them with text.

Open on GitHub

# 1. clone and install (Python 3.10 or newer)
git clone https://github.com/GabrieleSerussi/vedje.git && cd vedje
pip install -e .

# 2. train on a GPU
vedje train configs/vedje_vp_msrvtt.yaml

# 3. index your videos once, then search them with text
vedje index output/vedje_vp_msrvtt/checkpoint_03.pth my_videos/ --out my_index.pt
vedje search my_index.pt "a person walks across a field"

04 · Details

What the cache keeps, and what each part adds.

Sections 4.3 to 4.7 of the paper, with Tables 3 to 6 and Appendix Table 11.

Separate frame groups beat pooled caches at the same storage.

The pooled caches and the per-frame cache each hold 64 tokens from the same features, read by the same reranker. Keeping one token group per sampled frame raises R@1 by 2.0 points in both directions over attention pooling. Feature-change supervision then raises it further.

R@1 gain over attention poolingTable 4 · MSR-VTT · 48 KiB caches · longer is better

Feature-change supervision improves R@1 at both cache sizes.

The comparison trains the same model with and without the training-only head. In text to video, the gain grows from 0.6 points with four tokens per frame to 1.9 points with one token per frame.

R@1 gain from feature-change supervisionTable 3 · MSR-VTT · R@1 points · longer is better

On the same cache, the joint encoder scores better than trained MaxSim.

Both scorers read the same cache and candidates, with the first-stage score as input. The first-stage score and the joint features complement each other: either alone reaches at most 50.1, and together they reach 54.6.

Text-to-video R@1 gain over the VideoPrism first stageTable 5 · MSR-VTT · same cache and candidates

Papers: MaxSim

With caches in host memory, a complete query takes under 11 ms with 200 candidates, and under 7 ms with the 12 KiB cache.

Both cache sizes cost about the same at 20 candidates, since text encoding and search are shared. With more candidates, the smaller cache is faster: at 200 candidates its latency is 33% lower.

Complete-query latency against the number of candidatesTable 6 · MSR-VTT text to video · VideoPrism first stage · candidates on a log scale