BM25

BM25 scores how well a document matches a query.

$BM25(D,Q)={\sum }_{q\in Q}IDF(q)\cdot \frac{TF(q,D)\cdot (k_1+1)}{TF(q,D)+k_1\cdot (1-b+b\cdot \frac{|D|}{avgdl})}$

IDF (Inverse Document Frequency)

Measures how rare/important a word is.

$IDF(q)=\log \left(\frac{N-n_q+0.5}{n_q+0.5}\right)$ Where:

• $N$ = total documents in corpus
• $n_q$ = documents containing word $q$ Example:
• Word “the” appears in 1,000,000 out of 1,000,000 docs → IDF ≈ -7 (useless, penalized)
• Word “transformer” appears in 50 out of 1,000,000 docs → IDF ≈ 9.9 (very useful, boosted)

Common words (the, and, is) get low or negative scores. Rare, meaningful words get high scores.

TF Component (Term Frequency with Saturation)

Counts how many times the word appears in the document, but with diminishing returns.

$\frac{TF(q,D)\cdot (k_1+1)}{TF(q,D)+k_1}$ Where:

• $TF(q,D)$ = count of word $q$ in document $D$
• $k_1$ = saturation parameter (default 1.2)

If a doc mentions “AI” 1 time vs 100 times, the 100-time doc isn’t 100x more relevant. Repetition has diminishing returns.

Length Normalization

Fairness adjustment so long documents don’t always win.

$1-b+b\cdot \frac{|D|}{avgdl}$ Where:

• $b$ = normalization strength (default 0.75)
• $|D|$ = length of document (word count)
• $avgdl$ = average document length in corpus

A long document naturally has more word matches. But a short article specifically about a topic is more relevant than a passing mention in a textbook.