TF-IDF and BM25 Calculator
Compare classic TF-IDF and modern Okapi BM25 ranking side by side. Paste your own corpus and query terms to see how term frequency saturation and document length normalization alter search rankings, with every intermediate arithmetic step exposed.
1. Document Corpus
Edit the sample texts or add new documents (up to 10 documents, max 300 words each).
2. Search Query & Options
Enter one or more search terms. Calculations update instantly.
3. BM25 Parameters & Formulas
Adjust saturation and length normalization parameters to inspect their effect on scores.
Reference Formulas
Score(D, Q) = Σ TF(q, D) × ln(N / DF(q))Score(D, Q) = Σ IDF(q) × [ f(q, D) × (k1 + 1) ] / [ f(q, D) + k1 × B ]where B = (1 - b) + b × (|D| / avgdl)and IDF(q) = ln((N - DF(q) + 0.5) / (DF(q) + 0.5) + 1.0)4. Side-by-Side Scoring Results
Corpus: 5 documents | Average length:17.0 words | Active terms:"search"
| Document | TF | DF | IDF | TF-IDF Score | BM25 Score | TF-IDF Rank | BM25 Rank | Arithmetic |
|---|
Calculating results...
Understanding the Formulas: TF-IDF vs Okapi BM25
Classic Term Frequency-Inverse Document Frequency (TF-IDF) and Okapi BM25 form the mathematical bedrock of lexical information retrieval. While both ranking functions evaluate document relevance by matching query keywords against an inverted index, BM25 resolves two severe limitations that cause classic TF-IDF to fail in open web environments: unchecked keyword repetition and bias toward long documents.
In the results table above, Term Frequency measures how many times the query word occurs in a specific document, while Document Frequency records how many documents across the collection contain that word. Inverse Document Frequency (IDF) quantifies rarity. In classic TF-IDF, IDF is calculated as the natural logarithm of total documents divided by document frequency. BM25 uses the Robertson-Spärck Jones probabilistic formulation, adding smoothing constants to prevent negative weights when a word appears across most pages.
The primary divergence between the two models stems from how they score local keyword occurrences. Classic TF-IDF scales term frequency linearly: repeating a search term six times produces three times the score of mentioning it twice. In contrast, BM25 introduces term frequency saturation governed by the k1 parameter. As a keyword repeats, each additional occurrence yields diminishing score increases until reaching a strict mathematical ceiling. Setting k1 to 1.2 provides realistic saturation, preventing keyword stuffing from dominating search results. Setting k1 to zero eliminates term frequency entirely, transforming the algorithm into binary word matching.
The second divergence is document length normalization, controlled by the b parameter. Long texts naturally accumulate higher raw word counts purely through verbosity. BM25 calculates the ratio between each document's length and the average corpus length. When b is set to 0.75, documents longer than average suffer a score penalty, while concise documents with concentrated keyword density receive an explicit boost. Setting b to zero disables length normalization, treating short summaries and encyclopedic manuals identically.
To explore the probabilistic framework in greater depth, read our complete guide to BM25 ranking. For historical context on vector space models and inverse document frequency derivation, see our breakdown of TF-IDF mechanics.