How to Calculate TF-IDF: Step-by-Step Worked Examples

TF-IDF stands for Term Frequency-Inverse Document Frequency, a foundational mathematical weighting scheme in information retrieval that scores how informative a word is to a document within a broader collection. By balancing local word occurrences against corpus-wide rarity, TF-IDF enables search engines to rank documents by topical relevance rather than raw length. It forms the baseline for lexical scoring.

What TF-IDF is and why search engines use it

TF-IDF is a statistical formula that quantifies the relevance of a keyword to a specific document inside a collection of texts. The metric solves a primary challenge in information retrieval: common words like “the”, “and”, and “is” appear constantly across every document, yet provide zero topical insight. Conversely, specific technical terms like “photosynthesis” appear rarely across the web, yet indicate strong subject matter relevance whenever they occur.

In 1957, IBM researcher Hans Peter Luhn proposed that the frequency of a word within a text reflects its topical importance, establishing the concept of Term Frequency. In 1972, British computer scientist Karen Spärck Jones at Cambridge University introduced Inverse Document Frequency, demonstrating mathematically that terms appearing across fewer documents possess higher retrieval specificity. Cornell University professor Gerard Salton later combined both principles in the SMART retrieval system, creating modern TF-IDF.

Search engines use TF-IDF because pure keyword matching fails to distinguish between generic filler and meaningful subject matter. If a search engine simply tallied keyword occurrences, the longest documents containing thousands of words would always rank highest. TF-IDF normalizes for document length and discounts universal vocabulary, allowing search algorithms to identify documents that genuinely concentrate on a specific query topic.

In modern search engine architectures, TF-IDF values are pre-computed during indexing and stored directly within posting lists inside inverted index data structures. When a user enters a query, the search engine quickly multiplies query term weights against indexed document weights, generating an initial candidate pool in milliseconds.

The Term Frequency (TF) formula and variants

Term Frequency measures how often a specific term occurs within an individual document. The underlying intuition is straightforward: the more times an author uses a term within an article, the more likely that article covers the subject. However, calculating Term Frequency requires mathematical adjustments to prevent long documents from dominating short ones.

The most basic formulation is raw term count:

TF_raw(t, d) = f(t, d)

In this formula, f(t, d) represents the total number of times term t appears in document d. While simple, raw count introduces severe bias toward long documents. A twenty-thousand-word academic thesis naturally mentions a keyword more times than a focused five-hundred-word reference guide, even if the short guide addresses the topic more directly.

To correct for document length, computer scientists use relative Term Frequency, dividing raw counts by total document words:

TF_relative(t, d) = f(t, d) / TotalWords(d)

Another common variation is log-normalized Term Frequency. In natural language, seeing a term ten times does not make a document ten times more relevant than a document mentioning it once. To model diminishing returns, logarithmic scaling compresses raw counts:

TF_log(t, d) = 1 + log10(f(t, d))  [for f(t, d) > 0]

Under logarithmic scaling, one occurrence produces a score of 1.0, ten occurrences produce a score of 2.0, and one hundred occurrences produce 3.0. This sublinear curve prevents excessive keyword repetition from distorting document relevance.

Summary of Term Frequency Variations:
1. Raw Count:               TF = f(t, d)
2. Relative Frequency:      TF = f(t, d) / TotalWords(d)
3. Log-Normalized:          TF = 1 + log10(f(t, d))  [if f > 0]
4. Double Normalization:    TF = 0.5 + 0.5 * (f(t, d) / max_f(d))

The Inverse Document Frequency (IDF) formula and intuition

Inverse Document Frequency measures how common or rare a term is across the entire corpus of documents. While Term Frequency evaluates words locally inside a single document, IDF operates globally across all documents in an index. It serves as a mathematical penalty for ubiquitous words and a reward for specialized terminology.

In 1972, Karen Spärck Jones formulated IDF using logarithmic scaling to reflect information theory:

IDF(t, D) = log10(N / DF_t)

In this equation:

• N represents the total number of documents in the corpus collection D.
• DF_t represents the Document Frequency, the count of documents that contain term t at least once.
• log10 represents the base-10 logarithm used to scale the resulting ratio smoothly.

Corpus Rarity Spectrum (N = 1,000,000 Documents):
Term: "the"         DF = 1,000,000  ──> IDF = log10(1,000,000 / 1,000,000) = log10(1) = 0.00
Term: "internet"    DF = 100,000    ──> IDF = log10(1,000,000 / 100,000)   = log10(10) = 1.00
Term: "cryptography" DF = 1,000     ──> IDF = log10(1,000,000 / 1,000)     = log10(1000) = 3.00
Result: "cryptography" receives three times the weight of "internet", while "the" is zeroed out.

The logarithm plays a critical role in scaling. If a word appears in every single document across a corpus, DF_t equals N. The fraction evaluates to 1, and log10(1) equals 0. This outcome ensures that universal stop words naturally receive a weight of zero, eliminating the need for manual stop word removal filters in many retrieval tasks.

Conversely, if a specialized medical term appears in only one document out of one million indexed pages, the fraction evaluates to 1,000,000, producing an IDF of 6.0. When combined with Term Frequency, this high multiplier ensures that documents containing rare, informative terms immediately surface at the top of candidate retrieval pools.

The full TF-IDF formula combined

The complete TF-IDF score is calculated by multiplying the Term Frequency of a word in a specific document by the Inverse Document Frequency of that word across the corpus. Combining both metrics balances local term density against global rarity, producing a composite numerical score that reflects document relevance.

The standard composite mathematical formula is:

TF-IDF(t, d, D) = TF(t, d) * IDF(t, D)

Using relative Term Frequency and base-10 logarithmic Inverse Document Frequency, the expanded formula reads:

TF-IDF(t, d, D) = (f(t, d) / TotalWords(d)) * log10(N / DF_t)

In this combined equation, TotalWords(d) represents the total word count of document d. A term achieves a high TF-IDF score when it appears frequently within a short, focused document, and appears rarely across the rest of the corpus. Conversely, a term receives a low score if it appears infrequently, occurs inside a massive document, or appears across almost every page on the web.

The Four TF-IDF Score Quadrants:
┌──────────────────────────────┬──────────────────────────────┐
│ High TF + High IDF:          │ Low TF + High IDF:           │
│ MAX SCORE                    │ MODERATE SCORE               │
│ (Term mentioned often,       │ (Rare term mentioned once,   │
│ word is rare in corpus)      │ provides useful signal)      │
├──────────────────────────────┼──────────────────────────────┤
│ High TF + Low IDF:           │ Low TF + Low IDF:            │
│ LOW SCORE                    │ MINIMUM SCORE                │
│ (Common word repeated often, │ (Common word mentioned once, │
│ e.g., "the" or "search")     │ zero informative value)      │
└──────────────────────────────┴──────────────────────────────┘

This composite behavior makes TF-IDF resilient against document formatting variations. It rewards topical concentration while naturally filtering out background vocabulary, providing the primary foundation for search engine ranking algorithms.

Worked numerical example: Calculating TF-IDF by hand

To understand how the arithmetic functions in practice, consider a miniature search engine index containing a corpus of exactly three documents where N equals 3. We will calculate the TF-IDF score for each word across all documents step by step.

Here are the three documents in our sample corpus:

• Document 1 (D1): “search engine indexing” (Total words: 3)
• Document 2 (D2): “search engine crawling and indexing” (Total words: 5)
• Document 3 (D3): “web crawler and search engine basics” (Total words: 6)

Step 1: Count raw term frequencies and document frequencies

First, we compile the complete vocabulary of unique terms across the entire corpus and count how many times each word appears in each document, alongside the total number of documents containing that word (DF):

Step 2: Calculate the Inverse Document Frequency (IDF) for each term

With total corpus size N = 3, we calculate the base-10 logarithm IDF = log10(3 / DF):

• For search: log10(3 / 3) = log10(1) = 0.000
• For engine: log10(3 / 3) = log10(1) = 0.000
• For indexing: log10(3 / 2) = log10(1.5) = 0.176
• For crawling: log10(3 / 1) = log10(3) = 0.477
• For and: log10(3 / 2) = log10(1.5) = 0.176
• For web: log10(3 / 1) = log10(3) = 0.477
• For crawler: log10(3 / 1) = log10(3) = 0.477
• For basics: log10(3 / 1) = log10(3) = 0.477

Notice that the terms “search” and “engine” appear in all three documents. Because they offer zero discriminative power in this corpus, their calculated IDF score is exactly 0.000.

Step 3: Calculate relative Term Frequency (TF) for each document

Next, we divide each raw word count by the document’s total word length:

• Document 1 (Length = 3):
  • TF(search) = 1/3 = 0.333
  • TF(engine) = 1/3 = 0.333
  • TF(indexing) = 1/3 = 0.333
• Document 2 (Length = 5):
  • TF(search) = 1/5 = 0.200
  • TF(engine) = 1/5 = 0.200
  • TF(crawling) = 1/5 = 0.200
  • TF(and) = 1/5 = 0.200
  • TF(indexing) = 1/5 = 0.200
• Document 3 (Length = 6):
  • TF(web) = 1/6 = 0.167
  • TF(crawler) = 1/6 = 0.167
  • TF(and) = 1/6 = 0.167
  • TF(search) = 1/6 = 0.167
  • TF(engine) = 1/6 = 0.167
  • TF(basics) = 1/6 = 0.167

Step 4: Multiply TF by IDF to produce the final weight matrix

Now we multiply each cell’s relative TF by the term’s global IDF (TF * IDF):

Suppose a user searches for the query: “search indexing”. We sum the matching TF-IDF scores for each document:

• Document 1 Score: Score(D1) = 0.000 (search) + 0.059 (indexing) = 0.059
• Document 2 Score: Score(D2) = 0.000 (search) + 0.035 (indexing) = 0.035
• Document 3 Score: Score(D3) = 0.000 (search) + 0.000 (indexing) = 0.000

Document 1 ranks in first place. Even though both D1 and D2 contain the target word “indexing” exactly once, Document 1 is shorter and more topically dense. Its term frequency for “indexing” is thirty-three percent compared to twenty percent in D2, earning Document 1 the top position.

Multi-document vector space scoring and cosine similarity

In the vector space model developed by Gerard Salton, documents and search queries are represented as multi-dimensional vectors where each dimension corresponds to a distinct word in the vocabulary. The numerical value along each axis is the calculated TF-IDF weight. This geometric framework allows search engines to rank documents by calculating the angle between query and document vectors.

To evaluate relevance, search engines compute the cosine similarity between query vector q and document vector d:

Cosine Similarity(q, d) = dot_product(q, d) / (||q|| * ||d||)

In this formula, the numerator represents the dot product of matching term weights, while the denominator multiplies the Euclidean lengths (norms) of both vectors.

Vector Space Geometric Representation:
       Dimension: "indexing" (IDF: 0.176)
             ▲
             │          [D1] Document 1 (Vector: [0.00, 0.059])
             │        / 
             │       /  [Q]  Query Vector: [0.00, 0.176]
             │      / /
             │     / /  [D2] Document 2 (Vector: [0.00, 0.035])
             │    / / /
             │   / / /
             └────────────────────────► Dimension: "search" (IDF: 0.000)
             Smallest angle = Highest Cosine Similarity = Rank #1

Cosine similarity measures the orientation of the vectors rather than their absolute magnitude. If two documents discuss the exact same topic using identical term proportions, their vectors point in the exact same direction. Even if one document is ten times longer than the other, the cosine of the angle between them remains 1.0, indicating identical topical focus.

The denominator normalizes vector lengths to unit length. This mathematical normalization prevents longer articles from receiving an unfair advantage merely because they accumulate higher raw vector lengths across diverse topics. Vector space retrieval remains a core component of classical ranking mechanisms.

Limitations of TF-IDF and the evolution to Okapi BM25

While TF-IDF established the foundation of modern search retrieval, the pure formula exhibits significant mathematical limitations when applied to large web indexes. In production environments, pure TF-IDF suffers from linear term frequency distortion, inadequate document length penalties, and poor handling of repetitive keyword stuffing.

The primary flaw of TF-IDF is linear term frequency growth. Under standard TF formulations, a document that repeats a keyword fifty times scores fifty times higher than a document mentioning it once. This mathematical property incentivized early web publishers to spam repetitive keywords in footers, white-on-white text, and metadata to artificially manipulate search engine relevance scores.

To solve these systemic limitations, researchers Stephen Robertson and Karen Spärck Jones developed the Okapi BM25 algorithm at City University London. BM25 introduces an asymptotic saturation curve controlled by parameter k1. In BM25, repeating a keyword provides diminishing incremental gains, and scores plateau rapidly regardless of how many times a term is repeated.

TF-IDF Linear Growth vs BM25 Saturation Curve:
Term Score
    ▲                           / Linear TF-IDF (Unlimited growth)
    │                         /
    │                       /
    │   BM25 Saturation   /
    │   ┌────────────────────────── Asymptotic Ceiling (k1 + 1)
    │  /
    │ /
    │/
    └───────────────────────────────────────► Term Frequency in Document

BM25 also refines document length normalization through parameter b. Instead of dividing strictly by document length, BM25 compares a document’s length against the average length of all documents in the corpus. This prevents short snippets from artificially dominating search results while protecting comprehensive long-form articles.

While modern commercial search engines rely on BM25 and neural transformer models for primary retrieval, TF-IDF remains widely used for offline feature extraction, document clustering, text summarization, and baseline retrieval tasks across software engineering.

Common misconceptions about TF-IDF in modern SEO

Within the search engine optimization industry, TF-IDF is frequently misunderstood and marketed as a tool for content optimization. Many third-party SEO software applications promote TF-IDF analysis tools, claiming that calculating word frequencies against top-ranking pages reveals the exact keyword percentages required to rank on Google. These claims contradict modern information retrieval science.

The most common misconception is that search engines use TF-IDF to grade on-page keyword density. In reality, modern search engines do not compare your article’s term frequency against a competitor’s percentage to evaluate quality. Search engines operate deep learning semantic systems like BERT and RankBrain that understand topical context and user intent without requiring rigid keyword counts.

SEO Myth vs Engineering Reality:
Myth: "Match top 10 competitors' TF-IDF score for 35 secondary keywords to rank."
Reality: TF-IDF is an inverted index retrieval filter, not a content quality score.
Modern engines rely on neural matching, intent resolution, and entity graphs.

Another widespread myth is that TF-IDF can identify missing secret keywords that guarantee ranking improvements. While running TF-IDF across top-ranking articles can highlight broad subtopics or technical terms you may have overlooked during drafting, mechanically inserting those words into sentences does not improve relevance. Search algorithms easily detect unnatural keyword stuffing and prioritize helpful, coherent editorial analysis.

Treating TF-IDF as a tactical keyword checklist distracts developers and writers from genuine optimization goals. Success in modern search engine optimization requires creating authoritative, comprehensive content that addresses search intent, earns authentic citations, and provides exceptional user experiences as documented across Search Engine Basics.

Frequently asked questions

What is TF-IDF in simple terms?

TF-IDF is a mathematical metric that measures how important a word is to a document within a collection of documents. It rewards words that appear frequently inside a specific article while penalizing words that appear across almost every document on the web, highlighting meaningful topical keywords.

What is the formula for calculating TF-IDF?

The standard formula multiplies Term Frequency by Inverse Document Frequency: TF-IDF equals TF times IDF. Term Frequency measures the count or density of a term within a document, while Inverse Document Frequency calculates the logarithm of total documents divided by documents containing that term.

Why does the IDF formula use a logarithm?

The IDF formula uses a logarithm to scale document frequency ratios smoothly. Without a logarithm, rare words in large collections would receive disproportionately massive multipliers that overwhelm all other signals, while common words appearing in every document would fail to cancel out cleanly to zero.

How does TF-IDF prevent common words from dominating search results?

TF-IDF prevents common words from dominating search results through its Inverse Document Frequency calculation. When a word like “the” or “is” appears in every document across an index, the ratio N divided by DF equals one, and the logarithm of one evaluates to zero, eliminating common words.

What is the difference between TF-IDF and Okapi BM25?

TF-IDF increases linearly with keyword repetition, allowing excessive keyword stuffing to inflate document scores. Okapi BM25 introduces an asymptotic saturation curve where additional term repetitions yield diminishing marginal gains, alongside refined document length normalization comparing page lengths against the corpus average.

Can you calculate TF-IDF on a single webpage?

You cannot calculate TF-IDF on a single isolated webpage because the Inverse Document Frequency component requires a broader corpus of documents for comparison. Without knowing how frequently a term occurs across an entire collection of texts, you cannot determine whether that term is rare or ubiquitous.

How does document length affect raw TF-IDF calculations?

Document length distorts raw TF-IDF calculations because longer documents naturally mention keywords more times than short documents. To prevent long articles from monopolizing rankings unfairly, search engines divide raw term counts by document length or apply sublinear logarithmic scaling to normalize frequency.

Is TF-IDF still used in modern search engines?

Modern search engines have largely replaced pure TF-IDF with Okapi BM25 and neural vector models for live search ranking. However, TF-IDF remains fundamental as a baseline indexing concept, feature extraction technique for machine learning, and rapid candidate retrieval filter in distributed information retrieval architectures.

Sources

• A Statistical Interpretation of Term Specificity and Its Application in Retrieval (Spärck Jones, 1972)
• Introduction to Modern Information Retrieval (Salton and McGill, 1983)
• Introduction to Information Retrieval (Manning, Raghavan, Schütze, 2008)
• The Probabilistic Relevance Framework: BM25 and Beyond (Robertson and Zaragoza, 2009)