Overview

This is a classic string processing problem. We need to normalize the text (handle case and punctuation), split it into tokens, and count frequencies. Python's standard library gives us everything we need, so let's walk through the thinking from first principles before landing on a clean solution.

Clarifying the Problem

Before writing any code, let's pin down a few edge cases worth raising in an interview:

Case sensitivity: "Apple" and "apple" should count as the same word. We'll lowercase everything.
Punctuation: "hello," and "hello" should be the same token. We strip punctuation from word boundaries.
Contractions and hyphens: Does "don't" become one word or two? For simplicity here, we'll treat it as one token (strip non-alpha chars at the edges, keep internal structure simple). Worth clarifying with an interviewer.
Empty or whitespace-only input: Should return an empty dictionary.

Approach

The natural pipeline is three steps:

Normalize: Lowercase the entire string.
Tokenize: Split on whitespace to get raw tokens, then strip leading/trailing punctuation from each token.
Count: Use a dictionary (or collections.Counter) to tally occurrences, skipping any tokens that become empty after stripping.

We could also use a regular expression to extract all word-like sequences in one shot, which is often cleaner. Let's show both approaches.

Solution 1: Regex-based (preferred)

Using re.findall with a pattern that matches sequences of word characters is the most concise and correct approach. It naturally handles punctuation and whitespace in one step.

import re
from collections import Counter

def term_frequency(text: str) -> dict:
    # Normalize to lowercase, then extract all alphanumeric word tokens
    words = re.findall(r"[a-z0-9]+(?:'[a-z]+)*", text.lower())
    return dict(Counter(words))

The pattern [a-z0-9]+(?:'[a-z]+)* deserves a quick explanation:

[a-z0-9]+ matches the main body of a word (letters and digits).
(?:'[a-z]+)* optionally captures contractions like "don't" or "it's" as a single token. Remove this group if you want to split on apostrophes too.

For the input "Hello, world! Hello again.", this produces {'hello': 2, 'world': 1, 'again': 1}.

Solution 2: Manual strip (more transparent)

If you want to avoid regex (or the interviewer asks you to), here's the explicit version. It's slightly more verbose but very readable.

import string

def term_frequency(text: str) -> dict:
    counts = {}
    for token in text.lower().split():
        word = token.strip(string.punctuation)
        if word:  # skip tokens that were pure punctuation
            counts[word] = counts.get(word, 0) + 1
    return counts

The key move here is token.strip(string.punctuation), which removes leading and trailing punctuation characters. string.punctuation is a convenient built-in string containing all common punctuation symbols.

Tradeoffs Between the Two Approaches

The regex approach handles edge cases like multiple spaces, tabs, and newlines automatically, and the intent (extract word-like tokens) is captured directly in the pattern.
The manual approach is easier to read for engineers unfamiliar with regex, but strip only removes punctuation at the edges. It won't handle something like "word--word" cleanly.
Both run in O(n) time and space, where $n$ is the number of characters in the input. Counting is linear in the number of tokens, and each token is bounded by the input length.

Example Walkthrough

Let's trace through "The cat sat. The cat!" with the regex solution:

Lowercase: "the cat sat. the cat!"
re.findall extracts: ['the', 'cat', 'sat', 'the', 'cat']
Counter produces: {'the': 2, 'cat': 2, 'sat': 1}

The punctuation after "sat" and "cat" is simply never matched, so no stripping step is even needed.

Potential Extensions

An interviewer might push on any of these directions:

Stop words: Filter out common words like "the", "a", "is" using a predefined set. Easy to add as a filter on the token list.
Stemming or lemmatization: "running" and "runs" map to "run". This requires an NLP library like nltk or spacy, beyond pure stdlib.
Large files: If the document doesn't fit in memory, process it line by line and accumulate into a Counter. Counter supports += to merge counts incrementally.
Relative frequency (TF proper): Divide each count by the total number of words to get a value between 0 and 1, which is the standard definition of term frequency in information retrieval: $TF(t) = \frac{\text{count}(t)}{\sum_{w} \text{count}(w)}$.

2173. Calculate Term Frequency

Similar Questions