Text Preprocessing
Before any model can process text, raw strings must become numbers. The decisions made at this stage (how to split words, how to handle morphology, how to weigh term importance) affect everything downstream. This is the data stage of the ML project loop for language: tokenization and embeddings are the representation, and a bad choice here caps every model you build on top, no matter how good the architecture.
Tokenization
Tokenization splits text into discrete units (tokens) that the model processes. There are three common granularities:
| Granularity | Example | Pros | Cons |
|---|---|---|---|
| Character | "cat" → ['c','a','t'] | Handles any word | Very long sequences |
| Word | "cats ran" → ["cats","ran"] | Intuitive | OOV words; huge vocabulary |
| Subword | "unhappy" → ["un","happy"] | Balanced | Requires training |
Out-of-vocabulary (OOV) is the core problem with word-level tokenization: unseen words at test time become [UNK] tokens with no meaningful representation.
Byte-Pair Encoding (BPE)
BPE, used by GPT and many modern models, builds a subword vocabulary by iteratively merging the most frequent adjacent symbol pair: start from individual characters, repeatedly find the most common pair across the corpus and fuse it into a new token, until the vocabulary reaches its target size. The result is that common words become single tokens while rare words decompose into familiar subwords ("unhappiness" → ["un", "happi", "ness"]), so nothing maps to [UNK].
WordPiece (used by BERT) is similar but merges based on language model likelihood improvement rather than raw frequency, producing tokens prefixed with ## for word-continuation: "unhappiness" → ["un", "##happi", "##ness"].
Stopwords and normalization
Stopwords are high-frequency words with low discriminative value ("the", "is", "a"). Removing them reduces vocabulary size and noise for bag-of-words models. However, for sequence models (RNNs, Transformers), stopwords carry grammatical structure and should usually be kept.
What each preprocessing stage throws away
Twelve pairs that differ by one word
This is the state every preprocessing tutorial starts from, and the last point at which the corpus still contains what it started with.
Note that the only difference within each pair is the negation word itself.
pairs collapsed
0 of 12 now indistinguishable
stemming alone would collapse 0
what it bought
vocab 44 → 44
tokens/doc 5.2 → 5.2
Twelve matched pairs — the same sentence with and without a negation — pushed through a standard pipeline. Because the pairs differ by exactly one word, "preprocessing loses information" becomes a count: either a pair survives distinguishable or it does not. The stopword list is a standard English one of the kind shipped with NLTK and scikit-learn.
Push twelve matched sentence pairs (the same sentence with and without a negation) through a standard pipeline and the cost stops being abstract. Stopword removal does exactly what it advertises: vocabulary falls from 44 types to 28, documents shrink from 5.2 tokens to 2.2. It also collapses 10 of the 12 pairs into byte-identical token lists, because not is on the standard English stopword list, along with no, nor and cannot. "The food was not good" becomes [food good].
Adding stemming takes it to 12 of 12. But stemming is not the culprit — applied alone it collapses 0 of 12, and only closes the last two pairs that stopword removal had already opened.
The consequence is stronger than "accuracy drops". Two sentences with opposite labels now have identical feature vectors, so no classifier of any architecture can beat chance on them. The information was destroyed in a utility function, upstream of every model you will try, and nothing in the pipeline reports it.
Which is why the honest framing is task-specific rather than "stopword removal is bad". These lists were built for keyword retrieval, where a term appearing in nearly every document really is noise — matching not retrieves the whole corpus. Carrying the same list into a sentiment task is where it breaks. Choose the list for the task you have, and for anything sentiment-shaped, keep the negations.
Case normalization: lowercase all text. "Apple" and "apple" refer to the same concept in most contexts.
Punctuation removal: "Hello!" → "Hello". Keep punctuation for sentiment analysis (exclamation marks carry signal) or named entity recognition.
Stemming vs Lemmatization
Both reduce words to a common base form, but differently:
Stemming applies heuristic rules to strip suffixes. Fast but approximate:
- "running" → "run"
- "happiness" → "happi" (incorrect English but consistent)
- "studies" → "studi"
Lemmatization uses morphological analysis and a lexicon. Correct but slower:
- "running" → "run" (same result)
- "happiness" → "happiness" (not a conjugation, no change)
- "better" → "good" (knows this is comparative of "good")
When to use which
Use stemming for information retrieval tasks where approximate matching is fine and speed matters (search engines). Use lemmatization when you need linguistically correct base forms, or when working with morphologically rich languages. For deep learning models that use subword tokenization, neither is usually necessary.
TF-IDF
TF-IDF (Term Frequency–Inverse Document Frequency) weights terms by how distinctive they are to a document relative to the corpus.
where is the number of documents and is the number of documents containing term .
Key properties:
- A term appearing in (almost) every document has , so → TF-IDF (stopword behavior); the smoothing just keeps IDF finite
- A rare term appearing once has high IDF → high TF-IDF
- Frequent terms in a document are boosted by TF, but only if they're not ubiquitous across the corpus
Worked example: 3 documents, word "machine" appears in 1 of them, 3 times in that document (50 words total):
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
"the quick brown fox",
"the fox jumped over the dog",
"machine learning is fascinating",
]
vec = TfidfVectorizer()
X = vec.fit_transform(corpus)
print(vec.get_feature_names_out())
print(X.toarray().round(3))
Sentence and document representations
Beyond individual tokens, you often need a fixed-size vector for an entire sentence or document:
| Method | Description | Strength |
|---|---|---|
| Bag of Words (BoW) | Count vector over vocabulary | Interpretable, fast |
| TF-IDF vector | Weighted BoW | Better signal than raw counts |
| Average word embeddings | Mean of word vectors | Cheap, surprisingly effective |
| [CLS] token (BERT) | Transformer representation | Rich context-aware |
| Sentence-BERT | Siamese network fine-tuned for similarity | Best for semantic similarity |
Exercise
BPE tokenizes the word "unhappiness" into subword pieces like ["un", "happi", "ness"]. How would a word-level tokenizer handle an unseen word like "unhappiness" at test time?
Need a hint?
Think about what happens when a word isn't in the word-level vocabulary.
Exercise
A preprocessing pipeline converts "better" → "good". Is this stemming or lemmatization, and why?
Need a hint?
Stemming uses heuristic suffix rules; lemmatization uses linguistic knowledge.
Exercise
The word "the" appears in every document in a 10,000-document corpus. What is its TF-IDF score?
Need a hint?
IDF = log(N / (1 + df(t))). What happens when df(t) = N?
Common mistakes
- Lowercasing before NER: "Apple" (company) and "apple" (fruit) are different; lowercase only after entity extraction.
- Removing stopwords for sequence models: "not happy" vs "happy" differ only in the stopword "not". Sequence models need stopwords.
- Fitting TF-IDF on the entire dataset before splitting: IDF should be computed only on the training set to avoid leakage.
- Applying stemming to already-tokenized subwords: BPE subwords like "##ness" will be incorrectly stripped.
- Ignoring tokenizer vocabulary mismatch: the tokenizer used at inference must match the one used during fine-tuning exactly.
Key takeaways
- Tokenization granularity trades off vocabulary size vs sequence length; BPE subword tokenization is the modern standard.
- BPE handles OOV by decomposing rare words into known subword pieces; WordPiece does the same for BERT.
- Stemming is fast/approximate; lemmatization is slow/correct; neither is usually needed with subword models.
- TF-IDF weights terms by local frequency × global rarity; stopwords effectively have zero TF-IDF.
- For sequence models (RNNs, Transformers), skip stopword removal and stemming — they use raw subword tokens.
Related concepts
- Word Embeddings: turning tokens into dense vectors
- Attention & Transformers: models that process token sequences
- Dependency Parsing: recovering grammatical structure with transition- and graph-based parsers