Text PreprocessingOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Text Preprocessing

Before any model can process text, raw strings must become numbers. The decisions made at this stage (how to split words, how to handle morphology, how to weigh term importance) affect everything downstream. This is the data stage of the ML project loop for language: tokenization and embeddings are the representation, and a bad choice here caps every model you build on top, no matter how good the architecture.

Deep dive
The ML Project Loop
Includes runnable notebook

Tokenization

Tokenization splits text into discrete units (tokens) that the model processes. There are three common granularities:

GranularityExampleProsCons
Character"cat" → ['c','a','t']Handles any wordVery long sequences
Word"cats ran" → ["cats","ran"]IntuitiveOOV words; huge vocabulary
Subword"unhappy" → ["un","happy"]BalancedRequires training

Out-of-vocabulary (OOV) is the core problem with word-level tokenization: unseen words at test time become [UNK] tokens with no meaningful representation.

Byte-Pair Encoding (BPE)

BPE, used by GPT and many modern models, builds a subword vocabulary by iteratively merging the most frequent adjacent symbol pair: start from individual characters, repeatedly find the most common pair across the corpus and fuse it into a new token, until the vocabulary reaches its target size. The result is that common words become single tokens while rare words decompose into familiar subwords ("unhappiness" → ["un", "happi", "ness"]), so nothing maps to [UNK].

WordPiece (used by BERT) is similar but merges based on language model likelihood improvement rather than raw frequency, producing tokens prefixed with ## for word-continuation: "unhappiness" → ["un", "##happi", "##ness"].

Deep dive
Byte-Pair Encoding (BPE)
Includes runnable notebook

Stopwords and normalization

Stopwords are high-frequency words with low discriminative value ("the", "is", "a"). Removing them reduces vocabulary size and noise for bag-of-words models. However, for sequence models (RNNs, Transformers), stopwords carry grammatical structure and should usually be kept.

What each preprocessing stage throws away

The pipelinestep 1 of 4
Twelve pairs that differ by one word
Each row is a matched pair: the same sentence with and without a negation. Lowercasing and tokenising changes nothing that matters — all 12 pairs stay distinguishable, and the vocabulary is 44 types over 5.2 tokens per document.

This is the state every preprocessing tutorial starts from, and the last point at which the corpus still contains what it started with.

Note that the only difference within each pair is the negation word itself.

tokenizelowercase + tokenize · vocab 44 · 5.2 tokens/doc
+thefoodwasgood
−thefoodwasnotgood
+iwouldrecommendthisplace
−iwouldnotrecommendthisplace
+thestaffwerehelpful
−thestaffwerenothelpful
+thisisworththemoney
−thisisnotworththemoney
+theroomwasclean
−theroomwasnotclean
+iwashappywiththeservice
−iwasnothappywiththeservice
+thedeliverywasontime
−thedeliverywasnotontime
+itworksasdescribed
−itdoesnotworkasdescribed
+thescreenisbright
−thescreenisnotbright
+icanhearitclearly
−icannothearitclearly
+thebatterylastsallday
−thebatterydoesnotlastallday
+thiswasagreatexperience
−thiswasnotagreatexperience
pairs collapsed

0 of 12 now indistinguishable
stemming alone would collapse 0

what it bought

vocab 44 → 44
tokens/doc 5.2 → 5.2

token kepttoken droppedpair now identical

Twelve matched pairs — the same sentence with and without a negation — pushed through a standard pipeline. Because the pairs differ by exactly one word, "preprocessing loses information" becomes a count: either a pair survives distinguishable or it does not. The stopword list is a standard English one of the kind shipped with NLTK and scikit-learn.

Push twelve matched sentence pairs (the same sentence with and without a negation) through a standard pipeline and the cost stops being abstract. Stopword removal does exactly what it advertises: vocabulary falls from 44 types to 28, documents shrink from 5.2 tokens to 2.2. It also collapses 10 of the 12 pairs into byte-identical token lists, because not is on the standard English stopword list, along with no, nor and cannot. "The food was not good" becomes [food good].

Adding stemming takes it to 12 of 12. But stemming is not the culprit — applied alone it collapses 0 of 12, and only closes the last two pairs that stopword removal had already opened.

The consequence is stronger than "accuracy drops". Two sentences with opposite labels now have identical feature vectors, so no classifier of any architecture can beat chance on them. The information was destroyed in a utility function, upstream of every model you will try, and nothing in the pipeline reports it.

Which is why the honest framing is task-specific rather than "stopword removal is bad". These lists were built for keyword retrieval, where a term appearing in nearly every document really is noise — matching not retrieves the whole corpus. Carrying the same list into a sentiment task is where it breaks. Choose the list for the task you have, and for anything sentiment-shaped, keep the negations.

Case normalization: lowercase all text. "Apple" and "apple" refer to the same concept in most contexts.

Punctuation removal: "Hello!" → "Hello". Keep punctuation for sentiment analysis (exclamation marks carry signal) or named entity recognition.

Stemming vs Lemmatization

Both reduce words to a common base form, but differently:

Stemming applies heuristic rules to strip suffixes. Fast but approximate:

  • "running" → "run"
  • "happiness" → "happi" (incorrect English but consistent)
  • "studies" → "studi"

Lemmatization uses morphological analysis and a lexicon. Correct but slower:

  • "running" → "run" (same result)
  • "happiness" → "happiness" (not a conjugation, no change)
  • "better" → "good" (knows this is comparative of "good")

When to use which

Use stemming for information retrieval tasks where approximate matching is fine and speed matters (search engines). Use lemmatization when you need linguistically correct base forms, or when working with morphologically rich languages. For deep learning models that use subword tokenization, neither is usually necessary.

TF-IDF

TF-IDF (Term Frequency–Inverse Document Frequency) weights terms by how distinctive they are to a document relative to the corpus.

TF(t,d)=count of t in dtotal terms in d\text{TF}(t, d) = \frac{\text{count of } t \text{ in } d}{\text{total terms in } d} IDF(t)=log⁡ ⁣(N1+df(t))\text{IDF}(t) = \log\!\left(\frac{N}{1 + \text{df}(t)}\right) TF-IDF(t,d)=TF(t,d)×IDF(t)\text{TF-IDF}(t, d) = \text{TF}(t, d) \times \text{IDF}(t)

where NN is the number of documents and df(t)\text{df}(t) is the number of documents containing term tt.

Key properties:

  • A term appearing in (almost) every document has df≈N\text{df} \approx N, so IDF≈log⁡(N/N)=0\text{IDF} \approx \log(N/N) = 0 → TF-IDF ≈0\approx 0 (stopword behavior); the 1+df1+\text{df} smoothing just keeps IDF finite
  • A rare term appearing once has high IDF → high TF-IDF
  • Frequent terms in a document are boosted by TF, but only if they're not ubiquitous across the corpus

Worked example: 3 documents, word "machine" appears in 1 of them, 3 times in that document (50 words total):

TF=3/50=0.06,IDF=log⁡(3/2)=0.405,TF-IDF=0.024\text{TF} = 3/50 = 0.06, \quad \text{IDF} = \log(3/2) = 0.405, \quad \text{TF-IDF} = 0.024
Python
from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "the quick brown fox",
    "the fox jumped over the dog",
    "machine learning is fascinating",
]
vec = TfidfVectorizer()
X = vec.fit_transform(corpus)
print(vec.get_feature_names_out())
print(X.toarray().round(3))

Sentence and document representations

Beyond individual tokens, you often need a fixed-size vector for an entire sentence or document:

MethodDescriptionStrength
Bag of Words (BoW)Count vector over vocabularyInterpretable, fast
TF-IDF vectorWeighted BoWBetter signal than raw counts
Average word embeddingsMean of word vectorsCheap, surprisingly effective
[CLS] token (BERT)Transformer representationRich context-aware
Sentence-BERTSiamese network fine-tuned for similarityBest for semantic similarity

Exercise

BPE tokenizes the word "unhappiness" into subword pieces like ["un", "happi", "ness"]. How would a word-level tokenizer handle an unseen word like "unhappiness" at test time?

Need a hint?

Think about what happens when a word isn't in the word-level vocabulary.

Exercise

A preprocessing pipeline converts "better" → "good". Is this stemming or lemmatization, and why?

Need a hint?

Stemming uses heuristic suffix rules; lemmatization uses linguistic knowledge.

Exercise

The word "the" appears in every document in a 10,000-document corpus. What is its TF-IDF score?

Need a hint?

IDF = log(N / (1 + df(t))). What happens when df(t) = N?

Common mistakes

  1. Lowercasing before NER: "Apple" (company) and "apple" (fruit) are different; lowercase only after entity extraction.
  2. Removing stopwords for sequence models: "not happy" vs "happy" differ only in the stopword "not". Sequence models need stopwords.
  3. Fitting TF-IDF on the entire dataset before splitting: IDF should be computed only on the training set to avoid leakage.
  4. Applying stemming to already-tokenized subwords: BPE subwords like "##ness" will be incorrectly stripped.
  5. Ignoring tokenizer vocabulary mismatch: the tokenizer used at inference must match the one used during fine-tuning exactly.

Key takeaways

  • Tokenization granularity trades off vocabulary size vs sequence length; BPE subword tokenization is the modern standard.
  • BPE handles OOV by decomposing rare words into known subword pieces; WordPiece does the same for BERT.
  • Stemming is fast/approximate; lemmatization is slow/correct; neither is usually needed with subword models.
  • TF-IDF weights terms by local frequency × global rarity; stopwords effectively have zero TF-IDF.
  • For sequence models (RNNs, Transformers), skip stopword removal and stemming — they use raw subword tokens.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Text PreprocessingRecurrent Neural NetworksAttention & TransformersThe Transformer Architect…Word EmbeddingsSelf-AttentionThe ML Project LoopDependency ParsingByte-Pair Encoding (BPE)GloVe Training Procedure