Text Classification with Deep Learning

Text Classification with Deep Learning

The previous chapter ended on a clear limitation: bag-of-words classifiers ignore word order, so they cannot tell "not good" from "good" or catch the context that decides meaning. Deep learning fixes this. By combining word embeddings with neural networks that read text in sequence, a deep model can capture order, negation, and context in a way TF-IDF never could. This chapter shows how, and connects directly to the architectures from the Neural Networks and Deep Learning series.

The General Architecture

Almost every deep text classifier has the same three-part shape. First an embedding layer turns each token into a dense vector, the word embeddings from the representation chapter, either learned from scratch during training or loaded pretrained. Then a sequence model, usually a CNN or an RNN, reads those vectors and builds a single representation of the whole text that captures order and context. Finally a dense layer maps that representation to the output labels, ending in sigmoid for two classes or softmax for many, exactly as in the activation functions chapter. The interesting choices are all in the middle layer.

The Embedding Layer

Instead of a sparse TF-IDF vector, each word becomes a dense, learned vector. When the embedding layer is trained as part of the classifier, the vectors are tuned for your specific task, so words that matter for sentiment drift toward useful positions. You can also initialise it with pretrained embeddings like GloVe and either freeze or fine-tune them, which is the transfer learning idea from the Neural Networks series applied to text, and it helps a lot when your dataset is small.

Two Ways to Read the Sequence

The first option is a convolutional network for text. The CNNs we used for images also work on text by sliding 1D filters across the sequence of word vectors. Each filter learns to detect a useful phrase pattern, such as "waste of time" or "highly recommend," regardless of where it appears. Text CNNs are fast and strong, especially when local phrases carry the signal.

The second option is a recurrent network. An LSTM reads the text one word at a time, carrying a memory of what it has seen, which lets it model how earlier words change the meaning of later ones, exactly the negation and context that defeated bag-of-words. A bidirectional LSTM reads the text both forwards and backwards and usually does better, since context flows from both directions.

A Deep Sentiment Classifier in Keras

Here is the architecture as code: an embedding layer, a bidirectional LSTM to read the sequence, and a dense output for positive versus negative.

from tensorflow.keras import layers, models

vocab_size = 20000
max_len = 200          # pad/truncate every review to 200 tokens

model = models.Sequential([
    layers.Input(shape=(max_len,)),
    layers.Embedding(vocab_size, 128),          # learned word vectors
    layers.Bidirectional(layers.LSTM(64)),      # read sequence both ways
    layers.Dense(64, activation='relu'),
    layers.Dropout(0.5),                        # regularization
    layers.Dense(1, activation='sigmoid')       # positive vs negative
])

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.summary()

Two practical details. Text comes in different lengths, so every sequence is padded or truncated to a fixed length (here 200) before it reaches the model. And the Dropout layer is there because these models overfit easily on small text datasets, which is the regularization story from the hyperparameter tuning chapter.

Why This Beats Bag-of-Words

The improvement is structural, not incremental. Because the LSTM reads words in order while remembering context, "not good" and "good" produce genuinely different internal representations, and the embedding layer means the model already knows "great" and "fantastic" are related rather than treating every word as unrelated. Order, negation, and word meaning all become available to the model at once. On datasets with enough examples, deep classifiers clearly outperform TF-IDF, especially where context matters.

The Honest Caveat, and the Road Ahead

Deep models are not a free win. They need more training data than a TF-IDF classifier, take longer to train, and are harder to interpret. On a small dataset, a tuned logistic regression can still beat a neural network, which is why the previous chapter's classical baseline is always worth trying first. But there is a bigger shift coming. In modern practice you rarely train a text classifier from scratch at all. You take a Transformer that has already learned language from billions of words and fine-tune it on your labels, which beats both the classical and the from-scratch deep approaches. That requires understanding the attention mechanism and large language models, which is exactly where the series goes next.

Key Takeaways

• Deep text classifiers combine an embedding layer, a sequence model (CNN or RNN), and a dense output.
• The embedding layer gives words meaning; the sequence model captures order and context.
• Text CNNs detect phrase patterns; LSTMs and bidirectional LSTMs model word order and negation.
• Sequences are padded to a fixed length, and dropout helps control overfitting.
• Deep models beat bag-of-words when context matters and data is sufficient, but need more data and compute.
• Modern best practice is to fine-tune a pretrained Transformer rather than train from scratch.

What's Next

Every model so far, even the LSTM, processes text step by step and struggles to connect words that are far apart. The next part of the series introduces the idea that broke that barrier and now defines modern NLP: attention. We start with what attention is and why it changed everything.