Large Language Models: BERT and GPT

Large Language Models: BERT and GPT

Take the attention mechanism from the previous chapter, stack it into a deep architecture called the Transformer, and train it on a truly enormous amount of text. What you get is a large language model, or LLM, the technology behind BERT, GPT, and the assistants that brought NLP into everyday life. This chapter explains how these models are built and trained, the two great families they fall into, how you actually use them, and where they fall short.

The Key Idea: Pretrain, Then Adapt

The breakthrough was not just the architecture but the recipe. Instead of training a separate model from scratch for every task, you first pretrain one large model on a generic objective over a vast corpus, billions of words from the web and books. Through that, the model learns grammar, facts, and a great deal of how language works. Then you adapt that single pretrained model to specific tasks. This is exactly the transfer learning idea from the Neural Networks series, applied to language at scale, and it is why a small team with a modest dataset can now get results that once needed a research lab.

One detail carries over from the preprocessing chapter: these models use subword tokenization, splitting rare words into pieces, which lets a fixed vocabulary handle any word it meets.

Two Families: BERT and GPT

Most LLMs are one of two kinds, distinguished by how they read text and what they are trained to do.

BERT is an encoder. It reads the whole input at once, in both directions, and is pretrained by masking out random words and learning to fill them in. Seeing both the left and right context makes it excellent at understanding tasks, so BERT and its relatives are what you fine-tune for the jobs from earlier in this series: text classification, named entity recognition, and question answering. Fine-tuning a pretrained BERT is now the standard, best-performing way to do those tasks.

GPT is a decoder. It reads left to right and is pretrained to predict the next word over and over. That simple objective, at scale, produces models that generate fluent text and can be steered by a prompt. The GPT family powers text generation, summarisation, translation, and conversational assistants. A rough rule of thumb: reach for a BERT-style model when you need to understand or label text, and a GPT-style model when you need to generate it. There are also encoder-decoder models like T5 that frame every task as text-in, text-out.

Two Ways to Use Them

How you put an LLM to work depends on the family. With BERT-style models you fine-tune: you attach a small task-specific head and train briefly on your labelled data, adapting the model to your problem. With large GPT-style models you often do not train at all; you prompt. You describe the task in plain language, optionally with a few examples in the prompt, and the model performs it, an ability called in-context learning that emerges only at large scale. The most familiar GPT-style assistants are further refined to follow instructions and align with human preferences, which is what makes them feel conversational.

Using a Pretrained Model in a Few Lines

The Hugging Face Transformers library makes both families usable almost trivially. Its high-level pipeline downloads a suitable pretrained model and runs it:

from transformers import pipeline

# Sentiment with a fine-tuned BERT-style model (an "understanding" task)
classifier = pipeline("sentiment-analysis")
print(classifier("This series finally made attention click for me."))
# [{'label': 'POSITIVE', 'score': 0.9994}]

# Fill in a masked word, BERT's pretraining objective in action
unmasker = pipeline("fill-mask", model="bert-base-uncased")
print(unmasker("NLP is a field of [MASK] intelligence.")[0]["token_str"])
# artificial

Notice how the sentiment classifier here, a fine-tuned Transformer, will beat the TF-IDF and from-scratch LSTM classifiers from earlier chapters with no feature engineering at all. That is the practical payoff of the whole progression.

What They Cannot Do

LLMs are powerful, not magic, and using them well means knowing the limits. They hallucinate, producing fluent, confident text that is simply wrong, because they are trained to sound plausible, not to be correct. Their knowledge is frozen at training time, so they do not know recent events. They have a context window, a maximum amount of text they can consider at once. They can reflect biases present in their training data. And the largest ones are expensive to run. The most important of these for builders is the combination of hallucination and frozen knowledge, and the standard fix is to give the model relevant, current information at query time rather than relying on its memory. That technique, retrieval-augmented generation, is the subject of the RAG field manual, and it is the natural next step once you start building real applications.

Key Takeaways

• An LLM is a Transformer pretrained on huge amounts of text, then adapted to tasks.
• Pretrain-then-adapt is transfer learning at scale, and it is why small teams can get strong results.
• BERT-style encoders are best for understanding tasks; you fine-tune them for classification, NER, and QA.
• GPT-style decoders are best for generation; you often prompt them rather than train them.
• Hugging Face pipelines let you use both families in a few lines.
• LLMs hallucinate, have frozen knowledge and a limited context window, which retrieval-augmented generation helps address.

What's Next

We now have the full toolkit, from preprocessing to Transformers. The final part of the series puts it to use, starting with one of the most common applications people want to build: a chatbot. We will look at how conversational systems are designed and where modern LLMs fit in.