Named Entity Recognition and Sequence Labeling

Named Entity Recognition and Sequence Labeling

With the foundations in place, we can start solving real problems. This part of the series covers the core NLP tasks, and we begin with one of the most useful in practice: named entity recognition, or NER. NER is about finding and labelling the specific things mentioned in text, the people, places, organisations, dates, and amounts. It powers resume parsers, news aggregators, search engines, and the first step of many information-extraction pipelines.

What NER Actually Does

Given a sentence, NER picks out spans of text and assigns each a category. Take this sentence: "Tim Cook announced that Apple will open a new office in Bangalore in March 2026." A NER system should produce something like Tim Cook as a PERSON, Apple as an ORGANISATION, Bangalore as a LOCATION, and March 2026 as a DATE. The common entity types are person, organisation, location, date, time, money, and percentage, though you can train a system for any categories you like, such as drug names in medical text or part numbers in manufacturing.

The reason this is valuable is that it converts unstructured text into structured data you can actually query. Once you have extracted entities from ten thousand news articles, you can ask which companies were mentioned alongside which people, something impossible with raw text.

Why It Is Different From Classification

The text classification we cover in the next chapters assigns one label to a whole document. NER is different: it assigns a label to each token, and it has to get the boundaries right too, since "New York Times" is one organisation, not a location followed by a word. This kind of task is called sequence labeling, because the model walks along the sequence of tokens and tags each one. Part-of-speech tagging, which labels each word as a noun, verb, adjective, and so on, is another classic sequence labeling task that works the same way.

The BIO Tagging Scheme

To turn "label each entity span" into "label each token," NER uses a tagging scheme, most commonly BIO. Each token gets a tag: B-TYPE marks the beginning of an entity, I-TYPE marks a token inside the same entity, and O marks a token that is outside any entity. So "Tim Cook" becomes B-PER then I-PER, and ordinary words like "announced" get O. This simple scheme lets a token-level model represent multi-word entities and their boundaries.

Token        Tag
---------    -------
Tim          B-PER
Cook         I-PER
announced    O
that         O
Apple        B-ORG
will         O
open         O
...          O
Bangalore    B-LOC
in           O
March        B-DATE
2026         I-DATE

How NER Has Been Done

The approaches mirror the history of NLP as a whole. The earliest systems were rule-based, using dictionaries of known names (called gazetteers) and hand-written patterns like "a capitalised word following Mr. is a person." These are precise in narrow domains but brittle. Next came classical machine learning, where a model called a Conditional Random Field learned to tag tokens from features you designed, such as capitalisation, suffixes, and the surrounding words. CRFs were the workhorse for years because they model the fact that tags depend on neighbouring tags. Then came deep learning, where a BiLSTM, a bidirectional version of the recurrent networks from the Neural Networks series, reads the sentence in both directions and feeds a CRF layer, removing the need to hand-design features. Today the best results come from fine-tuned Transformers, which treat NER as token classification on top of a model like BERT, a path we reach in the chapter on large language models.

NER in Practice With spaCy

You do not have to build any of this yourself to get good results. The spaCy library ships with strong pretrained NER models and makes extraction a few lines:

import spacy

# python -m spacy download en_core_web_sm
nlp = spacy.load("en_core_web_sm")

doc = nlp("Tim Cook announced that Apple will open a new office "
          "in Bangalore in March 2026.")

for ent in doc.ents:
    print(f"{ent.text:<15} {ent.label_}")
# Tim Cook        PERSON
# Apple           ORG
# Bangalore       GPE
# March 2026      DATE

For many projects this is all you need. You reach for a custom-trained model only when your entities are domain-specific, such as gene names or legal clause types, that a general model was never trained to find.

Measuring How Well It Works

NER is evaluated with precision, recall, and their combination, the F1 score, measured at the level of whole entities rather than individual tokens. Precision asks what fraction of the entities the model found were correct, and recall asks what fraction of the real entities it managed to find. An entity usually only counts as correct if both its span and its type match exactly, which is a strict and sensible bar. These same metrics appear again for classification in the next chapter, so they are worth getting comfortable with.

Where It Gets Hard

• Ambiguity: "Washington" can be a person, a city, or a state, and only context decides.
• Boundaries: deciding whether "Bank of India" is one entity or two.
• Unseen entities: new company and product names appear constantly, so a model must generalise beyond what it saw in training.
• Domain shift: a model trained on news often does poorly on medical or legal text without retraining.

Key Takeaways

• NER finds and categorises entities like people, organisations, locations, and dates in text.
• It is a sequence labeling task: every token gets a tag, unlike document classification.
• The BIO scheme tags tokens as Beginning, Inside, or Outside an entity to capture multi-word spans.
• Approaches went from rules to CRFs to BiLSTM-CRFs to fine-tuned Transformers.
• spaCy gives strong NER out of the box; train custom models only for specialised domains.
• Evaluation uses entity-level precision, recall, and F1.

What's Next

NER labels pieces within a document. The next chapter tackles the most common task that labels the document as a whole: text classification, using sentiment analysis as the running example, and starting with the classical methods that still work remarkably well.