Natural language processing beyond the chat window

Natural language processing is older than chatbots. At its core it is the conversion of text into units a model can count, embed, and compare: characters, subwords, or words, then sequences. That pipeline is what I teach in data-science and analytics modules, because it is the same discipline whether the document is a tweet, an abstract, or a radiology report.

Representation choices change what the model can see. Bag-of-words and TF–IDF remain strong baselines for short, labelled collections. Dense embeddings capture similarity when phrasing varies. Sequence models, including transformers, add order and long-range agreement. None of these steps is optional decoration: if tokenisation splits clinical abbreviations badly, downstream accuracy is theatre.

Domain text is not generic web text. Protein names, drug strings, and imaging codes need a vocabulary that does not smash them into meaningless pieces. Preprocessing—normalising units, expanding a small list of abbreviations, stripping headers—often moves the score more than swapping one fashionable encoder for another.

When I supervise student projects, I ask for a baseline that a colleague could reproduce: a documented split, a simple lexical model, then a neural model, with errors inspected by hand. NLP for healthcare fails quietly if we only report a leaderboard number and never read the mistakes.