Large language models: what they are good at, and what they must not pretend to be

A large language model predicts the next token from a vast training distribution. That is enough to draft a paragraph, outline a lecture, or rephrase a methods section. It is not enough to certify a diagnosis from a scan. Image classifiers and detectors still need labelled pixels; language models still need sources when the answer must be true.

The failure mode that matters in academic and clinical settings is fluent invention. A model can produce a plausible citation, a tidy differential, or a confident dose, none of which existed. Temperature, prompt wording, and “act as an expert” instructions do not create knowledge. They change the style of the guess.

I treat LLMs as writing and teaching assistants, not as oracles. They can help a student see a first structure for a related-work section. They cannot replace reading the paper, checking the status of a result, or running the evaluation on held-out images. For medical imaging, a specialist vision model with a stated dataset and metric remains the right tool.

If an LLM is used near real records, the design questions are boring and essential: what data leave the machine, who logs the prompt, and how a human signs off. Capability demonstrations are easy. Governance is the actual work.