Vision transformers for medical images: patches, attention, and when CNNs still help

A vision transformer divides an image into patches, embeds each patch, and lets self-attention mix information across the whole field. That global view is useful when a finding is not a small local blob: a lesion may be defined by its relation to surrounding tissue, or a smear may need context from distant cells.

Pure transformers are data-hungry. Hospital datasets are often small, class-imbalanced, and noisy. In that setting I prefer residual and channel-boosted hybrids: a convolutional stem to gather local structure, then attention for longer-range dependencies. My published work on ARiViT addresses noisy brain images; ReLViT with YOLO is used for acute myeloid leukemia on bone marrow images. Related transformer lines in my portfolio, including CBViT, remain at the status recorded on the research pages and are not treated as published here.

Attention maps are not automatic explanations, but they are a useful sanity check. If the model attends to a ruler, a label, or a corner artefact, the pipeline is wrong even if the accuracy number looks high. For detection tasks, pairing a transformer classifier with an object detector such as YOLO can separate “what is this field” from “where is the blast cell”.

Students often ask whether ViTs have replaced CNNs. For medical imaging the honest answer is that the two families are tools. Use convolutions where local filters are efficient, attention where context is the bottleneck, and always report the status of each paper rather than collapsing everything into a single “transformer result”.