Data leakage in medical image splits: the accuracy you cannot trust

Medical datasets are not independent photographs. A patient may contribute many slices, frames, or crops. If you shuffle images at random, the same person leaks into both training and test. A vision transformer is very good at remembering identity: a scanner signature, a crop offset, a bit of anatomy that is not the disease. The test score then measures memorisation, not generalisation.

The split has to be at the patient or study level, not the file level. That is true for CNNs and for transformers. It is also true when you fine-tune a large pretrained backbone. Pretraining on ImageNet does not cancel leakage in your hospital set. Write the grouping rule in the methods, count patients in each split, and refuse a result that cannot name those counts.

Related traps are quieter. Tuning hyperparameters on the test set, using test-set normalisation statistics, or picking the checkpoint with the best test accuracy are all forms of peeking. A validation split that is also grouped by patient is the place for those choices. The test set is opened once.

When I review student work, the first question is not which backbone they used. It is whether two files from the same case could have met on both sides of the split. Hybrid models such as ARiViT and ReLViT are only as credible as that split. A transformer paper that skips the grouping is not a transformer result. It is a bookkeeping error with attention maps.