This spring I ran an independent study on psychiatric case reports with brain imaging. The first results were the kind you want to post straight away. I didn’t, and the reason is the most useful part of the project.
The setup
MultiCaRe is an open dataset of more than 93,000 clinical case reports from PubMed Central, each with narrative text, images and captions. I filtered it to adult psychiatric cases with brain imaging, which gave 1,351 case-image pairs from 4,901 unique cases.
Each case was embedded with CLIP, combining image and text, then reduced with UMAP and clustered with HDBSCAN. Diagnoses were extracted with clinical NLP that handles negation, so “no history of bipolar disorder” doesn’t count as bipolar disorder. Cluster enrichment was tested with chi-square and corrected for multiple comparisons.
The result that looked like a finding
The clusters were clinically coherent. One MRI cluster was strongly enriched for bipolar disorder (odds ratio 6.14, p < 0.001). A PET cluster was enriched for depression with dementia (odds ratio 14.25, p < 0.001).
Read quickly, that says the scans themselves separate psychiatric conditions. That would be a big claim.
Trying to break it
Two experiments decided what the result actually meant.
Ablation. I clustered images alone and text alone. The bipolar separation disappeared with images only. With text only it was stronger than before: 84% bipolar enrichment, p < 0.0001. The structure came from how clinicians wrote about the patients, not from what the scans showed.
External validation. I ran the imaging pipeline on 265 subjects from the CNP dataset, with confirmed diagnoses and no narrative text to lean on. Agreement between clusters and diagnoses was an adjusted Rand index of −0.027. That’s chance.
What I published
The paper reports the second story, not the first. Multimodal case reports are a real resource for phenotyping from clinical narrative. General-purpose image embeddings don’t pick up a detectable psychiatric signal in brain scans, and brain-specific models are needed before imaging can contribute.
It’s a less exciting headline. It’s also the one that will still be true next year.
The habit
The same habit applies to every system I build. A dashboard that looks right, a reconciliation that passes, a model that scores well: each is a claim, and the most useful thing you can do is try to break it before someone else relies on it.
Paper, code and an interactive atlas are on GitHub.