A 10-Year-Old CNN Beat GPT-6 Astra and Claude Fable 5.1 at Diagnosing Skin Lesions, 62% to 42%. The Lesson for Your Product Is Not About Medicine.
Dermatologist Magnus Lynch gave 200 biopsy-proven skin lesions to GPT-6 Astra, Claude Fable 5.1 and a fine-tuned Inception v3 CNN, an architecture that dates to 2015, and asked each for the most likely of six diagnoses. The CNN got 125 right (62%). Both frontier models got 84 and 85 (42%). The post hit the Hacker News front page with over 650 points, and the discussion mostly argued about medicine. I think the more useful argument is about your product.
What the test did
Each case had a clinical photo and a dermoscopic image. The 200 lesions came from the MILK10k dataset: 100 that should not be discharged (30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas, 20 actinic keratoses) and 100 safe to discharge (50 naevi, 50 seborrhoeic keratoses). The CNN was trained on about 68,000 images from the ISIC Archive, with the test lesions excluded. The LLMs got one default-settings run each.
Within the top two choices, the CNN was right 173 times, GPT-6 Astra 134 and Fable 5.1 138. On melanoma named first, the CNN got 13 of 30, GPT-6 Astra 8 and Fable 5.1 11. The CNN was clearly ahead on benign lesions, and the LLMs did better on actinic keratosis.
The caveats are the real lesson
The author lists his own limits, and they are worth reading. The sample is small, especially the 30 melanomas. Each LLM ran once. Every lesion had been biopsied, so these are harder than typical cases, and on ordinary archive images the CNN does much better (AUROC 0.96 against 0.86 here). The LLMs may also have seen the public dataset during training. And Inception v3 is a baseline, not a state-of-the-art model.
The safety numbers are where it gets interesting. At its preset threshold, the CNN kept 77 of 100 lesions that needed follow-up and discharged 18 of 30 melanomas. Tuned to match the LLMs' caution, it kept 98 or 99 of the unsafe lesions but discharged only 1 or 2 melanomas. The LLMs kept 96 or 97 unsafe lesions and discharged 3 or 4 melanomas. In other words, the CNN gives you a dial. You choose how many false alarms you accept to catch more real cases, and you can see the tradeoff.
The LLMs, asked patient-style questions, advised seeing a doctor in almost every case, including harmless lesions. That is a defensible default for a chatbot. It is not a product.
What this means for a solo builder
Most AI features in small products are classification in disguise: is this ticket urgent, is this image acceptable, is this lead real, which of six categories is this expense. For those, an LLM call is the easy first version, and it is often fine. This test is a reminder that it is a starting point rather than an endpoint.
A small model trained on your own labeled examples has three properties an LLM does not. You can tune its threshold and measure what you trade. It costs a fraction per call and runs in milliseconds. And its behavior does not change when a vendor ships a new version. If you collect 1,000 labeled examples of the thing your product decides, you probably have the data to beat a prompt.
Where this could be wrong
One small test, run once per LLM, on a dataset the LLMs may have trained on. A better prompt, a few examples or a stronger model setting could close some of the gap, and the author does not claim otherwise. Medicine is also unusual: images, a closed label set and lots of public training data. A task with messy text and open-ended answers is where LLMs earn their keep.
What I'd actually do
Pick the one LLM call in your product that returns a label from a fixed list. Log its inputs and the outputs you end up accepting for a month. If you reach a thousand or so examples, train a small classifier on them and compare it against the LLM on a held-out slice, looking at cost and error type as well as accuracy. Keep the LLM for the cases the small model is unsure about. That hybrid is cheap to build and easy to defend.
Author
Lukas
@lukcombinator