A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

Performance Comparison: CNN vs. Frontier LLMs

In a controlled diagnostic test, a fine-tuned Inception v3 convolutional neural network (CNN)—an architecture first introduced in 2015—demonstrated superior accuracy in identifying skin cancer compared to two leading frontier LLMs (GPT-6 Astra and Claude Fable 5.1). When presented with 200 biopsy-proven lesions, the CNN correctly identified the diagnosis in 62% of cases, while the LLMs achieved only 42% accuracy. The CNN also proved more effective at distinguishing between lesions that required clinical intervention and those that were safe for discharge.

Methodology and Dataset

The evaluation utilized the MILK10k dataset, which provides paired clinical and dermoscopic images for each lesion. The test set consisted of 200 biopsy-confirmed cases, split evenly between lesions requiring follow-up (melanoma, basal cell carcinoma, squamous cell carcinoma, and actinic keratosis) and those considered benign (moles and seborrhoeic keratoses).

To ensure a fair comparison, the CNN was fine-tuned on 68,011 images from the ISIC Archive, with all test images and patient-related data strictly excluded from the training and validation sets. The LLMs were provided with the same clinical and dermoscopic images and asked to provide a diagnosis, confidence score, and a discharge recommendation without additional clinical context.

Key Findings

• Diagnostic Accuracy: The CNN achieved a 62% success rate for the primary diagnosis, compared to 42% for both LLMs. When considering a second-choice diagnosis, the CNN reached 173/200 correct, while the LLMs reached 134 and 138 respectively.
• Failure Modes: The LLMs exhibited significant bias toward specific diagnoses; for instance, one model frequently misclassified various lesions as basal cell carcinoma, while the other favored actinic keratosis. The CNN showed fewer such systemic biases.
• Clinical Utility: The CNN was more capable of identifying lesions that should not be discharged, whereas the LLMs struggled to consistently differentiate between malignant and benign growths based solely on the provided imagery.

Industry Perspectives on Model Specialization

The results highlight a growing consensus among technical observers: general-purpose LLMs are not inherently optimized for narrow, domain-specific classification tasks. Commenters on the study noted that while LLMs are powerful for reasoning and multi-modal integration, they are often outperformed by dedicated, task-specific architectures like CNNs or XGBoost models when applied to specialized visual or quantitative data.

"It's no surprise that a dramatically simpler model fit to the narrow ~11 label prediction task outperforms an LLM where the same task is a drop in the ocean with regards to its training corpus." — Hacker News user

Experts suggest that the future of medical AI likely involves a hybrid approach. Rather than relying on LLMs for direct diagnosis, these models may serve as orchestrators that leverage specialized, high-accuracy classifiers to perform the actual diagnostic work. This allows the LLM to handle patient communication, data synthesis, and clinical management while delegating image analysis to models purpose-built for that specific modality.