Fine-Tuning Qwen 3:0.6B for Question Categorization

Fine-tuning a tiny local LLM can transform a poor-performing general model into a highly accurate specialized classifier. In a recent experiment, the Qwen 3:0.6B model was fine-tuned to categorize household-related questions for a metadata-aware RAG (Retrieval-Augmented Generation) system, increasing accuracy from a baseline of 10% to over 91%.

Using Metadata-Aware RAG for Better Retrieval

Categorizing questions before they reach a vector database allows a system to narrow the search space for vector ranking. By mapping a question like "When did we replace our pool pump?" to a specific category (e.g., "pool"), the system can filter indexed entries to only those matching that category, reducing noise and improving the precision of the retrieved context.

Baseline Performance: The Failure of Zero-Shot Prompting

For a model as small as Qwen 3:0.6B, zero-shot prompting is insufficient for reliable classification. In a baseline test using 131 integration tests, the model achieved only 9.92% accuracy.

Common failure modes in the baseline model included:

  • Label Overuse: The model frequently defaulted to broad labels like "electric" or "appliances" while ignoring specific categories like "pool" or "hvac".
  • Hallucinations: The model invented categories not present in the provided list (e.g., returning "apartments" instead of a valid category).

Fine-Tuning Strategy and Results

To improve performance, the model was fine-tuned using the Unsloth framework with QLoRA. The training dataset consisted of approximately 850 entries, split 70/15/15 into training, evaluation, and test sets.

First Attempt: Direct Category Mapping

The first fine-tuning attempt focused on teaching the model to map questions directly to category names. This increased accuracy to 79.39%. However, two primary issues remained:

  1. Syntactic Inconsistency: The model often emitted fragments of the correct category (e.g., "ac" or "air" instead of "hvac").
  2. Semantic Overlap: The model struggled with categories that shared similar meanings, such as "fountain," "water heater," and "pool."

Second Attempt: Opaque ID Mapping

To resolve syntactic errors and reduce semantic confusion, the prompt was modified to map categories to two-character opaque IDs (e.g., "AA" for appliances, "KK" for hvac). This removed the semantic meaning from the output token itself, forcing the model to learn a strict mapping.

This change resulted in a significant performance boost, reaching 91.6% accuracy. The remaining errors were primarily concentrated in "watery" categories, where the model still occasionally confused "water heater" with "pool."

Technical Alternatives and Community Insights

While the experiment successfully demonstrated the utility of a fine-tuned sLLM, technical discussions highlight several alternative approaches for text classification:

Traditional Machine Learning

Some practitioners suggest that for closed-set classification, traditional ML is often more efficient.

"If you are going to go to the bother of fine tuning for trivial problems like subject classification then I think you'll find Scikit Learn with a SGDClassifier on 2-grams will do probably just as well and be under 1MB for the trained classifier."

Encoder-Based Models

Others argue that BERT-based encoder models are better suited for pure classification tasks than decoder-only LLMs:

  • ModernBERT: Using a BERT-based model with a classification head is often faster and more performant for this specific use case.
  • Embedding-Based Classification: A "lazy learner" approach using embedding models (like Stella) to perform kNN via cosine distance between the query and pre-embedded category labels is often a faster baseline to implement.

Advanced Tuning Techniques

For those looking to further refine small models, the community suggested exploring:

  • Zero-shot encoders (e.g., GliNER).
  • GRPO training or GEPA prompt tuning.
  • Synthetic Dataset Generation: Using larger LLMs to generate "hard examples" where multiple categories might be valid, followed by DPO (Direct Preference Optimization) tuning.

Sources