LFM2.5-350M GRPO Fine-tuning Boosts IFStruct Score to 29.7%

TL;DR

Fine‑tuning the 350M‑parameter LFM2.5 model with Group Relative Policy Optimization (GRPO) for only 100 steps and ~500 samples lifts its IFStruct structured‑output compliance from 22.6% to 29.7%, a 7.1‑point gain that narrows the gap to much larger models.

Why Structured‑Output Compliance Matters

Structured output—returning data that is syntactically valid and conforms to a prescribed schema—is a prerequisite for integrating LLMs into downstream systems. Benchmarks that measure raw reasoning often overlook this requirement; IFStruct isolates schema adherence, making it a reliable proxy for real‑world deployability.

"Whether a model reliably returns valid, parseable output in the requested format and shape — schema compliance — is often what decides whether it can be wired into a downstream system at all." – Hugging Face blog

Baseline Evaluation on LFM2.5‑350M

  • Model: LiquidAI/LFM2.5-350M (GGUF BF16) served via llama.cpp.
  • Setup: Free‑tier Colab/Kaggle GPU for serving; evaluation run on a MacBook Pro (Apple M5 Max, 36 GB) using the IFStruct evaluator.
  • Result: 22.6% overall pass rate (452/2000 samples). Detailed breakdown:
    • JSON: 18.0%
    • YAML: 27.2%
    • Wrapper key: 28.5%
    • Bare list: 16.6%
  • Common errors: missing required fields, wrong item counts, type mismatches, and unclosed code blocks.

These numbers closely match the 21.1% reported in the original IFStruct release, confirming the baseline is sound.

GRPO Fine‑tuning Pipeline

The entire workflow is available as a public notebook on GitHub and runs on a free‑tier GPU.

Training Data

  • Source: nvidia/Nemotron-RL-instruction_following-structured_outputs (≈500 samples).
  • Augmentation:
    • 40% of prompts receive a "return the output inside a fenced code block" instruction, teaching the model to obey format directives.
    • 20% are transformed into top‑level‑array tasks, encouraging correct bare‑list generation and item‑count compliance.

Model and LoRA Adapter

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=[
        "q_proj", "k_proj", "v_proj", "out_proj", "in_proj",
        "w1", "w2", "w3",
    ],
)
  • Trains ~6 M parameters (≈1.66% of the 350 M model).

Reward Functions

Three scalar rewards (0‑1) evaluate structural correctness:

  1. json_format_reward – parses output and checks requested form (fenced vs. raw). Full credit for exact form, partial (0.2) for wrong but parseable, zero for unparseable.
  2. field_count_reward – compares the number of top‑level fields to the expected count; linear decay for mismatches.
  3. schema_validation_reward – validates against the provided JSON Schema, penalizing missing required keys and constraint violations.

The combined reward is a weighted sum with reward_weights = [1.0, 0.5, 2.0].

Training Configuration

training_args = GRPOConfig(
    output_dir="./outputs/lfm25-350m-nemotron-schema-grpo",
    learning_rate=5e-5,
    max_steps=100,
    warmup_steps=10,
    num_generations=8,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    steps_per_generation=2,
    max_completion_length=1024,
    mask_truncated_completions=False,
    temperature=1.1,
    beta=0.01,
    reward_weights=[1.0, 0.5, 2.0],
    logging_steps=1,
    save_steps=100,
)
  • 100 optimization steps, 8 sampled completions per prompt group, and a modest KL penalty (beta=0.01).
  • Training completes on a 16 GB GPU in under an hour.

Merging LoRA

After training, the LoRA adapter is merged back into the base weights and saved as a single checkpoint, ready for conversion to GGUF for llama.cpp serving.

Post‑Fine‑Tuning IFStruct Results

  • Model: Merged GRPO‑tuned checkpoint converted to BF16 GGUF.
  • Serving: Same llama.cpp parameters as baseline, different alias and port.
  • Result: 29.7% overall pass rate (594/2000 samples).
    • JSON: 31.9% (↑13.9 pts)
    • YAML: 27.5% (↑0.3 pts)
    • Wrapper key: 29.7% (↑1.2 pts)
    • Bare list: 29.7% (↑13.1 pts)
  • Error profile shifted: fewer missing‑field errors, but still notable type mismatches and extraneous fields.
Metric Baseline GRPO‑tuned Δ
Overall 22.6% 29.7% +7.1
JSON 18.0% 31.9% +13.9
YAML 27.2% 27.5% +0.3
Wrapper key 28.5% 29.7% +1.2
Bare list 16.6% 29.7% +13.1

The improvement is concentrated where the reward signals were targeted—JSON formatting and bare‑list generation—confirming the efficacy of the GRPO approach.

Implications and Takeaways

  • Cost‑effective scaling – A 100‑step GRPO run with ~500 examples costs less than a free‑tier Colab session, yet yields a 7‑point absolute boost in structured‑output compliance.
  • Model‑size parity – The tuned 350 M model approaches the performance of a 2 B‑parameter model (Qwen3.5‑2B scores 33.15% on IFStruct) while remaining far smaller and cheaper to run.
  • Reward design matters – Explicit rewards for format, field count, and schema validation directly translate into higher pass rates for the targeted output forms.
  • Reproducibility – All scripts, data links, and conversion steps are publicly available, enabling researchers to replicate or extend the experiment with other models or datasets.

How to Replicate

  1. Install uv, llama.cpp, and the IFStruct evaluation script as described in the blog.
  2. Clone the notebook repository and run the GRPO fine‑tuning cell with the provided GRPOConfig.
  3. Merge the LoRA adapter, convert the checkpoint to GGUF, and serve with llama-server.
  4. Execute the IFStruct evaluator against the served endpoint.

All commands and URLs are reproduced verbatim in the original Hugging Face post.

Limitations

  • The training data (≈500 samples) is a tiny fraction of the data used for large‑scale RLHF, so gains plateau quickly.
  • Improvements are modest for YAML outputs, indicating that the current reward weighting favours JSON formatting.
  • The benchmark still reflects synthetic tasks; real‑world integration may expose additional edge cases.

Future Directions

  • Expand the training set with more diverse schemas and formats (XML, CSV) to broaden compliance.
  • Experiment with alternative reward weightings or additional constraints (e.g., logical consistency).
  • Apply the same GRPO pipeline to other compact models (e.g., 200 M‑parameter variants) to assess scalability.

References

Sources