Hugging Face transformers-to-mlx Skill and Test Harness

Hugging Face transformers-to-mlx Skill and Test Harness

Hugging Face has introduced a Skill and a test harness designed to accelerate the porting of language models from the transformers library to mlx-lm. This system ensures that models added to transformers can be made available in MLX almost immediately, while providing the rigorous verification and documentation necessary to support human maintainers in the age of AI-generated code.

Solving the Agent-Generated PR Bottleneck

While code agents can now generate functional solutions from brief specifications, they often lack the implicit context of large-scale libraries like transformers. Agent-generated pull requests (PRs) frequently fail because they prioritize generic "best practices" over specific library design philosophies—such as the preference for flat hierarchies and human-readable model files—and may introduce subtle bugs or performance regressions.

To address this, Hugging Face developed a system that uses agents as aides for contributors rather than full automation. By treating the transformers library as the "source of truth" for model definitions, the system limits the agent's scope, ensuring that ports to mlx-lm are grounded in a verified implementation.

The transformers-to-mlx Skill

The Skill is a recipe—a text file containing detailed guidelines—that steers a coding agent (tested with Claude Code) through the complex process of model conversion.

Capabilities for Contributors

For the person porting the model, the Skill automates the scaffolding and technical verification:

  • Environment Setup: Manages virtual environments, downloads relevant models from the Hub via the hf CLI, and sets up editable installs of mlx-lm and transformers.
  • Technical Analysis: Diffs configurations to identify varying parameters across model variants and infers dtypes from safetensors metadata when not explicitly declared in the config.
  • Critical Bug Detection: Specifically checks for sensitive areas like RoPE (Rotary Positional Embedding) configurations and float32 precision contamination, which can silently degrade inference speed or long-sequence performance.
  • Iterative Debugging: Runs per-layer comparisons between transformers and MLX to pinpoint exactly where numerical divergence occurs, iterating until the results are satisfactory.

Support for Reviewers

To reduce the burden on maintainers, the Skill generates PRs that mimic careful human submissions:

  • Convention Adherence: The agent is instructed to use idiomatic MLX solutions, avoid unnecessary comments, and refrain from proposing unsolicited refactors of shared utilities.
  • High-Signal Reporting: PR bodies include a comprehensive report featuring summary variants, architectural differences, generation examples, numerical comparisons, and dtype verification.
  • Transparency: All agent-assisted PRs are explicitly disclosed as such and require contributor acceptance before being opened.

Non-Agentic Test Harness

To eliminate the risk of LLM hallucinations or complacency in the reported results, Hugging Face created a separate, non-agentic test harness. This tool provides a reproducible verification layer:

  • Systematic Testing: Runs a battery of tests on the converted code to ensure reproducibility.
  • Detailed Artifacts: Saves results as summary reports, per-model details, and raw input/output JSON files.
  • Human Judgment: The harness is not a CI gate; it provides the signal (e.g., relative logits difference) that allows human reviewers and contributors to make informed judgment calls on whether a port is acceptable.

Usage and Implementation

The Skill is intended for contributors who are prepared to engage in the iterative review process of mlx-lm. It can be installed via the following commands:

uv run https://raw.githubusercontent.com/huggingface/transformers-to-mlx/main/install_skill.py
uvx hf skills add --claude

Current Limitations and Future Roadmap

Known Shortcomings

  • Shared Utilities: The Skill favors self-contained model files (following transformers style), whereas mlx-lm reviewers often prefer refactoring repeated code into shared modules.
  • Architecture Gaps: It does not yet support Vision-Language Models (VLMs) or other non-LLM architectures.
  • Quantization: While it tests quantization, it does not automatically upload quantized models to the Hub.
  • Thinking Models: There are currently no specific tests to validate the thinking structure of "reasoning" models.

Next Steps

  • Expansion to mlx-vlm: Collaborating to support vision-language models, which require additional image pre-processing processors.
  • llama.cpp Integration: Exploring how similar scoped agents can help replicate processors in C++.
  • Harness Automation: Expanding the test battery and exploring safe automation for running tests on Hugging Face infrastructure.

Sources