Hugging Face Local Models for OpenClaw PR Triage

Hugging Face has implemented a real-time notification and triage system for the OpenClaw repository using local open-weight models. By deploying models such as Gemma 4 and Qwen 3.6 within an agent harness, the team achieved high-throughput classification of issues and pull requests (PRs) without relying on expensive closed-model APIs or sacrificing real-time responsiveness for batch processing.

Agentic Classification Architecture

The system utilizes "agentic classification," where a model is not simply given a prompt but can actively seek additional context before returning a structured result. The core components include:

• Agent Harness: The system uses Pi as a harness to call local model endpoints.
• Restricted Shell (reposhell): To prevent prompt injection attacks from malicious PRs or issues, the agent does not have full bash access. Instead, it uses reposhell, a restricted shell that only allows read-only operations (e.g., ls, find, cat, grep) on the OpenClaw repository.
• Structured Output: The agent uses a final_json tool to submit the final classification labels based on a predefined set of categories (e.g., local_models, self_hosted_inference, agent_runtime).

Orchestration Pipeline

The workflow from a GitHub event to a Discord notification follows a semi-agentic path to maximize speed and resource efficiency:

1. Mirroring: openclaw/gitcrawl mirrors the repository locally, normalizing new PRs and issues into a SQLite database.
2. Context Building: A worker creates a GitHub context object containing the title, body, labels, and diff excerpts, reducing the need for the model to browse GitHub directly.
3. Inference: The localpager-agent processes the prompt. The agent may use reposhell to inspect the codebase for better accuracy before outputting the final JSON classification.
4. Notification: Deterministic rules route the classified item to Discord based on user-defined notification policies.

Model Performance and Benchmarks

Testing was conducted on a 330-row evaluation set where labels were validated by a consensus of GPT-5.5 and Opus 4.8. The models were run on an NVIDIA GB10 with 128 GB of unified memory.

Metric	gemma-4-26b-a4b	qwen3.6-35b-a3b	DeepSeek-V4-Flash
Precision	0.716 ± 0.010	0.831 ± 0.007	0.938
Recall	0.905 ± 0.004	0.818 ± 0.006	0.714
F1	0.800 ± 0.008	0.824 ± 0.002	0.811
Exact Match	0.410 ± 0.014	0.540 ± 0.014	0.509
False Positives	227.0 ± 10.5	105.7 ± 6.4	30
False Negatives	60.0 ± 2.6	115.3 ± 4.0	181
Wall seconds / row	1.41 ± 0.04	13.51 ± 0.79	144.14
Output tok/s (Agg)	402.6	145.3	13
Total Parameters	26B	35B	284B
Active Parameters	4B	3B	13B

Key Findings:

• Gemma 4 (26b-a4b): Demonstrated the highest recall and the fastest wall-clock time per row. It achieved over 700 aggregate output tokens per second in separate probes using vLLM with NVFP4 quantization.
• Qwen 3.6 (35b-a3b): Provided higher precision and fewer false positives than Gemma.
• DeepSeek-V4-Flash: While having the lowest false positive rate, its size and low throughput (13 tokens/s) made it impractical for real-time local execution on the specified hardware.

Validation and Real-Time Auditing

To validate the local system, Hugging Face runs a parallel audit loop using GPT-5.5 via OpenClaw every two hours. This SOTA model acts as a judge to identify false positives and negatives produced by the local models. This audit process costs approximately $9/month and serves as a calibration mechanism to ensure the local classifier remains reliable.

Broader Implications for High-Throughput Triage

This implementation demonstrates that medium-sized local models can perform zero-shot classification with sufficient accuracy to replace expensive cloud APIs for high-volume filtering tasks. The team suggests this "agentic classification" approach—combining fast local models with restricted tool access—can be applied to other domains, including news categorization, customer support ticket triaging, and content moderation appeals.