Migrating GitHub CI to Hugging Face Jobs

Hugging Face has introduced a method to migrate GitHub Continuous Integration (CI) workflows to Hugging Face Jobs, allowing developers to run GitHub Actions on serverless infrastructure with flexible hardware options. This approach enables the use of GPU-accelerated CI for machine learning projects and, in the case of the Trackio project, reduced CPU job runtimes by approximately 30%.

Overview of Hugging Face Jobs

Hugging Face Jobs allows users to execute commands or scripts on serverless infrastructure using a variety of hardware configurations. Each Job is defined by a command, a Docker image (sourced from Docker Hub or a Hugging Face Space), a hardware flavor (such as CPU, t4-small, or h200 GPU), and optional environment variables and secrets.

For ML libraries, this is particularly advantageous as it allows for the execution of test suites on actual CUDA hardware without the need to maintain permanent, always-on runners.

System Architecture

The integration is powered by huggingface/jobs-actions, a bridge that converts a GitHub Actions job into an ephemeral self-hosted runner within an HF Job. The workflow operates as follows:

1. Trigger: A pull request triggers a GitHub Actions workflow.
2. Queueing: GitHub queues jobs with specific labels (e.g., hf-jobs-cpu-upgrade or hf-jobs-t4-small) and sends a workflow_job.queued webhook to a dispatcher.
3. Dispatching: A dispatcher Space verifies the webhook, generates a short-lived GitHub runner registration token, and starts an HF Job on the specified hardware.
4. Registration: The HF Job boots an ephemeral GitHub Actions runner and registers it using the one-shot token.
5. Execution: GitHub assigns the pending job to the runner, which executes the CI steps, reports the status, and exits.

Implementation Steps

1. Deploy the Dispatcher Space

Users must first duplicate the huggingface/jobs-actions-dispatcher Space. For production CI, the cpu-upgrade hardware flavor is recommended to ensure the dispatcher remains available for webhooks and avoids the sleep cycles associated with cpu-basic.

2. Configure the GitHub App

Through the dispatcher Space, users create and install a GitHub App on their target repository. This App requires permissions to listen for queued workflow jobs and create registration tokens for ephemeral runners. An HF_TOKEN with permissions to launch Jobs must be saved as a secret in the dispatcher Space.

3. Finalize Dispatcher Settings

Users can optionally set an HF_NAMESPACE variable in the Space to bill jobs to a specific Hugging Face user or organization.

4. Update Workflow Configuration

To migrate a job, the runs-on label in the GitHub Actions YAML file is changed from ubuntu-latest to a supported HF Jobs label, such as:

• hf-jobs-cpu-upgrade for CPU tasks.
• hf-jobs-t4-small for GPU tasks.

Performance Results

Using the Trackio project as a benchmark, Hugging Face reported the following performance improvements over the GitHub ubuntu-latest baseline:

Runner Setup	Runtime	Comparison to GitHub Baseline
GitHub ubuntu-latest	1m 40s	Baseline
HF Jobs CPU (Playwright image)	1m 10s	~30% faster (-30s)
HF Jobs GPU (t4-small)	45s	N/A (No GitHub-hosted GPU baseline)

Technical Optimizations and Capabilities

Docker Image Selection

To maximize efficiency, Hugging Face recommends using specialized Docker images rather than generic Ubuntu images to avoid installing system packages during every run. For example, Trackio utilized mcr.microsoft.com/playwright:v1.60.0-jammy for UI tests and nvidia/cuda:12.4.0-runtime-ubuntu22.04 for GPU jobs.

Log Management

HF Jobs provides a CLI-based method for fetching logs (hf jobs logs <job_id>), which is often more efficient for large logs than the GitHub web UI. The jobs-actions bridge mirrors GitHub Actions logs into the HF Job logs for comprehensive debugging.

Volume Mounting

HF Jobs supports mounting volumes, which can be used to quickly load datasets or models from Hugging Face as part of the CI process.