Building an OpenGL ES 3.0 Linux GPU Driver for the M4 Mac Mini in One Month

TL;DR

Cody Ho and Niklas delivered a working OpenGL ES 3.0 Linux driver for the M4 Mac Mini and MacBook Neo in roughly one month, proving that large‑language‑model‑assisted reverse engineering can replace years‑long manual effort.

Project Overview

  • Goal: Create a conformant OpenGL ES 3.0 driver (future Vulkan) for Apple‑silicon GPUs (M4, A18 Pro, M5) on Linux.
  • Timeline: ~4 weeks of intensive work, far shorter than the typical multi‑year effort for a new GPU driver.
  • Key Results: Chrome/Firefox WebGL demos run, Minecraft reaches 200 fps, and a complete Linux kernel driver plus user‑space stack were produced.
  • Methodology: Clean‑room reverse engineering using a hypervisor to capture hardware traces, combined with LLMs (Codex, GPT‑5.6 Sol, GPT‑6 Astra) for ABI discovery, code generation, and systematic debugging.

Reverse‑Engineering the Firmware ABI (Kernel Space)

  • Apple’s Model: The GPU runs a custom RTOS (RTKit) and exposes a shared‑memory ABI rather than direct hardware registers.
  • Complexity: The A18 Pro ABI has ~1.5× more structs and twice the pointer count of the M1/M2 ABI, making it substantially harder to decode.
  • Approach:
    1. Live Probing: Use the hypervisor to observe macOS GPU activity, capture the entire GPU memory state at the first "kick" after firmware start.
    2. Replay & Reduce: Iteratively trim the captured state until only the minimal necessary data remains, forcing the LLM to infer struct layouts.
    3. Targeted Captures: For compute work, boot into single‑user mode, launch a tiny Metal compute program early, and capture its trace.
    4. Partial Renders: Generate synthetic workloads that trigger TVB (Tiled Vertex Buffer) overflow, then replay and learn the save‑and‑resume protocol.
  • Outcome: A full description of the AGX firmware ABI, published in the agx-re repositories, enabling a Linux kernel driver that talks to RTKit.

Building the Linux Kernel Driver

  • From Prototype to Production: A Python prototype was rewritten in Rust over three days, following the existing drm‑shim design.
  • Key Steps:
    1. Port drm‑shim to Rust for a synchronous driver.
    2. Convert the frontend to asynchronous while keeping GPU submission synchronous.
    3. Replace polling with event‑driven fences tied to firmware notifications.
    4. Add low‑level optimizations such as batched submissions.
  • LLM Role: Codex generated most of the Rust scaffolding and aggressively used hypervisor snapshots to debug mismatches, dramatically accelerating development.

User‑Space Stack (Mesa Integration)

  • Hardware vs. Mesa First: Two strategies were explored:
    • Hardware‑First (Cody): Exhaustively reverse‑engineer instruction semantics, then write a spec for the LLM to implement.
    • Mesa‑First (Niklas): Incrementally build Mesa drivers, using RE only when required for missing functionality.
  • Result: Niklas’s Mesa‑first approach progressed faster because it kept the LLM focused on concrete driver goals.
  • Key Components:
    • IR/Shader Compiler: Custom compiler from Mesa’s NIR to AGX ISA, reusable for Vulkan.
    • Command Stream Builder: Constructs the buffers that the firmware consumes.
    • Feature Discovery: Identified undocumented hardware capabilities, e.g., a native 64‑bit add, 128× anisotropy, new matrix unit mode, and 7‑bit immediates for uniform_mov.
  • Conformance: OpenGL ES 3.0 CTS passes with only optional extensions missing.

Deliverables

Remaining Work & Upstreaming Challenges

  • Feature Roadmap: Vulkan 1.4, OpenGL 4.6, OpenGL ES 3.2, OpenCL 3.1, Direct3D 12 (via Proton), and ray tracing.
  • Upstream Barriers: The Asahi Linux project enforces a strict no‑AI policy; the M1/M2 driver is not yet upstream, so the M4 driver must wait for that precedent.
  • Legal & Community Concerns: Some commenters question the legality of clean‑room RE when LLMs may have been trained on proprietary binaries, and note the author’s former Apple employment.
  • Human Review Needed: Extensive testing, code review, and refactoring are required before any upstream submission.

Community Reaction (Hacker News Highlights)

"It's extremely impressive that they were able to make a working driver so quickly. I think this is one of the best use cases for LLMs." – ndiddy

"All this work is tainted since the poster is ex‑Apple. There is no way Linux will take that code..." – thrwy19940314

"The biggest pain point of Asahi Linux is lack of GPU acceleration on M3 and newer. Asahi’s no‑AI policy means this work can’t be upstreamed." – porphyra

"Someone can now white‑box reimplement this if they are concerned." – getcrunk

"Fact that a person who was not previously driver developer can achieve this in a few weeks is pure wonder. Leave legal questions to the Linux Foundation lawyers." – SXX

Lessons Learned

  1. LLM‑Driven RE Works: Large language models can autonomously replay captured GPU state, infer struct layouts, and generate driver code, dramatically shortening the reverse‑engineering cycle.
  2. Capture Early, Keep Small: The most reliable captures are taken immediately after firmware start and contain the minimal workload (e.g., a tiny compute kernel).
  3. Task Prioritization Matters: Directing the LLM to the simplest missing feature (compute) before tackling harder ones (partial renders) yields faster overall progress.
  4. Clean‑Room Discipline: The team avoided Apple binaries, treated any required blobs as opaque, and published all traces, preserving a verifiable clean‑room process.
  5. Community Policies Influence Adoption: Projects with strict anti‑AI stances (e.g., Asahi Linux) may reject LLM‑generated code, limiting upstream potential despite technical merit.

Future Directions

  • Broader Hardware Support: Apply the same methodology to M5 and future Apple‑silicon generations; the user‑space code appears largely portable.
  • Vulkan Front‑End: Leverage the existing NIR‑to‑AGX compiler to implement a Vulkan driver, enabling modern graphics APIs on Linux.
  • Machine‑Learning Workloads: Investigate integration with PyTorch or Metal Performance Shaders to expose the GPU for AI workloads.
  • Open‑Source Governance: Engage with the Linux Foundation and Asahi maintainers to define a policy for LLM‑assisted driver contributions.

This article is based on Cody Ho’s blog post “I Came, I Prompted, I Left Part 2: Building a GPU Driver From Scratch in One Month” and the top‑voted Hacker News comments.

Sources

Related