ARahim3/kaggle-tpu-lab

Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

What it solves

It allows users to run large open-source AI models on Kaggle's free TPU v5e-8 hardware without needing a GPU or paying for cloud infrastructure. It transforms these TPU sessions into public endpoints that are compatible with OpenAI and Anthropic APIs, making them usable by coding agents and CLI tools.

How it works

The project provides Kaggle notebooks and kernel scripts that attach public datasets containing model weights and compile caches. It builds a serving engine across eight TPU chips and uses a Cloudflare tunnel to expose the server via a public URL and API key. It supports different engines depending on the model, such as vLLM-TPU for Qwen or a custom JAX engine for GLM-5.3-Flash.

Who it’s for

Developers and AI enthusiasts who want to host powerful LLMs for free using Kaggle's TPU quota to power coding agents like Claude Code, Codex CLI, or opencode.

Highlights

  • Free Hosting: Leverages Kaggle's free TPU v5e-8 quota (approx. 20 hours/week).
  • API Compatibility: Provides endpoints that speak OpenAI and Anthropic APIs.
  • High Performance: Achieves significant throughput (e.g., ~540 tok/s for Qwen3.8-27B with multiple streams).
  • Rapid Deployment: Can go from starting a notebook to having a live URL in about 6 to 22 minutes.
  • Flexible Deployment: Supports both Kaggle notebooks and direct terminal launches via the Kaggle CLI, as well as rented Google Cloud TPU VMs.

Related

  • Dispatch
    openTPU: An AI-Developed Open-Source Inference Accelerator

    openTPU is an open-source AI accelerator featuring RTL, ISA, and a compiler, developed using AI agents to demonstrate the feasibility of AI-driven hardware design for LLM inference.

  • Project
    vllm-project/tpu-inference

    vLLM TPU is an open‑source plugin that lets you run PyTorch or JAX LLMs on Google TPUs using the same vLLM interface. It supports high‑performance features (async scheduling, KV‑cache offload, LoRA, speculative decoding, etc.) and is tested on many top models such as Gemma‑4, Llama‑3.1, and Qwen 3. Recommended TPUs are v7x, v5e, and v6e. The project provides quick‑start docs, a support matrix, and an active community for contributors.

  • Project
    bentoml/OpenLLM

    OpenLLM is a tool for self-hosting open-source LLMs as OpenAI-compatible APIs, providing a simplified workflow for local serving and enterprise cloud deployment.

  • Project
    ztxz16/fastllm

    FastLLM is a C++‑based, PyTorch‑free inference engine for large language models. It supports dense and MoE models, hybrid GPU/CPU/NUMA/disk deployment, multi‑GPU tensor parallelism, speculative decoding (MTP/DSpark/DFlash2), many quantization formats, and offers a Python CLI, web UI, and OpenAI‑compatible API server. Install via `pip install ftllm`, run a model with `ftllm run`, or launch a service with `ftllm server`. Extensive docs cover model‑specific guides, performance tuning, and community support.

  • Dispatch
    AI Frontier Roundup: Open‑Weight Multimodal Models, 1‑Bit Quantization, and Edge Robotics Surge

    Open‑weight multimodal models, 1‑bit quantization, and new robotics hardware are democratizing local AI and embodied agents while the ecosystem builds agentic tooling and infrastructure.