hr98w/jev-visual

An educational Jev-like visual inference experiment on Apple Silicon: shared context, direct candidate scoring, and local visual demos.

What it solves

It addresses the inefficiency of autoregressive structured generation (like generating JSON) when a vision-language model needs to answer multiple questions about a single image. Instead of generating text, it uses a direct-scoring method to quickly determine the most likely answer from a set of candidates.

How it works

The project uses a "shared prefill" strategy on Apple Silicon via MLX. It processes the image and context once, forks the cache, and then batches multiple question suffixes. Rather than letting the model "think" or write out an answer, it reads the model logits to score candidate outputs (such as A/B/C options or yes/no) and assembles the results in code.

Who it’s for

Developers and researchers interested in optimizing vision-language model (VLM) inference on Mac hardware, specifically those looking for alternatives to slow text generation for classification or multiple-choice tasks.

Highlights

  • Optimized for Apple Silicon: Built to run locally on M-series chips using MLX.
  • High-Efficiency Inference: Demonstrates a significant speedup (e.g., 37.30s down to 2.40s for 64 decisions) by using shared-prefix batched scoring instead of independent generation.
  • Direct Scoring: Avoids the unpredictability of structured text generation by scoring candidate tokens directly from logits.
  • Interactive Demos: Includes visual game demos like a simplified Breakout and an AI sorting factory to showcase real-time visual classification.

Related

  • Project
    atopos31/llmio

    LLMIO is a Go-based LLM gateway that provides a unified API, weighted load balancing, and comprehensive observability for managing multiple AI model providers.

  • Project
    sybil-solutions/glm-flash-lite

    A serving stack that runs GLM-5.3-Flash (125 GB) on a single RTX 3090 24 GB by splitting its 12,096 routed experts across VRAM, RAM, and NVMe, delivering up to ~28 tok/s decode with near-exact output.

  • Project
    mattn/go-tflite

    go‑tflite is a Go binding for TensorFlow Lite’s C API, enabling Go programs to load *.tflite* models, run inference, and optionally use hardware delegates like XNNPACK or Edge‑TPU. The project provides pre‑built TensorFlow Lite libraries for major platforms, clear build instructions, and a Binder notebook demo.

  • Project
    flagos-ai/awesome-LLM-driven-kernel-generation

    A curated list of papers, datasets, and tools exploring how large language models and agentic systems can automate the generation and optimization of high-performance computing kernels, organized by research stream and technique.