tiiuae/Falcon-Perception

Falcon‑Perception – Multimodal Vision‑Language Model

What it is – A lightweight, open‑source PyTorch (and Apple‑Silicon MLX) inference library for Falcon Perception, a dense, autoregressive transformer that can detect objects, segment instances, or run OCR from a natural‑language query. The repo ships the model weights (via Hugging Face), a set of highly‑optimized inference engines, a REST server, and several notebooks that demonstrate how to use the model for vision‑language tasks.

Why it matters – Falcon Perception unifies three classic computer‑vision problems—object detection, instance segmentation, and optical‑character‑recognition—under a single language‑conditioned model. The 270 M‑parameter Falcon OCR 1.5 variant delivers end‑to‑end OCR quality comparable to much larger VLMs while staying ~3× smaller, making it practical for real‑time or edge deployment.

---

Key components

Component	What it does
PyTorch Paged Inference Engine	Uses FlexAttention, CUDA‑graph capture and a virtual‑page KV cache to achieve continuous batching with minimal latency (≈ 100 ms pre‑fill, 200 ms up‑sample, 50 ms decode on H100).
Paged OCR Engine	Adds a lightweight layout detector (PP‑DocLayoutV3) and per‑region OCR, still using the same paged KV cache for high throughput.
Batch Engine	Simple left‑padded batch inference useful for debugging or reproducing the training forward pass.
MLX Engine	Same model converted to the MLX framework, enabling batch inference on Apple‑Silicon Macs without any PyTorch/transformers dependencies.
Inference Server	FastAPI‑based REST service that spins up one engine per GPU (data‑parallel) and exposes a Streamlit UI for interactive demo.
vLLM Docker image	Pre‑built container for Falcon‑OCR‑only serving with OpenAI‑compatible API, supporting both end‑to‑end and layout‑plus‑OCR pipelines.
Notebooks	Step‑by‑step Colab notebooks for perception, OCR, a perception‑agent (VLM‑orchestrated reasoning), and open‑vocab multi‑object tracking.

---

Quick start (from source)

# Clone and install (auto‑detect backend)
git clone https://github.com/tiiuae/Falcon-Perception.git
cd Falcon-Perception
pip install -e .               # PyTorch on Linux, MLX on macOS
# or explicitly choose a backend
pip install -e "[torch]"      # CUDA GPU
pip install -e "[mlx]"        # Apple Silicon

The package includes optional extras: ocr (adds layout detector), dev (tensorboard, matplotlib, ipykernel), server (FastAPI & Uvicorn).

---

Running the model

1. Perception (detection / segmentation)

# GPU example – downloads model & sample image automatically
python demo/perception_single.py
# Custom image / query
python demo/perception_single.py --image myphoto.jpg --query "cat on the left"
# Only bounding boxes (no masks)
python demo/perception_single.py --task detection

2. OCR (text extraction)

# End‑to‑end OCR (default, works for most documents)
python demo/ocr_single.py --image doc.png
# Layout‑aware OCR (for dense multi‑column pages)
python demo/ocr_single.py --task ocr_layout   # requires the [ocr] extra

3. Batch / Paged processing

# Benchmark on the PBench dataset (50 samples streamed from HF)
python demo/perception_benchmark.py
# Full OCRBench‑v2 benchmark
python demo/ocr_benchmark.py

4. Server & UI

# Start the multi‑GPU REST server
python -m falcon_perception.server --config.num-gpus 2 --config.port 7680
# In another terminal, launch the Streamlit demo UI
streamlit run demo/streamlit_app.py

The UI lets you upload images, choose a task, and visualises bounding boxes, masks, or OCR output with timing info.

---

Deploying Falcon‑OCR with vLLM (Docker)

# Pull the official image and run on one or two GPUs
docker run -d --name falcon-ocr \
  --gpus '"device=0,1"' \
  -e EXPOSED_GPU_IDS=0,1 -e VLLM_GPU=0 -e PIPELINE_GPU=1 \
  -p 8000:8000 -p 5002:5002 ghcr.io/tiiuae/falcon-ocr:latest

API example (end‑to‑end OCR)

curl -X POST http://localhost:5002/falconocr/parse \
  -H "Content-Type: application/json" \
  -d '{"images": ["data:image/jpeg;base64,<...>"], "skip_layout": true}'

The container reports ~5,800 tokens / s and ~2.9 images / s on an A100‑80GB when running the full layout + OCR pipeline.

---

Who should use this?

• Researchers exploring multimodal VLMs that need a compact, open‑source baseline for detection, segmentation, or OCR.
• Developers building applications that require language‑conditioned visual reasoning (e.g., “segment the cat on the left”).
• Deployers looking for a production‑ready server (FastAPI or vLLM Docker) that can serve OCR at high throughput on a single GPU.
• Apple‑Silicon users who want to run the same model on Macs without installing PyTorch.

---

Citation

If you use Falcon‑Perception in a publication, cite the arXiv tech report:

@article{bevli2026falcon,
  title   = {Falcon Perception},
  author  = {Bevli, Aviraj and Chaybouti, Sofian and Dahou, Yasser and Hacid, Hakim and Huynh, Ngoc Dung and Le Khac, Phuc H. and Narayan, Sanath and Para, Wamiq Reyaz and Singh, Ankit},
  journal = {arXiv preprint arXiv:2603.27365},
  year    = {2026},
  url     = {https://arxiv.org/abs/2603.27365}
}

---

Acknowledgments

The project builds on work from PyTorch Titan, Flex‑Attention, Moondream, AnyUp, and several open‑source vision libraries (Roboflow trackers, etc.).