Falcon Perception and Falcon OCR Release

Falcon Perception and Falcon OCR Release

TL;DR

Hugging Face and the Technology Innovation Institute (TII) have released Falcon Perception, a 0.6B-parameter early-fusion Transformer designed for open-vocabulary grounding and segmentation, and Falcon OCR, a 0.3B-parameter model for high-throughput document understanding. By replacing modular pipelines with a single-stack Transformer architecture, these models achieve state-of-the-art performance in compositional grounding and OCR throughput while remaining computationally efficient.

Falcon Perception: Architecture and Design

Falcon Perception utilizes a single early-fusion Transformer backbone that processes image patches and text tokens in a shared parameter space from the first layer, eliminating the need for separate vision backbones and late-fusion decoders.

Hybrid Attention and Sequence Processing

To handle the differing structures of visual and textual data, the model employs a hybrid attention mask:

  • Image tokens use bidirectional attention to build global visual context.
  • Text and task tokens use causal attention, attending to the visual prefix and all preceding text.

Chain-of-Perception Interface

To manage variable-length dense predictions (from zero to hundreds of instances), the model uses a structured autoregressive interface called Chain-of-Perception. Each instance is decomposed into three sequential steps:

  1. <coord>: Predicts the center of the instance.
  2. <size>: Predicts the spatial extent.
  3. <seg>: Produces a single embedding that is dot-producted with upsampled image features to generate a full-resolution binary mask.

Specialized Output Heads

  • Coordinate & Size Heads: Use Fourier feature encoding (random Gaussian projection into high-dimensional sinusoidal space) to overcome spectral bias and improve localization precision. Decoded coordinates are re-injected into the sequence to condition subsequent tokens.
  • Segmentation Head: Performs a dot product between the <seg> token's hidden state and content-aware upsampled image features, removing the need for Hungarian matching or separate mask-query machinery.

PBench: A New Diagnostic Benchmark

TII introduced PBench, a benchmark designed to isolate specific perception failures by categorizing samples by the dominant capability required:

Level Capability Example Prompt
L0 Simple objects "car"
L1 Attributes & subtypes "red car", "broken fence"
L2 OCR-guided identification "Diet Coke bottle", "Nike shoes"
L3 Spatial understanding "car on the left", "third window from left"
L4 Relations & interactions "person holding umbrella", "tallest building"
Dense Crowdedness stress test Hundreds of instances per image

Training Methodology

Multi-Teacher Distillation

Falcon Perception is initialized via distillation from two vision teachers to provide a strong foundation:

  • DINOv3 (ViT-H): Provides local features for segmentation.
  • SigLIP2: Provides language-aligned features for open-vocabulary understanding.

Data Pipeline and Scale

The model was trained on 54M images, 195M positive expressions, and 488M hard negatives. The pipeline included hierarchical clustering via DINOv3, VLM-driven listing, and an ensemble consensus (SAM 3, Qwen3-VL-30B, and Moondream3) requiring an IoU > 0.8 for automatic acceptance.

Three-Stage Training Recipe

  1. In-Context Listing (450 GT): Learns to list scene inventories autoregressively, capturing object co-occurrence.
  2. Task Alignment (225 GT): Modifies the attention mask to simulate independent queries at inference, focusing gradients on presence classification and localization.
  3. Long-Context Finetuning (10 GT): Adapts the model for extreme crowd density by raising the mask limit to 600 per expression.

Performance Results

SA-Co Benchmark

Falcon Perception (0.6B) achieved a 68.0 Macro-F1 on the SA-Co open-vocabulary segmentation benchmark, surpassing SAM 3's 62.3. It showed significant gains in attribute-heavy (+8.2), food & drink (+12.2), and sports equipment (+4.0) splits, though it lags SAM 3 in presence calibration (MCC 0.64 vs 0.82).

PBench Results

Falcon Perception demonstrates a significant advantage over SAM 3 as prompt complexity increases:

Capability SAM 3 Falcon Perception Gap
L0: Simple objects 64.3 65.1 +0.8
L1: Attributes 54.4 63.6 +9.2
L2: OCR-guided 24.6 38.0 +13.4
L3: Spatial 31.6 53.5 +21.9
L4: Relations 33.3 49.1 +15.8
Dense 58.4 72.6 +14.2

Falcon OCR

Falcon OCR is a 0.3B-parameter variant of the early-fusion architecture trained from scratch to optimize for fine-grained glyph recognition and stroke-level discrimination.

Capabilities and Benchmarks

Falcon OCR handles multi-column layouts, mathematical formulas, tables, and handwriting. It achieved 80.3% on olmOCR (within 1.7 points of the top system) and 88.64 on OmniDocBench, leading all models on Multi-Column (87.1%) and Tables (90.3%) tasks.

Throughput and Efficiency

Due to its small size, Falcon OCR offers high serving throughput. On a single A100-80GB using vLLM, it achieves 5,825 tok/s and 2.9 img/s for a full Layout + OCR pipeline.

Inference Stack

The release includes an inference stack based on PyTorch’s FlexAttention, featuring:

  • Paged KV cache and continuous batching to eliminate padding waste.
  • CUDA graph capture for the decode loop.
  • HR feature cache: An LRU cache for upsampled image features to skip expensive upsampling on subsequent queries for the same image.

Sources