LFM2.5-Encoders Release

LFM2.5-Encoders Release

Liquid AI has released two new general-purpose encoder models, LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, designed for high-efficiency long-context inference. These models enable document-scale NLP tasks to be performed on standard CPU hardware without sacrificing the quality typically associated with much larger models.

High-Performance Long-Context Inference on CPU

LFM2.5-Encoders provide a significant speed advantage over existing encoders when processing long sequences on CPU hardware. At a context window of 8,192 tokens, the LFM2.5-Encoder-230M is approximately 3.7x faster than ModernBERT-base, completing a forward pass in about 28 seconds compared to over 90 seconds for ModernBERT-base.

While ModernBERT-base may lead in speed for short inputs (below 1,000 tokens on Apple GPU), LFM2.5-Encoders become the faster choice as input length increases, taking the lead from approximately 2,000 tokens onward.

Technical Architecture and Training

LFM2.5-Encoders are derived from LFM2 decoder backbones (LFM2.5-230M and LFM2.5-350M) and converted into bidirectional encoders through three primary modifications:

  1. Bidirectional Attention Mask: Tokens can attend to both preceding and succeeding tokens.
  2. Non-causal Short Convolutions: Symmetrical padding is used so convolutions incorporate neighbors from both sides.
  3. Masked Language Modeling (MLM): 30% of tokens are masked during training.

Training occurred in two distinct stages:

  • General Language Competence: A short-context MLM objective on a large web corpus using a 1,024-token context.
  • Long-Context Adaptation: Context was extended to 8,192 tokens using a full data mix to improve factual, legal, and multilingual capabilities.

Benchmark Performance

LFM2.5-Encoders compete effectively against significantly larger models across 17 tasks from GLUE, SuperGLUE, and multilingual classification.

  • LFM2.5-Encoder-350M ranks fourth among 14 tested models, trailing only models that are larger, including one that is nearly 10 times its size (3.5B parameters).
  • LFM2.5-Encoder-230M outperforms ModernBERT-base and all EuroBERT models while maintaining a smaller parameter count.

Practical Applications and Deployment

These encoders are optimized for high-volume understanding tasks that must remain cost-effective and fast, such as:

  • Intent Routers: Scoring prompts against routing lanes in a single pass.
  • Policy Linters: Checking text against company rules at the token level.
  • PII Detection: Identifying personal information across 16 languages.
  • Text Classifiers: General classification of long documents, transcripts, or support threads.

Model Selection Guide

Model Primary Use Case
LFM2.5-Encoder-350M Use when maximum accuracy is the priority.
LFM2.5-Encoder-230M Use for tighter hardware constraints or higher throughput requirements.

Implementation and Usage

LFM2.5-Encoders are open-weight and accessible via the transformers library. Users can employ them for masked-token prediction or attach custom heads for classification, regression, or retrieval tasks. For maximum efficiency on supported GPUs, Flash Attention 2 is recommended.

Sources