Clef 27B Multimodal Decision Model Available on Ollama Cloud

Clef, a 27B multimodal decision model developed by Cloudflare, is now available on Ollama's cloud and can be run locally, allowing users to implement sophisticated decision-making logic across text and image inputs.

Clef Model Overview and Architecture

Clef is a specialized decision model designed to convert a defined state and a schema of typed questions into concrete decisions. It is fine-tuned from the Qwen3.8-27B base model and is distributed under the Apache 2.0 license.

Multimodal and One-Pass Processing

The model supports multimodal input, enabling it to make decisions based on text, JSON, or images, including forms, receipts, screenshots, and photographs. A key architectural feature is its 'one pass' processing; instead of generating a response sequentially word by word, Clef reads the entire input once and simultaneously scores every answer for every question.

Performance Claims

Clef is reported to lead Cloudflare’s Decision Index and outperforms Jev on the majority of its benchmark suite, establishing it as a high-accuracy decision engine.

Practical Applications and Use Cases

Clef is designed to handle structured, high-stakes decision tasks, supporting several specific use cases for developers:

Structured Decisioning

  • Route a Request: Clef can be used to define multiple destinations and determine which applies, returning the chosen destination along with the probability associated with each option.
  • Check a Condition: It supports defining a yes-or-no question and evaluating the provided evidence, returning a true or false decision alongside the probability of that outcome.
  • Apply a Policy: Developers can define specific rules and allowed outcomes, allowing Clef to return a typed decision based on supplied text or images.

Data and Agent Interaction

  • Read a Document: Clef can evaluate a photo of a document page against defined fields, providing field-by-field decisions complete with probabilities.
  • Tool-Call Moderation: The model can serve as a safety layer by checking an agent’s proposed tool call before that action is executed.

Benchmarking and Performance Metrics

Clef demonstrates strong performance across various industry benchmarks, as detailed below.

Jev Decision Index Results

Evaluations scored on the Jev Decision Index show Clef's competitive edge against models like Jev, DiffusionGemma Jev, and Kev 9B. Key performance highlights include:

  • ToolRet · nDCG@10: Clef achieved a score of 69.19, leading the comparison.
  • API-Bank · accuracy: Clef scored 91.93, surpassing the Jev score of 88.19.
  • BANKING77 · macro-F1: Clef achieved 94.20, significantly higher than Jev's 79.74.
  • CLINC150+OOS · macro-F1: Clef scored 97.43, demonstrating high performance in out-of-scope scenarios.
Benchmark Clef Clef-flash Jev DiffusionGemma Jev Kev 9B Laya
BFCL · case exact 98.47 98.76 95.75 96.52 94.51 38.13
ToolRet · nDCG@10 69.19 66.43 65.28 61.21 64.26 12.69
API-Bank · accuracy 91.93 93.11 88.19 83.66 56.30 11.41
Home appliances · case exact 82.95 97.73 52.27 42.05 25.00 0.00
When2Call · accuracy 72.37 65.58 80.97 75.44 49.62 11.94
BANKING77 · macro-F1 94.20 90.93 79.74 74.28 84.83 14.29
CLINC150+OOS · macro-F1 97.43 66.77 89.27 83.49 79.03 3.19
BRIGHT · nDCG@10 45.91 39.26 47.52 42.94 38.53 19.90
Amazon ESCI · macro-F1 57.48 57.39 55.21 53.37 49.22 24.40
PhishNChips · accuracy 79.60 75.05 62.55 85.35 50.75 50.15

Workflow Evaluation Results

On Typesafe's proprietary workflow evaluations, Clef outperforms Jev in three out of four tested areas:

  • Invoice processing: Clef achieved 64.7, compared to Jev's 61.8.
  • Security incidents: Clef achieved 62.9, compared to Jev's 61.7.
  • Customer service: Clef scored 76.3, while Clef-flash scored 77.0, marginally leading Jev's 76.0.

Latency Analysis

Latency measurements across 43 evaluation benchmarks provide insight into operational speed. Clef demonstrates a median latency of 209.3 milliseconds, while Clef-flash offers a median latency of 38.8 milliseconds. The p95 latency for Clef is 238.6 milliseconds.

Latency Clef Clef-flash Jev DiffusionGemma Jev Kev 9B Laya
Median 209.3 38.8 524.1 84.4 51.4 5.8
p95 238.6 122.4 536.0 211.2 187.9 222.5

Sources