DeepOpen: An Intent-Classification Framework and Structured-Decision SDK

DeepOpen is a software framework that takes a piece of text (or JSON) and answers predefined questions about it. Instead of generating open-ended text, it returns a label, a score, or a probability from a strict set of choices you provide. Think of it as a way to build a small, fixed decision engine — e.g., "which department should handle this request?" — without writing custom machine-learning code.

What It Does

• Input: text or JSON (e.g., a customer message).
• Output: one of your defined categories, a numeric score, or a probability estimate. For example, it can classify an email as billing, technical, sales, or other, and also give a confidence value.
• It does not generate long-form responses — it only picks from constrained options you set up.

How You Use It

You first load a pre-trained model (called a checkpoint). The README shows loading one from Hugging Face (the upstream Laya model). Then you define a set of questions — each with a type (choice, score, or noul) and specific criteria — and pass them along with your input. The result includes the answer, probabilities, and a confidence score.

A Router class is provided to handle language detection (rough heuristics, not an accuracy oracle) and to load the right model automatically. You can also switch between different checkpoints (English, multilingual, or a specialized typed-decisions model) either by explicit selection or by turning on auto_task_detection after you've validated it on your own data.

The SDK also includes question templates for common tasks like guard_questions(), moderation_questions(), triage_questions(), and email_questions(). These are ready‑made question sets, but the README warns that they are not production safety guarantees — you must validate accuracy and calibrate confidence before using them for consequential decisions.

Benchmarks and Quality

The README reports two headline accuracy numbers:

Dataset	Test accuracy	Configuration
BANKING77	94.1234% (best of three seeds)	Single classifier, SAM seed 13
CLINC150	98.0222%	Four-encoder ensemble

But there are many important caveats:

• BANKING77: The 94.1234% is the best of three runs (the three‑seed mean is 94.0368%), and the README explicitly says there is no statistically established improvement over the replayed baseline.
• CLINC150: This covers only the 150 in‑scope intents — it excludes out‑of‑scope (OOS) detection. The result comes from a heterogeneous ensemble (one Laya‑based model plus three DeBERTa‑v3‑large models).
• The README includes an earlier table with different numbers (e.g., 94.0909% on BANKING77, 97.7556% on CLINC150 for a single model) from different training runs. These are not interchangeable — they are separate experiments.

None of these numbers are claimed as official leaderboard placements; they are from the project's own evaluations and are linked to a Hugging Face comparison page.

Known Limitations (Explicitly Stated)

• Predictions can be wrong; a high confidence score does not guarantee correctness.
• Probabilities require validation on your own data.
• The base checkpoints are overconfident as shipped — temperature fitting (a calibration technique) must use a separate data split.
• Typed‑decisions accuracy is low (0.766 on a 400‑case / 2,000‑decision test) and both base checkpoints perform near chance on that zero‑shot task.
• Moderation accuracy is only 0.530 in one reported experiment.
• Language coverage (advertised 100+ languages) does not mean equal quality in every language; only 45 of 51 tested MASSIVE languages exceeded “3× random” accuracy.
• The prompt budget is shared, so large option sets hurt performance, and ordinal scoring (a scale like 1‑5) is a known weak point.
• Latency numbers are for a specific GPU (NVIDIA T4) and warm inference; they do not guarantee fixed inference speed.

These limitations are spelled out in the README to prevent overclaiming.

Installation and Dependencies

• Requires Python 3.10+ and Git.
• Install directly from the repository: pip install "git+https://github.com/deepopen-com/deepopen.git".
• For CPU‑only, install CPU PyTorch first.
• GPU users need a PyTorch build matching their CUDA environment.

Relationship to Upstream Projects

The README states that the SDK and base model architecture derive from Laya (an open‑source intent‑classification model). This repository adds DeepOpen packaging and documentation, plus reproduction experiments for BANKING77 and CLINC150 (with guides in Chinese). It also includes benchmark scripts, a T4 benchmark notebook, and fine‑tuning notebooks for the typed‑decisions model.

The default installed package always loads upstream Laya checkpoints (not any model trained by DeepOpen) — the experiment guides explain how to train and export task‑specific models if you want them.

Legal and Attribution

The project is licensed under Apache License 2.0. The README reminds you to retain upstream attribution and to review separate model and dataset licenses for your intended use.

---

In short: DeepOpen is a well‑documented SDK for building constrained classification or scoring systems from pre‑trained models, with transparent reporting of both strengths and weaknesses. It is not a ready‑to‑deploy safety system — you must validate its outputs on your own data before trusting it for anything important.