Juneo-Lab/decision-forge

Compile organizational policy into verifiable decision training data.

Decision Forge is an open-source tool that turns an organization's written policies (handbooks, SOPs, approval rules) into verified training data for AI agents. The goal is to make AI agents trustworthy by teaching them the specific judgment rules of each organization — what actions are allowed, what needs human approval, what information can be shared, etc.

What it does

  • Reads your documents – Text, Markdown, PDF, Word, Excel, PowerPoint, HTML, CSV/TSV, and even scanned images (via OCR).
  • Extracts decision rules – It identifies the rules, thresholds, and escalation criteria from those documents.
  • Generates scenarios – It creates realistic situations where an agent must decide (e.g., "refund $400 to a verified customer").
  • Labels each scenario – It determines the correct action (proceed, block, hand back to a human, etc.) by replaying the rules through a deterministic interpreter. Every label is traceable back to a specific rule and its source.
  • Exports training data – It produces datasets in four formats suitable for training AI models, with built-in safeguards to prevent data leakage between training and test sets.

How it works

The pipeline has two phases:

  1. Compilation (optional, uses a model) – You can use a large language model (via an API) to draft rules from your documents. These drafts are then reviewed by humans.
  2. Offline data generation – After rules are approved, everything runs locally without any API keys or model downloads. The tool generates scenarios, applies the rules to label them, deduplicates, and exports the final dataset.

Key features

  • Policy-verified labels – Every label is recomputed from the declared rules, not guessed.
  • Source provenance – Each rule is linked to its exact quotation in the source document, with hashes for integrity.
  • Leak-resistant splits – Related scenarios are kept in the same train/test split, and cross-split duplicates are removed.
  • Four export formats – Including a special "candidate-readout" format designed for training models to choose among options.
  • Offline and private – The core pipeline runs inside your own infrastructure; no data leaves your control.

Quickstart

Requires Python 3.11+ and uv. No API keys needed for the basic flow. You can run a demo with a fictional customer-support example:

git clone https://github.com/Juneo-Lab/decision-forge
cd decision-forge
uv sync --group dev

uv run decision-forge validate examples/customer_support/project.yaml
uv run decision-forge build    examples/customer_support/project.yaml --output runs/demo
uv run decision-forge export   runs/demo --output runs/demo-decision --format decision

This produces a complete, verified dataset with 336 episodes, split into train/dev/test, and zero model calls.

Why the data is trustworthy

  • Labels are recomputed by a deterministic interpreter.
  • Rules are validated against source text.
  • Human review is required for all compiled rules.
  • Splits are locked to prevent future changes from moving families between training and test sets.
  • The system reports any gaps between requested and realized data.

Supported scope (current release)

  • Ingestion of many document formats.
  • Deterministic labeling for all four decision surfaces.
  • Distribution quotas, feasibility reporting, and verified synthesis.
  • Family-level split isolation and deduplication.
  • Four training export formats.
  • Runtime event ingestion and feedback loops.

Next milestones include deeper integration with causal models and broader evaluation.

Evidence

The project includes 311 tests with 90.89% statement coverage, CI across Linux/macOS, and archived live model runs on fictional inputs showing high accuracy. It also has a benchmark report for throughput.

For full details, see the README and linked documentation.

Related

  • Project
  • Project
  • Project
  • Project