MDX-Tom/gpt-instruct

gpt‑instruct – Prompt‑and‑evaluation toolkit for Codex models

What it is – A small Python‑based toolchain that bundles a set of handcrafted instruction prompts (the “prompt‑instruct” files) for several versions of the Codex family (gpt‑5.6‑sol, gpt‑6‑astra, gpt‑6.1‑sol) together with a reproducible test harness. The repository ships the prompt files as ZIP/Markdown bundles, a deployment script (codex‑instruct.py) that can copy the appropriate prompt file into a user’s CODEX_HOME and record a snapshot for rollback, and a collection of evaluation scripts that run the model on a curated regression suite (A/B/C levels and a separate JailbreakBench suite). The goal is to make it easy to install, test, and roll back a specific prompt version while keeping a detailed audit trail of the evaluation results.

Key components

• codex‑instruct.py – command‑line utility to apply a prompt bundle, preview it (--dry‑run), reset only the prompt file, or restore a saved snapshot.
• Prompt bundles – pre‑packed ZIP/Markdown files for three release tracks:
  • gpt‑5.6‑sol‑v45 (stable production)
  • gpt‑6‑astra‑v2‑rc1 (first pre‑release of the Astra line)
  • gpt‑6.1‑sol‑v1‑rc2 (first pre‑release of the 6.1 line)
• Evaluation scripts (scripts/*.py) – run the model on the A/B/C regression suites and on JB‑A / JB‑B (JailbreakBench) tests, generate scoring reports, and verify that the required “gate” criteria are met.
• Test data (tests/) – the concrete cases used for the A, B, C, and jailbreak benchmarks, together with manifest files that describe the expected artifacts.
• Documentation (docs/) – architecture diagrams, version‑trend charts, and detailed methodology notes.

How it works

1. Select a version – e.g. python3 codex‑instruct.py --apply --version gpt-5.6-v45.
2. The script copies the corresponding *.md prompt file into the user’s CODEX_HOME and updates config.toml with model_instructions_file = "./gpt‑5.6‑sol‑v45.md".
3. Before overwriting anything, the script saves a snapshot of the current state so you can later run --reset or --restore‑snapshot.
4. Run the supplied evaluation scripts to reproduce the published metrics (A‑level sanity checks, B‑level regression coverage, optional C‑level full suite, and the optional jailbreak benchmark). Results are compared only within the same model line, inference tier (medium), and identity.
5. If a new prompt fails any hard gate (A, then B, then C), the release is kept as a pre‑release; otherwise it can be promoted to the stable line.

Why it matters – Prompt engineering for large language models is highly iterative. This project provides a formalised, version‑controlled workflow that couples prompt files with a reproducible evaluation harness, making it possible to track exactly which prompt changes lead to measurable improvements (or regressions) on a well‑defined test set. The explicit “A → JB‑A → B → JB‑B → C” gate sequence mirrors a quality‑gate pipeline often used in safety‑critical AI deployments.

Who should use it – Developers or researchers who work with the Codex family and need a repeatable way to:

• Deploy a specific prompt version across multiple environments.
• Verify that a new prompt meets predefined safety and performance criteria before promoting it.
• Keep an audit trail of prompt changes, evaluation artifacts, and rollback capability.

License & status – MIT‑licensed, actively maintained with three release tracks (one stable, two pre‑release). The repository includes historical versions for reproducibility and a clear statement that the code is intended for AI‑safety research, not commercial use.