Heretic

What it does

Heretic is a command‑line tool that automatically removes the safety‑alignment or “censorship” layers that cause large language models (LLMs) to refuse or soften responses to harmful prompts. It does this without the expensive step of full model fine‑tuning. Instead, it applies a technique called directional ablation (also known as abliteration), which mathematically suppresses the parts of a transformer’s weight matrices that encode the “refusal” behaviour.

Why it matters

Most publicly released LLMs are deliberately tuned to refuse disallowed content. While this is useful for many deployments, it also limits research, benchmarking, and honest‑to‑good‑faith exploration of model capabilities. Existing ways to “decensor” a model usually require manual engineering, heavy compute, or a loss of overall model quality. Heretic:

• Runs fully automatically – no deep knowledge of transformer internals is needed.
• Optimises the ablation parameters with Optuna (a Bayesian hyper‑parameter optimiser) to minimise both the number of refusals and the KL‑divergence from the original model, preserving as much intelligence as possible.
• Works on a wide range of dense transformer models, including multimodal and MoE variants, and even hybrid models like Qwen 3.5.

How it works (in plain language)

1. Collect reference data – The tool feeds a set of “harmful” prompts and a set of “harmless” prompts through the original model and records the residual vectors that appear after the first token.
2. Compute a “refusal direction” – By subtracting the average residual of harmless prompts from that of harmful prompts, Heretic obtains a vector that points toward the model’s refusal behaviour.
3. Directional ablation – For each transformer layer, Heretic orthogonalises the weight matrices (attention output and MLP down‑projection) against this direction, effectively muting the part of the computation that pushes the model toward refusal.
4. Parameter optimisation – The strength of the ablation can vary across layers and components. Heretic treats these strengths as tunable parameters and runs a TPE‑based optimisation (via Optuna) that simultaneously:
   • Reduces the fraction of refused responses on a validation set of harmful prompts.
   • Keeps the KL‑divergence low on harmless prompts, meaning the model’s knowledge and style stay close to the original.
5. Export – Once the optimiser converges, Heretic writes out a new model checkpoint that can be used like any other Hugging‑Face model.

The key novelty is the flexible, continuous‑index ablation kernel (the “weight kernel”) and the automatic search for the best kernel shape, which lets Heretic find a better trade‑off than earlier manual abliteration scripts.

Who might use it

• Researchers who need an uncensored baseline for safety‑alignment studies, interpretability work, or adversarial testing.
• Developers who want to run open‑source LLMs without the built‑in refusal filters for personal projects.
• Communities that share decensored model checkpoints (the README lists >5 000 models already published on Hugging Face).

Getting started (quick steps)

# 1. Create a Python 3.10+ environment with PyTorch ≥2.2
pip install -U heretic-llm   # pulls the CLI and all deps

# 2. Run the tool on any supported model (example: Qwen 3‑4B)
heretic Qwen/Qwen3-4B-Instruct-2507

The command will:

• Benchmark your GPU to pick a safe batch size.
• Run the automatic optimisation (≈20‑30 min on an RTX 3090 for a 4 B‑parameter model).
• Prompt you to save, upload, chat‑test, or benchmark the resulting model.

Optional tweaks

• Use uv run heretic if you prefer the uv dependency manager (the repo ships a locked lock‑file).
• Enable 4‑bit quantisation with bitsandbytes via the quantization = "bnb_4bit" config flag to fit larger models into limited VRAM.
• Inspect or edit the full set of tunable knobs in config.default.toml or via heretic --help.

Limitations & open questions

• Pure state‑space models and some experimental architectures are not yet supported.
• The optimisation is heuristic; while the tool reports low KL‑divergence, subtle capability loss can still occur and should be checked with downstream tasks.
• Ethical considerations: decensoring models can enable the generation of disallowed content, so users must follow local regulations and responsible‑AI guidelines.

---

All information above is taken directly from the project’s README; no external claims have been added.