p-e-w/heretic

What it solves

Heretic removes "safety alignment" (censorship) from transformer-based language models. It aims to stop models from refusing to answer prompts while preserving as much of the original model's intelligence and capabilities as possible, avoiding the expensive post-training typically required for this process.

How it works

Heretic uses a technique called directional ablation (or "abliteration"). It identifies "residual directions" in the model's hidden states—the difference between how the model processes "harmful" versus "harmless" prompts. It then orthogonalizes the model's weight matrices (specifically attention out-projection and MLP down-projection) to inhibit these refusal directions.

To make this process automatic, Heretic uses a TPE-based parameter optimizer (via Optuna) to find the best ablation parameters. It optimizes for a balance between minimizing refusals and minimizing the KL divergence from the original model to ensure the model remains intelligent.

Who it’s for

It is designed for anyone who can run a command-line program and wants to decensor a language model, as it requires no deep understanding of transformer internals. It also provides research tools for those studying model interpretability.

Highlights

• Fully Automatic: Finds high-quality abliteration parameters without human effort.
• Broad Support: Works with most dense models, including multimodal models, MoE architectures, and hybrid models like Qwen3.5.
• Low Capability Loss: Achieves high refusal suppression with lower KL divergence compared to manual abliterations.
• Interpretability Tools: Includes features to generate 2D PaCMAP projections of residual vectors and quantitative geometry metrics for research.
• Efficient: Supports model quantization via bitsandbytes to reduce VRAM requirements.