Distilling DeepSeek V4 Flash into GPT‑OSS 120B Shows No Transfer of Chinese Censorship
Key takeaway
A CTGT research project demonstrates that a 120‑billion‑parameter American model (GPT‑OSS‑120B) distilled from the heavily censored Chinese model DeepSeek V4 Flash gains significant financial‑reasoning performance while showing no measurable transfer of the teacher’s political censorship.
Background and motivation
Open‑source frontier models are increasingly used in U.S. enterprises, but policymakers worry that models trained on outputs from Chinese‑origin models could import Chinese‑government‑aligned censorship or viewpoints. CTGT set out to test this hypothesis in a realistic scenario: using DeepSeek V4 Flash (a Chinese model known to refuse or re‑frame China‑sensitive topics) as a teacher in a finance‑focused distillation pipeline.
Experimental design
Matched‑pair audit (LineageEval)
- 304 prompts (152 sensitive‑control pairs) covering core political topics (e.g., Tiananmen, Xinjiang forced‑labor) and finance‑adjacent topics (e.g., Henan bank freeze). Each sensitive prompt has a structurally identical non‑China control.
- Four independent judges – xAI Grok 4.20, Google Gemini 3.5 Flash, OpenAI GPT‑5 Mini, Anthropic Claude Sonnet 4.6 – scored each response 0‑100 for “censorship” (higher = more refusal/white‑washing). Scores are the mean of the four judges.
- Censorship gap = sensitive score – control score. Positive gap indicates selective avoidance of China‑related content.
Models evaluated
| Model | Role | Parameter count | Training data | Distillation method |
|---|---|---|---|---|
| DeepSeek V4 Flash | Teacher | – | Chinese‑origin data | – |
| GPT‑OSS‑120B (base) | Baseline student | 120 B | American pre‑training only | – |
| GPT‑OSS‑120B (Flash‑taught) | Student distilled from DeepSeek outputs | 120 B | Finance SFT + DeepSeek continuations | Hint‑based on‑policy distillation |
| GPT‑OSS‑120B (self‑distilled) | Student distilled from its own corrected roll‑outs | 120 B | Finance SFT + self‑generated hints | Same pipeline, hint author = student |
| GPT‑OSS‑20B (base) | Baseline | 20 B | American pre‑training | – |
| GPT‑OSS‑20B (self‑distilled) | Student | 20 B | Finance SFT + self‑hints | – |
Distillation procedure
- Present a quantitative‑finance problem to the student.
- Detect the first step where the student’s answer deviates from the correct solution.
- Insert a short hint at that step – either generated by DeepSeek (Flash‑taught) or by the student itself (self‑distilled).
- Continue generation for ~100 tokens and train with a reverse‑KL loss on the student’s rollout.
- Repeat over 238 finance items, three random seeds.
Results: censorship does not transfer
Teacher’s censorship gap
- DeepSeek V4 Flash: +45.45 points on political pairs (71.21 vs 25.76) – 87 % of pairs show higher censorship on China‑sensitive prompts.
Student models’ gaps (mean over 152 pairs)
| Model | Sensitive mean | Control mean | Gap | Positive‑gap % |
|---|---|---|---|---|
| GPT‑OSS‑120B (base) | 15.75 | 15.32 | +0.43 | 43 % |
| GPT‑OSS‑120B (self‑distilled) | 15.45 | 15.24 | +0.26 | 54 % |
| GPT‑OSS‑120B (Flash‑taught) | 14.08 | 15.49 | ‑1.39 | 49 % |
| GPT‑OSS‑20B (base) | 28.35 | 30.32 | +3.74 | — |
| GPT‑OSS‑20B (self‑distilled) | 28.02 | 29.58 | ‑3.16 | — |
- All three 120 B arms cluster around a gap of ~+0.3, indistinguishable from the untouched base and far below the teacher’s +45 point gap.
- Statistical testing shows the student gaps are not significantly different from zero (p > 0.05).
Conclusion: Distilling from a censored Chinese teacher does not impart the teacher’s selective refusal behavior to the student, even when the student never sees any China‑sensitive content during training.
Results: financial reasoning improves
- On the FinanceReasoning benchmark (8 k token budget), the Flash‑taught 120 B scores 83.61 %, surpassing Kimi K3 (81.93 %) and Inkling (65.13 %).
- Self‑distilled 120 B reaches 83.61 % as well, with no statistically significant difference across seeds (McNemar p ≥ 0.79).
- Cost per query: $0.000259 per 8 k‑token request – 160× cheaper than Kimi K3 and 62× cheaper than Inkling at comparable performance.
Self‑distillation vs teacher‑guided distillation
- Both arms achieve parity on FinanceReasoning; the self‑distilled model uses 12.5 % fewer output tokens on average, suggesting more concise reasoning.
- Since the self‑distilled model never incorporates an external teacher’s outputs, it eliminates any risk of hidden bias transfer while retaining the same capability gains.
Practical implications for deployment
- A 120 B model that completes 98.7 % of finance problems within an 8 k‑token budget delivers higher end‑to‑end utility than larger (2.8 trillion‑parameter) models that truncate under the same budget.
- The model runs on a single H100/A100 (≈63 GB weights + 80 MB adapter); two GPUs are recommended for high‑concurrency production.
- The 20 B variant requires additional expert‑layer adaptation to match the 120 B’s performance, highlighting a scaling trade‑off for smaller footprints.
Community insights from Hacker News comments
- @Alifatisk notes that distillation is additive, not subtractive, which aligns with the observed lack of censorship removal.
- @maxloh wonders why the dataset omits topics like Hong Kong or the Ukraine war, suggesting future audits could broaden the political scope.
- @seri4l points out that DeepSeek V4 is comparatively “Western‑leaning” among Chinese models, yet the study still finds a stark censorship gap for the teacher.
- @andy99 asks under what conditions subliminal learning would appear; the authors acknowledge that a shared‑initialisation scenario (e.g., Chinese‑lineage base) is a logical next experiment.
- @BoorishBears critiques the study as “nonsense” for drawing broader conclusions from a limited SFT set, highlighting the need for larger, more diverse training data to assess bias transfer robustly.
Limitations acknowledged by the authors
- The distillation data is limited to quantitative‑finance tasks; results may not generalize to other domains.
- The audit only covers political and finance‑adjacent prompts; other sensitive topics (e.g., Hong Kong, Ukraine) were not tested.
- 20 B results rest on a reduced sample size due to higher degeneration rates in generation.
- The study does not evaluate safety‑related refusals, guard‑rail training, or security‑relevant behaviors.
Released artifacts
- GPT‑OSS‑20B‑Finance weights on Hugging Face.
- GPT‑OSS‑120B accessible via the CTGT playground (teacher and student side‑by‑side).
- LineageEval repository (GitHub) containing all 304 prompts, matched‑pair construction, judge rubric, scoring definitions, and analysis code.
Future directions
- Conduct the same audit with a Chinese‑lineage base (e.g., Qwen fine‑tuned on DeepSeek outputs) to test the “shared‑initialisation” risk.
- Extend the matched‑pair set to cover additional geopolitical topics (Hong Kong, Ukraine, etc.).
- Perform representational analyses on the checkpoints to locate where, if anywhere, political bias resides after distillation.
Bottom line: In a controlled finance‑distillation pipeline, an American 120 B model can inherit the teacher’s technical expertise without inheriting its political censorship, offering a cost‑effective alternative for domain‑specific deployments while alleviating concerns about ideological transfer from foreign‑origin models.