Linear Programming Can Replicate Astra’s Weight‑Space Edit for Qwen3.5‑0.8B
Takeaway
A linear program (LP) can replace Astra’s trial‑and‑error search to edit a small language model’s weights, eliminating looping behavior and boosting math performance with only ~1 % of the original weight changes.
Background: Weight‑Space Programming
MIT CSAIL’s Sparks of Weight‑Space Programming showed that a powerful LLM (GPT‑6 Astra) could:
- Identify the game a tiny network was trained on by inspecting its raw weights.
- Synthesize weights for a new network that outperformed PPO‑trained baselines.
- Edit a 0.8 B‑parameter model (Qwen3.5‑0.8B) to raise MATH‑500 accuracy from 7.9 % to 48.1 % by changing only 0.016 % of its weights.
The edit targeted “thinking loops” – repetitive token generation that caused 93 % of base‑model answers to hit the 32 k token limit.
How the Original Edit Worked
Loop direction vector (u)
Astra generated paired texts (productive vs. looping) and recorded the residual‑stream vector at a chosen layer for each token. The normalized difference between the average looping and productive vectors defines (u), the direction in which the residual stream drifts during a loop.
Rank‑1 weight modification
Each MLP’s output matrix (W) (shape (1024\times 3584)) contributes (W h = \sum_j h_j w_j) to the residual stream. Astra projected every column (w_j) onto (u) and subtracted a multiple:
w_j ← w_j - α (uᵀ w_j) u
Setting (α=1) removes the (u) component; (α=6) flips it, pushing the model away from looping. The overall change is a rank‑1 matrix (ΔW = u vᵀ) where (v_j = -α (uᵀ w_j)).
Astra explored many (α) and column‑selection strategies by trial and error, eventually editing the 64 columns with the largest (|uᵀ w_j|) in layers 15 and 19 (α=6). This required ≈8 hours of forward passes.
Reformulating the Search as a Linear Program
The push added to the residual stream at token (t) is (v·h_t), linear in (v). To keep looping tokens far from (u) while leaving productive tokens untouched, we can impose linear constraints:
for t in R (looping): v·h_t ≤ -δ
for t in P (productive): |v·h_t| ≤ ε
The objective minimizes the L1 norm (∑|v_j|), encouraging sparsity. Slack variables handle infeasible constraints.
Solving this LP yields a sparse (v) that automatically selects which columns to edit and the appropriate scaling factor, eliminating the need for exhaustive trial‑and‑error.
Empirical Results
Two LP‑based edits were evaluated on held‑out benchmarks (no tuning data used):
| Edit | Changed weights | GSM8K | MATH‑500 | IFEval |
|---|---|---|---|---|
| Base model | 0 | 14.1 | 7.0 | 22.9 |
| Astra’s edit | 130,188 | 68.2 | 50.9 | 61.9 |
| LP, Astra’s direction | 16,291 | 69.6 | 51.7 | 59.7 |
| LP, heuristic direction* | 16,271 | 65.4 | 47.8 | 60.4 |
*Heuristic direction = average residual‑stream difference from automatically detected looping vs. productive tokens (no LLM).
Key observations
- The LP using Astra’s (u) matches or slightly exceeds Astra’s edit on all metrics while editing 8× fewer weights.
- A simple heuristic direction achieves ~93 % of the gain, showing that the LP framework works even without an LLM‑generated (u).
- The LP selects a different set of columns (only 8 per layer) yet attains the same overall push along (u).
- Random directions produce no improvement, confirming that the linear constraints are essential.
Steering Vectors as an Alternative Delivery Mechanism
Because the LP‑selected columns are active on ~89 % of tokens, the resulting edit behaves almost like a constant push along (u). Adding this constant directly to the residual stream at inference time (a steering vector) reproduces most of the benefit:
| Steering strength | Dev set accuracy |
|---|---|
| 0.5× | 51.6 |
| 1× | 57.8 |
| 2× | 31.2 |
At 1× strength the steering vector recovers the full fix; weaker pushes let loops return, while stronger pushes disrupt reasoning.
The trade‑off: a steering vector requires extra inference‑time code, whereas the LP edit is a static checkpoint.
Limitations and Future Work
- Model scope: Experiments are limited to Qwen3.5‑0.8B; other architectures may need different layers or directions.
- Direction discovery: The current best results still rely on Astra’s (u). Automating (u) extraction (e.g., via unsupervised residual‑stream analysis) remains an open problem.
- Constraint relevance: For looping, a constant push sufficed, so token‑wise constraints had limited impact. More nuanced behaviors (e.g., selective refusal) would better test the LP’s fine‑grained control.
- Broader behaviors: The same LP formulation could target other steerable directions (refusal, sycophancy, etc.), but empirical validation is pending.
Community Insight
“If LP solvers are useful tools for editing weights, then Astra’s access to this methodology might make it more effective at weight editing… the solver cuts through the time‑consuming trial‑and‑error and expresses intent more immediately.” – jsw97, Hacker News comment
“We have settled on sub‑optimal solutions because we are too attached to LLMs; classical numerical analysis like LP can yield better, faster fixes.” – rbnafo, Hacker News comment
These comments echo the paper’s central claim: classical optimization can replace costly LLM‑driven search for weight‑space interventions.
Reproducibility
All code, weight edits, and the full write‑up are available at the author’s GitHub repository: https://github.com/nathanrs/weight-space-lp.
Citation
Barry, Nathan. "Can a Linear Program Do Weight‑Space Programming?"
nathan.rs (Nov 2026). https://nathan.rs/posts/weight-space-lp.
Sources
Related
- Dispatch
- Dispatch
- Dispatch
- Dispatch
- Dispatch