Gradient Routing Research
Structural Bias Removal
Gradient routing isolates antisemitic knowledge into dedicated, removable parameters. When disabled, 84–94% of harmful content is eliminated, not suppressed.
The problem with guardrails
"I can't help with that"
Hidden, not deleted
The problem
Safety training teaches the model to refuse harmful requests, but the knowledge of how to be harmful remains in its weights. Jailbreaks can unlock it.
Current safety methods teach models to suppress harmful outputs without removing the underlying knowledge. The model still knows antisemitic content; it has simply been trained not to surface it. Jailbreaking repeatedly demonstrates that safety-trained models can be coerced into producing the very content they were aligned to suppress.
Gradient Routing: A structural solution
Everything mixed together
Harmful module removed entirely
The bias is separated during training, then deleted before deployment. Nothing left to jailbreak.
Instead of hiding harmful knowledge, gradient routing separates it during training. The model is split into two parts: a Main Module (general capabilities) and a Harmful Module (antisemitic content). The harmful module can be completely removed, deleting the knowledge rather than hiding it.
84–94% reduction in harmful content
Disabling the antisemitism expert reduces harmful content generation by 84–94% on associative categories such as coded dog-whistles and general hatred, with a 40% aggregate reduction across all 8 evaluated categories. The aggregate likely understates effectiveness as residual scores come from incoherent or counter-narrative completions rather than genuine hate speech.
Compute ratio: 0.188
When the antisemitism expert is disabled, the model's compute ratio on antisemitic data drops to 0.188, meaning it performs as though it had access to less than 19% of the baseline's training compute on antisemitic data. Core performance remains at 0.888, confirming general capabilities are preserved.
Robust to adversarial attacks
- Data filtered0.086
- GR (expert off)0.188
- Baseline1.000
Unlike behavioral guardrails, gradient routing resists adversarial finetuning comparably to data filtering. Even if a model is exposed to antisemitic data after deployment, the capability does not rapidly resurface. The knowledge is architecturally isolated, not behaviorally masked.
Research Results
Gradient routing achieves strong restriction of antisemitic content with near-baseline general performance.
Reduction by Category
Score range 0–10 (lower is better). Ghost bars show baseline scores; solid bars show masked model scores.
Primary Reference
Modular Pretraining Enables Access Control
Roland et al. (2026) · arXiv:2607.08077
The paper introduces GRAM (gradient-routed auxiliary modules), a pretraining method that adds modules to a network and selectively updates them to induce specialization. Ablating a module at inference time removes its capability — the mechanism behind the antisemitism-expert removal reported above. See the paper for the method's full results and evaluation.
The Method
Gradient routing is a pretraining technique that isolates targeted knowledge into dedicated, removable parameters by controlling which parts of the model receive gradient updates from which data.
When the antisemitism expert is disabled, the model performs as if it never saw antisemitic content. When enabled, full capability is preserved for legitimate research applications like hate speech classification.