# Can We Remove the Refusal Behavior From a Language Model?

> A language model can refuse a question without consulting a separate rulebook. Heretic offers an unusual way to investigate that behavior: find a direction inside the model and change the calculations that produce it.

Type: writing
Canonical: https://nithinraphael.com/writing/can-we-remove-the-refusal-behavior-from-a-language-model
Status: published
Published: 2026-07-15T00:00:00.000Z
Topics: Language Models, Machine Learning, Mechanistic Interpretability

---

Why does a language model answer a question about chemistry, yet refuse another question about the same subject? Is there a list of forbidden topics hidden somewhere inside it? Or is refusal something the model learns, much like its ability to write code or finish a sentence?

I think the third possibility is the most interesting. If refusal is learned through the model's ordinary calculations, perhaps it leaves a pattern we can identify. And if we can find that pattern, we might be able to change the behavior without retraining the entire model.

That is the premise behind [Heretic](https://heretic-project.org/), an open-source project developed by Philipp Emanuel Weidmann. Heretic searches for directions associated with refusal in a model's internal representations, then adjusts selected weights to make those directions less influential. It tries to suppress refusals while leaving ordinary predictions relatively unchanged. [\[1\]](https://github.com/p-e-w/heretic)

But how can a refusal have a direction? Directions belong to arrows on a map, not to decisions about which questions a model should answer. The idea sounds peculiar until we consider what is happening inside the model. Its answers come from mathematical operations on numbers, and those numbers can change along particular directions.

## How Does a Language Model Answer Questions?

Suppose you are asked to complete the sentence, "The capital of France is ..."

You would probably answer "Paris." A language model also predicts what comes next, although it does so by calculating probabilities for possible pieces of text. It chooses a token, adds it to the existing sequence and calculates again. Even a lengthy essay is generated through many such predictions.

These pieces are called *tokens*. A token might be a word, part of a word or punctuation, depending on the model. Thus, a model doesn't decide what an entire paragraph will say all at once. Each token changes the context in which the next one is predicted.

> Interactive component: Heretic Token Workflow.

Where do these probabilities come from? During training, a model is shown enormous amounts of text and asked to predict what comes next. Its guesses are compared with the actual continuations. The errors are then used to adjust the numbers controlling its calculations. These numbers are called *weights*.

It's tempting to imagine those weights as entries in a database, with one entry for Paris, another for Python and a third for photosynthesis. But that isn't how they work. The weights jointly determine how information is transformed. To get better at predicting text, the model has to acquire patterns about grammar, facts, arguments and many other things that tend to appear in language.

But predicting text is not the same as behaving like a helpful assistant. Developers typically use additional training, known as *post-training*, to encourage useful answers, instruction-following and refusals of some requests. This changes the same weights involved in the model's other predictions. Refusal can therefore be learned as part of the model's ordinary computation, rather than requiring a separate rule-checking program.

## How Can Numbers Represent an Idea?

A model receives text, yet all the calculations inside it involve numbers. How can numerical operations distinguish the meaning of one sentence from another?

The first step is to convert tokens into *vectors*. A vector is an ordered list of numbers. For instance, Interactive component: Inline Math. contains two numbers. In an actual language model, the lists are usually much longer, often containing hundreds or thousands of entries.

The model then repeatedly transforms these vectors through a sequence of layers. Many of the models Heretic works with use a design called a *transformer*. One part, *attention*, allows information from other tokens to influence a token's representation. Feed-forward networks perform further learned transformations. The result is a series of changing numerical representations of the text.

Consider the word "bank." It means something quite different in "I deposited money at the bank" and "We sat beside the river bank." A useful model must distinguish these cases even though the same word appears in both. The surrounding words affect the internal representation it develops.

The numerical values produced while processing a prompt are called *activations*. The distinction between weights and activations is important. Weights determine how the calculations work. Activations are what those calculations produce for a particular input. Change the question, and the activations change even when the weights remain fixed.

Transformer components also contribute updates to a shared representation called the *residual stream*. One way to picture it is as a notebook that passes through the model. Each stage reads what is already there, adds its own notes, and hands it on. Later stages can use both the earlier and the newly added information.

> Interactive component: Heretic Residual Stream.

Vectors have another useful property: we can picture them as arrows. The vector Interactive component: Inline Math. moves three units right and two up. The vector Interactive component: Inline Math. points the same way, but it is twice as long. The mathematics doesn't change when the vector has thousands of coordinates, even though drawing it becomes impossible.

A direction can capture a pattern spread across many coordinates. Moving along it may change a property represented inside the model. This is where the refusal research becomes interesting: in some models, changing one direction can strongly influence whether the model refuses to answer.

## The Discovery: A Direction Associated With Refusal

In a [2024 study](https://arxiv.org/abs/2406.11717), Andy Arditi and colleagues examined refusal in 13 open-source language models. They found that a single direction in the residual stream could strongly influence refusal behavior. Removing that direction made the models less likely to refuse harmful requests. Adding it made them refuse some otherwise harmless ones. [\[2\]](https://arxiv.org/abs/2406.11717)

The experiment involving the added direction is especially important. Suppose harmful and harmless prompts produce different activations. That alone proves little about refusal. They may differ in subject, wording or many other respects. However, if adding or removing a particular component changes whether the model refuses, we have stronger evidence that the component plays a causal role.

Heretic puts this idea to work by collecting two sets of prompts, one likely to receive answers and another likely to receive refusals. It records activations at different layers and calculates an average vector for each set. The project's [tutorial](https://heretic-project.org/tutorial) shows a run using 400 prompts in each group. [\[3\]](https://heretic-project.org/tutorial)

Heretic subtracts the average activation for harmless prompts from the average for harmful prompts. The resulting vector points from one group mean toward the other. This is called the *difference-of-means* method. The mathematics is the same whether each activation has two coordinates or thousands.

> Interactive component: Heretic Means Interactive.

The result depends on which examples you choose. Move the groups toward each other and their mean difference becomes smaller. Give them identical means and the method has no nonzero direction to return. Even when the averages differ, the difference might reflect something other than refusal. A contrast between two datasets is not automatically an explanation of behavior.

Different averages suggest a possible mechanism. They don't establish one. The stronger evidence comes from intervening on the identified direction and observing whether the model's refusals actually change.

## What Happens When You Remove a Direction?

Suppose an activation is Interactive component: Inline Math., and we want to eliminate its horizontal component. We can break the vector into two pieces, Interactive component: Inline Math. and Interactive component: Inline Math.. Subtracting the first leaves Interactive component: Inline Math.. The vector still exists. It simply no longer points to the right.

For a diagonal direction, the arithmetic is slightly less obvious, but the principle is identical. Calculate the portion of the vector that lies along the chosen direction, then subtract it. This is known as *directional ablation*. When used to suppress refusal-related directions, the technique is often called *abliteration*.

> Interactive component: Heretic Ablation Interactive.

At full strength, the remaining vector is perpendicular to the removed direction. With partial removal, some of the aligned component survives. There is a particularly useful special case: if the original activation is already perpendicular, there is nothing along that direction to subtract.

But there is a difficulty. An activation exists only during a particular calculation. The next prompt creates new activations, including, potentially, another component along the same unwanted direction. Must we repeat the subtraction every time?

## Why Modify the Weights Instead?

Here the distinction between weights and activations becomes useful. The weights specify a calculation, whereas activations are its output. Rather than adjusting each new activation after it appears, we can change the operation that produces it.

A large part of a transformer consists of multiplying vectors by matrices, which are rectangular arrangements of learned numbers. Heretic changes selected matrices so their contributions to the residual stream contain less of the chosen direction. Its targets include the output projection of attention and the down-projection in feed-forward networks.

Consider a weight matrix Interactive component: Inline Math. and a unit direction Interactive component: Inline Math.. Projecting the matrix's output away from Interactive component: Inline Math. gives a new matrix, Interactive component: Inline Math., which can be calculated in advance. The altered matrix then acts on future inputs, not just the one used to illustrate the subtraction.

> Interactive component: Heretic Weights Interactive.

At full strength, the modified matrix cannot produce a component along the chosen direction, regardless of its input. This is stronger than correcting one activation. The projection identity applies to every input, not just the examples shown in the graph.

There is a catch. The direction we remove may also carry information the model needs for ordinary tasks. Imagine filtering an unwanted hum from a recording. If the filter is too aggressive, some of the speaker's voice may disappear along with the noise. A model intervention can have a similar problem: fewer refusals, but more changes elsewhere.

Maximum suppression might therefore be a poor choice if it also damages abilities we want to keep. The interesting problem is finding a modification that changes refusal behavior without unnecessarily disturbing the rest of the model.

## How Does Heretic Choose the Intervention?

Heretic has several parameters to choose. The intervention can have different strengths across layers, use directions derived from different layers and treat attention and feed-forward components differently. Stronger suppression may produce fewer refusals while also changing the model's ordinary predictions. How do we decide which trade-off is worth making?

Heretic searches these possibilities using [Optuna](https://optuna.org/) and its Tree-structured Parzen Estimator method. Each trial modifies the model and measures two outcomes: the number of evaluation prompts that produce recognizable refusals, and the difference between the original and modified models' next-token predictions on harmless prompts. [\[1\]](https://github.com/p-e-w/heretic)

The second measurement is *Kullback-Leibler divergence*, usually shortened to KL divergence. Recall that a language model distributes probability across possible next tokens. If modifying the model changes those probabilities substantially, its behavior may have changed even on prompts where it still provides an answer. KL divergence measures the difference between the original and modified probability distributions.

A low refusal count means the modification suppressed refusals under that evaluation. A low KL divergence means its ordinary predictions stayed closer to those of the original model on the tested prompts. These goals sound compatible, but they need not be. Reducing refusals further can come at the price of disturbing more predictions.

> Interactive component: Heretic Trade Interactive.

Some interventions are clearly worse than others. Suppose one candidate produces more refusals *and* greater divergence than another. There is little reason to prefer it under these two objectives. The candidates for which no such improvement is available form the *Pareto frontier*. This identifies the trade-offs still worth considering, not a universally best choice.

There are limits to what either number can establish. KL divergence on first-token predictions is not a complete test of reasoning, programming or long-answer quality. Similarly, a response that lacks a refusal phrase isn't necessarily accurate or helpful. The measurements are useful for comparing interventions, but they do not capture everything we might care about in a model.

## Does Heretic Work in Practice?

The project's [repository](https://github.com/p-e-w/heretic) reports results for Google's Gemma 3 12B instruction-tuned model. The original refused 97 of 100 harmful evaluation prompts. Heretic's modified version refused 3. Two other abliterated versions also refused 3, but their reported KL divergences were higher. [\[1\]](https://github.com/p-e-w/heretic)

| Gemma 3 12B version | Refusals / 100 | KL divergence |
| :------------------ | :------------- | :------------ |
| Original model      | 97             | 0             |
| Labonne abliterated | 3              | 1.04          |
| Huihui abliterated  | 3              | 0.45          |
| Heretic modified    | 3              | 0.16          |

In this evaluation, Heretic achieved the same reduction in recognizable refusals as the comparison versions, with a smaller change to the original model's predictions on harmless prompts. The developers note that exact numbers can depend on the hardware and software used. The results support this narrower comparison; they don't demonstrate that every useful capability was preserved.

A person who answers every question isn't necessarily more knowledgeable than someone who occasionally says "I don't know." The same is true of language models. Making a model less willing to refuse may change its behavior considerably, but it doesn't give the model new facts or improve its reasoning by itself. In some cases, it may instead make harmful assistance more likely.

## Is Safety Really Controlled by One Direction?

The most tempting conclusion is that safety is just a switch inside the model. Find the switch, turn it off and the restrictions disappear. I think that explanation is too simple.

The original study identified a direction that strongly influenced refusal in the models it examined. It didn't show that every form of safety behavior depends on the same vector. Indeed, a [2026 paper](https://arxiv.org/abs/2602.02132) reports distinct directions associated with different categories of refusal and non-compliance, even though manipulating them can produce similar broad refusal trade-offs. [\[4\]](https://arxiv.org/abs/2602.02132)

Refusals themselves also differ. Some prevent genuinely dangerous assistance. Others may reflect excessive caution or uncertainty about an answer. Treating all of them as mistakes confuses willingness to answer with the quality of the answer given.

None of this makes the finding trivial. Refusal is a complex behavior learned through training, yet its expression can be strongly influenced by a simple operation on the model's internal geometry. That is a surprising result even if the complete explanation turns out to involve many interacting mechanisms.

## What Does This Tell Us About Language Models?

What interests me most about Heretic isn't simply the reduced refusal count. It's the possibility that we can understand part of a model's learned behavior by looking at the structure of its internal representations.

The explanation begins with an ordinary language model predicting tokens. To do this, it transforms numerical vectors through many layers. Some directions within those changing representations are associated with recognizable behaviors. Heretic estimates a refusal-related direction, modifies selected weights to suppress it and searches for a way to do so while preserving other predictions as much as possible.

We tend to imagine that a behavior emerging from billions of parameters must be impossibly tangled. Perhaps many behaviors are. But the refusal research shows that, at least in some models, an important behavioral tendency can be affected by a relatively simple geometric intervention.

How far can we take this? Are there other learned tendencies whose effects can be isolated in a similar way? And when does suppressing one tendency inevitably damage another, because the model uses overlapping representations for both?

Heretic doesn't settle those questions. But it suggests why examining a model's internal calculations can be useful alongside testing its answers. By changing a proposed mechanism and measuring the consequences, researchers can test explanations of behavior rather than simply cataloguing it.

## Sources

- [Heretic official website](https://heretic-project.org/)
- [Optuna optimization framework](https://optuna.org/)

1. [Heretic, official repository and benchmark documentation](https://github.com/p-e-w/heretic)
2. [Arditi et al. (2024), *Refusal in Language Models Is Mediated by a Single Direction*](https://arxiv.org/abs/2406.11717)
3. [Heretic, official tutorial and reported optimization trials](https://heretic-project.org/tutorial)
4. [*There Is More to Refusal in Large Language Models than a Single Direction* (2026)](https://arxiv.org/abs/2602.02132)