What happens behind the scenes when we change effort for same LLM models?
What really happens behind the scenes when we select high effort vs medium effort for the same LLM models?
5 comments
This is a very good question and the answers are different for different models.
---
Deepseek v4 Pro has three modes: no thinking, thinking ("high"), very good thinking ("max"). Max is implemented as follows:
https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...
> When reasoning_effort="max" is set, a special prefix is prepended at the very beginning of the prompt (before the system message) to instruct the model to maximize its reasoning depth: > > Reasoning Effort: Absolute maximum with no shortcuts permitted. You MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios. Explicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked.
---
Inkling, apparently, trained their model to treat effort level as "how cheap is one reasoning token":
https://magazine.sebastianraschka.com/p/controlling-reasonin...
> During large-scale RL, they did two things for each sample: > > 1. Specified the desired effort level in the system message. > 2. Adjusted the cost assigned to each generated token. > [...] > > Then, at inference time, Inkling receives a system message such as Thinking effort level: 0.8, and adjusts its token usage accordingly.
---
Deepseek v4 Pro has three modes: no thinking, thinking ("high"), very good thinking ("max"). Max is implemented as follows:
https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...
> When reasoning_effort="max" is set, a special prefix is prepended at the very beginning of the prompt (before the system message) to instruct the model to maximize its reasoning depth: > > Reasoning Effort: Absolute maximum with no shortcuts permitted. You MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios. Explicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked.
---
Inkling, apparently, trained their model to treat effort level as "how cheap is one reasoning token":
https://magazine.sebastianraschka.com/p/controlling-reasonin...
> During large-scale RL, they did two things for each sample: > > 1. Specified the desired effort level in the system message. > 2. Adjusted the cost assigned to each generated token. > [...] > > Then, at inference time, Inkling receives a system message such as Thinking effort level: 0.8, and adjusts its token usage accordingly.
I wish I could link it, but I had also read another strategy for reasoning effort, perhaps the most naive implementation, described as whether or not you ignored or accepted the end of sequence token, n-times, n being some count of subsequent ignores.
Now how true that is, I don't know. You could reasonably implement this locally with an inference provider, but I don't know any that do, or I haven't used them at least.
I may also have the technical detail incorrect, but it did come down to essentially "continuing inference beyond a natural termination."
Now how true that is, I don't know. You could reasonably implement this locally with an inference provider, but I don't know any that do, or I haven't used them at least.
I may also have the technical detail incorrect, but it did come down to essentially "continuing inference beyond a natural termination."
Sounds similar to Qwen3 but the exact opposite: "stopping thinking before a natural termination"
https://arxiv.org/pdf/2505.09388
> Thinking Budget. An additional advantage of Thinking Mode Fusion is that, once the model learns to respond in both non-thinking and thinking modes, it naturally develops the ability to handle intermediate cases—generating responses based on incomplete thinking. This capability lays the foundation for implementing budget control over the model’s thinking process. Specifically, when the length of the model’s thinking reaches a user-defined threshold, we manually halt the thinking process and insert the stop-thinking instruction: “Considering the limited time by the user, I have to give the solution based on the thinking directly now.\n</think>.\n\n”.
^-^
https://arxiv.org/pdf/2505.09388
> Thinking Budget. An additional advantage of Thinking Mode Fusion is that, once the model learns to respond in both non-thinking and thinking modes, it naturally develops the ability to handle intermediate cases—generating responses based on incomplete thinking. This capability lays the foundation for implementing budget control over the model’s thinking process. Specifically, when the length of the model’s thinking reaches a user-defined threshold, we manually halt the thinking process and insert the stop-thinking instruction: “Considering the limited time by the user, I have to give the solution based on the thinking directly now.\n</think>.\n\n”.
^-^
You have a restaurant and have 2 types of chef.
One is the sous chef , who knows recipes and cooks fast, handle most dishes as quick as possible. He does not think much. That is your medium effort llm in charge.
One is the head chef, who is inspecting, pause, consider edge case. Slower but catches what sous chef missed. That is your high effort llm in work.
One is the sous chef , who knows recipes and cooks fast, handle most dishes as quick as possible. He does not think much. That is your medium effort llm in charge.
One is the head chef, who is inspecting, pause, consider edge case. Slower but catches what sous chef missed. That is your high effort llm in work.
But I was thinking Chef is the LLM model. So I was wondering if it is the same model, what actually changes the effort? Is that part of the inference?
So are you saying that effort level isn't a parameter to an LLM, but that each level actually corresponds to a slightly differently tuned LLM?
I think the analogy here is time. Especifically performance time. Head chef performs slower. High effort is longer inference time. For LLMs, the ability to think in more detail requires more time. Maybe this will change in the future but it is the present now.
llamacpp has the basics: reasoning-budget limits thinking token output and reasoning-message thats injected when budget is exhausted. client can set these per request so dynamics are possible.
models dont seem to care if you cut them off.
models dont seem to care if you cut them off.