Ask an LLM a multi-step math problem and demand an immediate answer, and it might get it wrong — even if it "knows" all the individual facts needed to solve it. Ask the same question but add "think step by step," and accuracy often jumps dramatically. That gap is what chain-of-thought prompting is all about. Here's how it works and why it matters so much for reasoning tasks.
The Basic Definition
Chain-of-thought (CoT) prompting is a technique that encourages a language model to generate intermediate reasoning steps before arriving at a final answer, rather than jumping straight from question to conclusion. Instead of producing just the final output, the model "shows its work" — breaking a complex problem into a sequence of smaller logical steps, much like a student solving a math problem on paper rather than blurting out a guess.
A Simple Example
Without chain-of-thought:
Prompt: "A store had 23 apples. They sold 8 in the morning and received a delivery of 15 more in the afternoon, then sold 12 more. How many apples are left?"
Response: "18" (possibly wrong — the model jumped straight to an answer)
With chain-of-thought:
Prompt: "A store had 23 apples. They sold 8 in the morning and received a delivery of 15 more in the afternoon, then sold 12 more. How many apples are left? Think step by step."
Response: "Start with 23 apples. After selling 8 in the morning: 23 − 8 = 15. After receiving 15 more: 15 + 15 = 30. After selling 12 more: 30 − 12 = 18. The store has 18 apples left."
In this simple case both might land on the same number, but as problems get more complex, the step-by-step version becomes dramatically more reliable — because the model is working through the problem sequentially rather than trying to compute the final answer in one uninterrupted leap.
Why This Actually Improves Accuracy
This connects directly back to how LLMs generate text, covered earlier in this series: token by token, with each new token predicted based on everything generated so far. When a model jumps straight to a final answer, it has to essentially compute the entire solution "in its head" within the probability distribution for a single token (or a very short span of tokens) — with no opportunity to build on intermediate results.
When a model generates reasoning steps first, each step becomes part of the context for predicting the next one. The model is, in effect, using its own generated text as a kind of scratchpad — offloading intermediate results into the sequence itself, rather than trying to hold the entire computation implicitly within its internal representations. This dramatically increases the chance of arriving at a correct final answer, especially on tasks involving arithmetic, logic, or multi-step reasoning.
Where Chain-of-Thought Helps Most
CoT prompting isn't equally useful everywhere. It tends to produce the biggest improvements on:
- Math word problems — where multiple calculations need to happen in sequence
- Logical reasoning tasks — puzzles, deductions, or problems with several interdependent conditions
- Multi-step planning — tasks that require breaking a goal into ordered sub-steps
- Complex decision-making — weighing multiple factors or criteria before reaching a conclusion
For simple factual questions or straightforward requests, chain-of-thought adds little value and just makes responses longer than necessary.
Two Common Ways to Trigger It
Explicit instruction (zero-shot CoT). Simply adding a phrase like "think step by step," "explain your reasoning," or "work through this systematically" to an otherwise normal prompt is often enough to trigger noticeably better reasoning — no examples required. This is sometimes specifically called zero-shot chain-of-thought, since it combines the CoT effect with the no-examples approach covered in the earlier zero-shot prompting post.
Demonstrated reasoning (few-shot CoT). Rather than just instructing the model to reason step by step, you provide one or more worked examples in the prompt that themselves show full reasoning chains — not just a final answer, but the entire path to it. The model then tends to mimic that same step-by-step structure when solving the new problem, combining the pattern-matching strength of few-shot prompting (from the previous post) with the structured reasoning benefit of CoT.
Why This Matters for Trust and Verification
Beyond accuracy, chain-of-thought prompting has a second, often underrated benefit: transparency. When a model shows its reasoning, you can actually inspect the intermediate steps — catching a flawed assumption, a miscalculation, or a misunderstood part of the question, rather than just seeing a final answer with no visibility into how it was reached.
This matters a lot given what was covered in the earlier "Why LLMs Hallucinate" post — a fluent-sounding final answer offers no built-in signal about whether it's actually correct. A visible reasoning chain gives you something to actually check: does each step logically follow from the last? Does the reasoning use accurate information? A confidently wrong final answer is much easier to spot when the flawed step in the reasoning is visible, rather than buried inside an opaque, single-token leap to a conclusion.
An Important Caveat: Reasoning Isn't Always Faithful
It's worth being precise about what's actually happening. The reasoning steps a model generates aren't necessarily a transparent window into some separate internal "thought process" the way a human's spoken reasoning might be. The model is still fundamentally predicting plausible-sounding next tokens — it's just now predicting a plausible sequence of reasoning steps, which happens to make correct answers more likely as a side effect, but doesn't guarantee that the stated reasoning was the actual causal path to the answer, or that it's free from its own errors.
In other words: chain-of-thought reliably improves the quality of many outputs, but the visible reasoning shouldn't be treated as a perfectly faithful account of "what the model was really thinking" in some deeper sense.
Chain-of-Thought and Modern "Reasoning Models"
More recent models take this idea further by being specifically trained to generate extended internal reasoning before producing a final answer — sometimes automatically, without needing an explicit "think step by step" instruction from the user at all. These are often described as reasoning models, built and trained specifically to allocate more computation toward working through complex problems step by step before committing to a final response, particularly for math, coding, and multi-step logic tasks. This represents an evolution of the same core chain-of-thought idea — baked directly into how certain models are trained and how they operate by default, rather than relying entirely on prompt phrasing.
Trade-Offs to Keep in Mind
Chain-of-thought prompting isn't free:
- Longer outputs — reasoning steps add tokens, which (as covered in the tokens and pricing posts) increases both cost and response time.
- Not always necessary — for simple, well-represented tasks, it can add unnecessary length without meaningfully improving accuracy.
- Not a guarantee of correctness — a model can still generate confident, well-structured, but ultimately flawed reasoning; step-by-step doesn't mean error-free.
The Bottom Line
Chain-of-thought prompting improves an LLM's performance on complex reasoning tasks by encouraging it to generate intermediate steps before a final answer — turning its own output into a working scratchpad rather than forcing the entire solution into a single leap. It's a simple technique, often as easy as adding "think step by step" to a prompt, but it produces some of the most reliable accuracy gains available in prompt engineering, especially for math, logic, and multi-step problems. It also offers a practical side benefit: visible reasoning gives you something concrete to check, rather than just a confident-sounding answer you have to take on faith.