🔹 Episode 3: Chain of Thought. The Internal Monologue and Digital Draft of an LLM

“An agent that has only executive arms (ReAct), but does not know how to plan its steps, is just an impulsive automaton. Today we open the door to the internal monologue of an LLM and understand why AI needs to ‘think’ on a digital sheet of paper before taking action.”

Welcome to the third chapter of our journey. So far, we have understood the skeleton and the communication connectors. But if you want to move from a chatbot that “guesses” words to a system that solves complex problems, you need to understand how we taught models not to give answers off the cuff.

The secret behind today’s huge leap in intelligence does not lie in magic, but in a very down-to-earth concept: Inference-time compute. Put simply, we give the model time and space to build its reasoning on an internal draft before delivering the final answer to us.

🔹The Need for a Draft: Why LLMs Cannot Calculate Everything Instantly

A language model is, at its core, a probabilistic token predictor. It reads a text and calculates which token is most likely to come next. If you ask it to solve a complicated logic problem directly, in a single pass, it has a good chance of getting it wrong.

Think of a simple analogy: if I ask you to multiply 4739 x 842 in your head and give me the result instantly, you are likely to fail. Your brain does not have enough working memory to process all the digits simultaneously. But if I give you a sheet of paper, you start writing down the intermediate steps.

The sheet of paper is not just storage; it becomes an extension of your thinking process. You read what you wrote in Step 1 so that you can work out Step 2. In a similar way, Chain of Thought (CoT) allows an LLM to generate intermediate reasoning steps and use them as context for subsequent steps. When the model builds its reasoning this way, we can think of it as having a digital draft. Instead of jumping directly to the conclusion, it can use intermediate steps to arrive at the final answer.

🔹The Internal Monologue: Visible vs. Hidden (The Security Wall)

At first, Chain of Thought became known mainly as a “prompt engineering” technique (the famous “Let’s think step by step”). The model explicitly generated the intermediate reasoning steps, and these could appear directly in the response. Today, newer reasoning models can use a Hidden Internal Monologue, in which part of the reasoning process is no longer shown to the user.

The interface may show you a “Thinking…” animation while, behind the scenes, the model may generate a large number of reasoning tokens before formulating the final answer. Why would Silicon Valley companies want to keep this draft behind the curtain?

• 1. Preventing Distillation: If the draft were made fully public, reasoning chains could be collected at scale and used to train or distill other models. Keeping them hidden therefore protects part of the model’s technological advantage.
• 2. Safety and Control: Internal reasoning may contain intermediate hypotheses, failed attempts, or formulations that do not represent the final answer. Separating this process from the response shown to the user allows the system to apply safety filters and policies before delivering the final information.

🔹The Bill and the Context: Reasoning Tokens

This “deep thinking” comes at a cost. When you use a model with reasoning capabilities, even if the final answer is only two sentences long, the model may consume hundreds or thousands of tokens for the intermediate reasoning process. These are called Reasoning Tokens.

They are tokens generated and processed by the model in the Cloud. They consume computing power but are not displayed in the final response. In other words, the bill is not necessarily calculated based on how long the text you see is. A question that receives a two-sentence answer may require a much longer internal process.

This is also where an important distinction from the context window comes in. Reasoning Tokens consume resources and a computational budget during processing, but this consumption should not be confused with permanently keeping those tokens in the conversation history. Exactly how they are accounted for and how they interact with the context limit depends on the model and the API being used.

Your cognitive infrastructure is fully configured. Let’s move to action!

Stay Free! Stay Hidden! Stay Autonomous!

⚙️ Appendix: AI Engineering Mini-Course (Technical Clarifications)

• Chain of Thought (CoT): A technique in which a model generates intermediate reasoning steps to solve complex problems.
• Reasoning Tokens: Tokens generated and processed during the internal reasoning of certain models. They consume computing resources and, depending on the model and API, may be accounted for separately or as part of the output tokens, without being displayed to the end user.
• Inference-time Compute: The computing power consumed while the AI “deliberates” before providing an answer. Unlike training, the model uses these resources to solve the specific problem in front of it.
• What Is a Token: It is not necessarily a word, but a processing unit. AI models break text down into smaller pieces. A common word may be a single token, while a complex technical term may be split into several pieces.
• Tokens in Chat vs. API (The Local Agent): In a chat interface, you usually pay a fixed subscription. In an API, the cost is generally calculated based on the number of tokens processed:
  • Input Tokens: Everything the Local Agent sends to the model (the request + instructions + JSON schema and, where applicable, the conversation history).
  • Output Tokens: The tokens generated by the model for the response. For reasoning models, these may also include reasoning tokens, depending on how the provider accounts for them.
  • Billing: In an API, the cost may include tokens used for internal reasoning, even if they do not appear in the response displayed to the user.
• Tokenization Differences: Each model has its own “dictionary” (tokenizer). This means that the way the same text is “broken down” into tokens can differ from one technology to another. Choosing a model through an API is not just a matter of performance; it can also matter when optimizing costs.