Chapter 4: LLMs are stateless

You may be surprised to learn that large language models are completely stateless.

When you send a request to an LLM, the model does not remember who you are, what you asked ten seconds ago, or what it previously replied. Programmers have to manually build memory capabilities using host-side code.

The amnesia experiment

To see this stateless nature firsthand, let us run a simple two-part experiment.

First, we send an initial request introducing ourselves:

const response1 = await sendPost("https://acme-ai.com/v1/chat", {
    sender: "user",
    message: "My name is Alice."
});

console.log(response1);
// { sender: "bot", message: "Nice to meet you, Alice!" }

The model receives the greeting, processes the tokens, and returns a friendly reply.

Now, a minute later over the exact same network connection, we ask the model a follow-up question in isolation:

const response2 = await sendPost("https://acme-ai.com/v1/chat", {
    sender: "user",
    message: "What is my name?"
});

console.log(response2);
// { sender: "bot", message: "I don't know your name. We are at the start of our conversation." }

Even though the two calls were made only seconds apart to the exact same API, the model has completely forgotten the earlier exchange.

Why are LLMs stateless?

To understand why this happens, consider what happens on the remote AI server during inference:

1. Ephemeral Connection: The server opens an incoming HTTPS connection and parses the JSON request body.
2. Pure Computation: The input tokens pass forward through billions of transformer weights. The neural network computes probabilities for the next tokens.
3. Stream & Terminate: Once the stop token is generated, the server streams the response back to your client and immediately closes the connection.
4. Memory Clearance: The execution thread terminates. No database record is created, no session cookie is written, and no internal state persists on the model cluster.

The LLM behaves like a pure mathematical function: $f(\text{input}) = \text{output}$. Given the exact same weights and input tokens, it executes without any awareness of the past or future.

The Scribe's Illusion: Creating conversational memory

If the model remembers nothing, how do tools like ChatGPT or Claude maintain flowing conversations?

The answer is that the client application acts as the scribe.

// The client accumulates the full transcript in host memory
const chatHistory = [
    { sender: "user", message: "My name is Alice." },
    { sender: "bot", message: "Nice to meet you, Alice!" },
    { sender: "user", message: "What is my name?" }
];

const response3 = await sendPost("https://acme-ai.com/v1/chat", {
    messages: chatHistory
});

console.log(response3);
// { sender: "bot", message: "Your name is Alice!" }

By sending the entire accumulated transcript on every turn, the model re-reads the full history from top to bottom. To the human user, it appears as though the AI has a sharp, continuous memory. In reality, you are simply re-feeding the entire story on every single request.

Why statelessness is an engineering superpower

While sending the full transcript might initially feel inefficient, statelessness provides massive architectural advantages:

• Fault Tolerance and Idempotency: If a network request drops or times out, you can safely retry it without worrying about corrupting remote server state.
• Conversation Branching: Because history lives in your code as a plain array, you can fork conversations, rewind turns, edit past prompts, or generate multiple variations from any checkpoint.
• Complete Auditing: You have absolute visibility into every piece of data the model saw when making a decision, making compliance and auditing straightforward.
• Dynamic Context Injection: You can surgically inject new documentation, tool results, or system instructions directly into the transcript right before sending the payload.

In the next chapter, we will look at how to use system prompts to shape model persona, enforce constraints, and establish guardrails.