Chapter 3.5: LLMs are stateless

You may be surprised to learn that LLMs are stateless by default. Programmers have to manually build memory capabilities using code. They build these capabilities around the LLM itself. But the LLM itself remembers nothing at all.

Previously, we sent the following request to an LLM:

const botResponse = sendPost("https://acme-ai.com/v1/chat", {
    sender: "user",
    message: "Hello, World!"
})

console.log(botResponse);
// { sender: "bot", message: "Why, hello there!" }

So far, we've sent the LLM a request and it has sent us back a response.

Now, let's test the LLM's memory. Does it remember what it said?

Here's the code we'll be executing.

const botResponse = sendPost("https://acme-ai.com/v1/chat", {
    sender: "user",
    message: "What did you just say?"
})

What do you think the response will be?

Think about it. I'll wait.

...

If you guessed "nothing," you're on the right track. Let's console log it and see exactly what it said.

console.log(botResponse);
// { sender: "bot", message: "I didn't say anything. We are at the beginning of the conversation." }

This tells us that the LLM doesn't have an in-built memory. It also tells us that we can have a "conversation" with this AI, and that the AI is trained to have conversations. We'll come back to conversations later.

As You will see in the next chapter that the stateless nature of LLMs greatly simplifies our work and gives us greater control over the LLM.