How does ChatGPT work?

ChatGPT is used by hundreds of millions of people, but most of them have a slightly wrong picture of what it actually does. The common assumption is that it searches the web, or that it "thinks" through your question the way you might. Neither is true. What it actually does is both simpler and more surprising than that, and understanding it will change how you use it.

The short version: ChatGPT is a type of machine learning model that can perform natural language processing tasks, such as generating and classifying text, answering questions, and translating text. The engine underneath it is called a large language model, or LLM. These models are trained on large amounts of text data, and during that training they learn patterns and relationships within language, which they then use to predict the likelihood of a next word based on the words that came before. Word by word, that is how every response you have ever received was built.

The autocomplete that read the whole internet

The word-prediction framing sounds almost too simple, so it helps to think about what that prediction is built on. LLMs are trained on massive datasets containing text from a variety of sources, including books, articles, websites, and more. The scale is genuinely hard to grasp: training starts with billions or trillions of words from books, articles, websites, code and other text sources.

From all of that, the model does not memorise individual sentences. Instead, it learns patterns, like how sentences are structured, which words often appear together, how jokes are phrased, and what makes a sentence sound polite or angry. Those patterns get compressed into what are called parameters, which are essentially numerical weights. Neural networks can be very small, or they can be very large, as is the case with LLMs, which might have millions of neurons with many hundreds of billions of connections between them, with each connection having its own weight.

When you type a message, the model runs through all those weights to figure out what word should come next. Then it does it again for the word after that. Think of it like autocomplete, except with no user input once it gets started. This is repeated over and over for each token until the response is complete.

Tokens, not words

You will hear the word "token" a lot in discussions about LLMs. A token is not quite the same thing as a word. When you send ChatGPT a prompt, it starts by breaking the text into tokens. Tokens are small units of text, sometimes a full word, sometimes just part of one. The word "unbelievable" might be split into two or three tokens. Common short words like "the" are usually a single token. This matters because models have a limit on how many tokens they can handle at once, which is why very long conversations can cause them to "forget" what you said at the start.

The transformer: what makes modern AI different

LLMs use a particular neural network architecture called a transformer, which is designed to process and generate textual data in sequence. The transformer was developed by Google in 2017 and transformed (the pun is unavoidable) the field almost overnight. What made it special is a mechanism called attention, which lets the model weigh how relevant each word in your prompt is to every other word. So when you write "the bank by the river", the model can figure out you mean the riverbank rather than a financial institution, because the surrounding words pull its attention in the right direction.

The transformer architecture is particularly effective at understanding and generating human-like text, using mechanisms called attention and self-attention to process and generate language. Before transformers, earlier models would read text word by word in order, which made it hard to connect a pronoun near the end of a sentence to the noun it referred to at the beginning. Transformers handle the whole sequence at once, which is why the responses feel coherent over longer passages.

Why the same question gets different answers

You might have noticed that if you ask ChatGPT the same thing twice, you get slightly different replies each time. This is intentional. If the model always picked the word with the highest probability, you would get monotone, dull and uncreative text. Instead, it sometimes picks from lower-ranked words to keep things interesting and varied, and this randomness is the reason that if you ask ChatGPT the same thing twice, you will get different answers.

This randomness is captured in a parameter called "temperature" that tweaks how often the model picks those lower-ranked words. Turn the temperature right down and you get very consistent, predictable output. Turn it up and you get more variety, though also a greater risk of the model going off in an odd direction. Most consumer chatbots sit somewhere in the middle.

Pre-training is only the first step

Predicting the next word on internet text produces a model that can write fluently, but it does not automatically produce a model that is helpful, honest, or safe to talk to. A raw language model trained only to predict text will happily complete a sentence in whatever direction the statistics point, including harmful ones. So OpenAI added a second stage of training on top.

Reinforcement learning from human feedback (RLHF) is an approach to training and fine-tuning a large language model, allowing it to follow human instructions correctly. With RLHF, the model can understand the user's intention even if it is not described explicitly.

In practice, this meant hiring human trainers who rated different model responses, marking some as better and others as worse. RLHF involves training a reward model to simulate human language preferences, and then using reinforcement learning to train the language model to generate responses that increasingly align with those preferences. The model learns, in effect, what kinds of answer humans approve of, and steers itself in that direction. Learning from human feedback has led ChatGPT to produce more desirable, non-toxic, and factual outputs.

This is also where guidelines about what ChatGPT will and will not do come from. The model does not have a built-in moral compass; it has been rewarded for certain kinds of responses and nudged away from others.

What ChatGPT does not do

It is worth being clear about what is not happening. ChatGPT does not think in ideas like humans; it calculates how likely the next word is to be a certain term. There is no understanding in the human sense, no internal experience, and no memory between separate conversations (unless you have turned on a memory feature). When a session ends, the model retains nothing about you.

It also does not look things up in real time by default. The model's knowledge comes from its training data, which has a cut-off date. Anything that happened after that cut-off is simply unknown to it, unless it has been given a tool that allows web browsing, which some versions now have as an optional add-on.

LLMs have limitations, such as generating biased or incorrect information if the training data contains biases or errors. They can also produce plausible-sounding but incorrect or nonsensical answers, and they may lack true understanding beyond pattern recognition. This is the root cause of AI "hallucinations", where a model states something confidently that is simply wrong. The model may assemble a coherent answer, but a fact stated in that answer may be incorrect. Such errors are referred to as hallucinations. The fluency of the writing gives no signal as to the accuracy of the content underneath it.

Why this matters for how you use it

Once you understand the mechanics, some practical lessons follow naturally. ChatGPT is excellent at tasks that rely on language patterns: summarising, drafting, explaining, reformatting, translating, brainstorming. It is less reliable for factual lookups, recent events, precise numbers, or anything where being wrong is costly. The model produces text that is statistically plausible, not text that is verified.

Giving it more context in your prompt genuinely helps, because the more words it has to work with, the better the attention mechanism can figure out what you actually need. A vague prompt gets a vague answer not because ChatGPT is being unhelpful, but because probability calculations on sparse input produce uncertain results.

And if something it tells you matters, check it. The writing will always sound confident. That confidence is a property of how language models generate text, not a signal that the underlying fact is correct.

If this one was useful, there's plenty more on the site. Pieces on how AI works, plus coverage of AI news, the downsides included. All free to read, no account needed.

References

1. What Are Large Language Models (LLMs)?, IBM
2. What Are Large Language Models?, Oracle
3. What is a large language model?, Dataiku
4. Large Language Models (LLMs) with Google AI, Google Cloud
5. AI Demystified: Introduction to large language models, Stanford University IT
6. How Does ChatGPT Work?, Zapier
7. How Does ChatGPT Work? Full Breakdown, TrustRadius
8. Exploring Reinforcement Learning with Human Feedback, Kili Technology
9. What is Reinforcement Learning from Human Feedback?, DataCamp
10. What is Reinforcement Learning from Human Feedback (RLHF)?, TechTarget