Foundations — LLM literacy for PMs

Chapter 1 · Phase 0 · ~3–5 hours · practice in Week 1

What you will know after this chapter

Why tokens drive cost and architecture · what context windows force you to design around · when your team ships inference vs trains · why grounding prevents hallucinations · what embeddings do · how to decide between RAG and fine-tuning

1. Tokens — the unit of LLM computation

LLMs do not read words or characters. They read tokens — chunks that fall somewhere between characters and words, determined by a tokenizer trained on the model's corpus. A token is typically 3–4 characters of English, but this varies by language and content type.

Token

The smallest unit of text a model processes. A tokenizer converts raw text into a sequence of integer IDs before the model sees anything. The same process runs in reverse to produce output text.

Tokenizer

An algorithm (commonly BPE — Byte Pair Encoding) that segments text into tokens. Each model family has its own tokenizer. GPT-4o, Claude, and Gemini all tokenize the same sentence differently.

Tokenization example

The sentence "Retrieval-augmented generation reduces hallucinations." tokenizes roughly as:

Retrieval-augmented generation reduces hallucinations.

7 tokens. The hyphen, spaces, and suffixes each become their own token or part of one. Code and non-English text can be 2–5× more tokens per word.

Why tokens matter for PMs

💰

Cost

Every API call is billed on input + output tokens. A 10-page document ≈ 3,000 tokens. At $15/M tokens, 1M such documents = $45,000 in input cost alone before any output. Token efficiency is a product lever, not just an engineering concern.

⏱️

Latency

Output tokens are generated one at a time (autoregressive). 500 output tokens at 50 tok/s = 10 seconds. Streaming makes this feel faster, but token count drives wall-clock time for the user.

🧱

Architecture

Once you hit the context limit, you cannot simply "add more tokens." You must summarize, chunk, or retrieve. Every architectural decision downstream (RAG, agents, summaries) is a response to token limits.

PM rule of thumb

1 word ≈ 1.3 tokens (English). 1 page ≈ 300–400 tokens. Code is denser. PDFs with tables or images (as text) can be 5–10× denser. Always estimate token budgets before scoping a feature.

2. Context window — the model's working memory

Every LLM call has a context window: the maximum number of tokens the model can "see" in a single request. Everything outside the window is invisible. There is no background memory, no implicit file system. What you put in the prompt is literally all the model knows.

Context window

The hard upper limit on tokens per request (input + output combined for some models, or just input for others). Also called "context length" or "context limit." Exceeding it causes an API error or silent truncation.

Current model context sizes (2025)

Model	Context	~Pages of text
GPT-4o	128K tokens	~300 pages
Claude 3.5 Sonnet	200K tokens	~450 pages
Gemini 1.5 Pro	1M tokens	~2,200 pages
Llama 3.1 70B	128K tokens	~300 pages
Mistral Large	32K tokens	~70 pages

Larger context ≠ better quality on long inputs. "Lost in the middle" research shows models perform worse on content in the middle of a long context vs. at the beginning or end.

What context limits force you to design around

When content exceeds the window

Raw content
e.g. 500-page wiki

→

Exceeds limit

→

Option A: Summarize

Option B: Chunk + retrieve (RAG)

Option C: Fine-tune (teach behavior)

PM implication

"Just dump the whole knowledge base into the prompt" never scales. Every LLM product that handles large corpora needs an explicit strategy for what fits in the window and how to retrieve what doesn't.

3. Inference vs training — what product teams actually ship

Training

Adjusts model weights on large datasets
Requires GPUs, weeks of compute, billions of examples
Done by model labs (OpenAI, Anthropic, Google, Meta)
Pre-training: learns language from scratch on the internet
Fine-tuning: adapts a pretrained model to a specific task or style
PM teams almost never run pre-training; fine-tuning is occasional

Inference

Running a trained model to produce output
Every API call, every user interaction is inference
Weights are frozen — the model does not learn from your calls
This is what product teams ship
Cost = tokens × price per token
Latency = time to first token + time to complete

Training vs inference pipeline

Training data
trillions of tokens

→

GPU cluster
weeks / months

→

Model weights
frozen file on disk

⬇ model weights are then loaded for inference

Your prompt
input tokens

→

Model (inference)
milliseconds–seconds

→

Response
output tokens

4. Hallucination vs grounding

Hallucination

A model generating text that is fluent and confident but factually wrong. The model does not "know" it is wrong — it produces the most statistically likely token sequence given its weights and the prompt, regardless of factual accuracy.

Grounding

Anchoring a model's response to verifiable sources. A grounded response cites specific retrieved documents, database rows, or tool outputs — and the claim can be traced back to that source.

Why models hallucinate

LLMs are trained to predict the next token — not to be accurate. If a plausible-sounding but false completion is statistically consistent with the training data, the model will generate it. Confidence in tone does not correlate with factual correctness.

Common hallucination patterns

Fabricated citations — plausible-looking paper titles, authors, and DOIs that do not exist
Wrong numbers — revenue figures, dates, statistics that are off by an order of magnitude
Invented APIs — function signatures that look correct but are not in the actual library
Confident synthesis — combining two real facts into a third claim that is false

Grounding techniques

RAG (Retrieval-Augmented Generation)

Retrieve relevant document chunks and inject them into the prompt. Instruct the model to answer only from the provided context. See the RAG chapter.

Tool / function calling

Give the model a tool that queries a database or API. The model calls the tool; you execute it; actual data comes back. The answer is grounded in live data, not training weights.

Constrained output + verification

Ask for structured output (JSON with citations). Parse and verify the cited source actually says what the model claims. Reject responses where citations do not match.

Self-critique prompting

Ask the model to review its own answer: "List any claims above that might be uncertain. For each, state the basis." This does not eliminate hallucination but surfaces low-confidence sections for human review.

5. Embeddings — semantic coordinates

Embedding

A dense vector (list of ~768–3072 floating-point numbers) that represents the semantic meaning of a piece of text. Texts with similar meaning have vectors that are close together in high-dimensional space, measured by cosine similarity.

Embeddings are produced by a separate embedding model (not the same as the generation model). OpenAI's text-embedding-3-large, Cohere's embed-v3, and Google's text-embedding-004 are common choices.

Embedding intuition — semantic space

"dog" → [0.21, -0.44, 0.88, ..., 0.03] ← 1536 dims
"puppy" → [0.19, -0.41, 0.91, ..., 0.05] ← very similar vector
"cat" → [0.18, -0.38, 0.82, ..., 0.11] ← nearby
"quantum" → [-0.61, 0.22, -0.13, ..., 0.77] ← far away

cosine_similarity("dog", "puppy") ≈ 0.97 ✓ very similar
cosine_similarity("dog", "quantum") ≈ 0.12 ✓ unrelated

What embeddings enable

Semantic search — find documents by meaning, not keyword. "How do I reset my password?" matches "account recovery instructions" even with no word overlap.
RAG retrieval — the core mechanism: embed a query, find nearest document chunks, inject them into the prompt.
Clustering and classification — group support tickets by topic, detect duplicate issues, build recommendation systems.
Anomaly detection — items that are outliers in embedding space are unusual (fraud signals, out-of-scope queries).

6. RAG vs fine-tuning — which bet to make

This is one of the most common PM decisions in AI product work. The answer depends on what problem you are actually solving.

RAG — Retrieval-Augmented Generation

Model weights are not changed
At query time, relevant documents are retrieved and added to the prompt
Best for: factual recall from a corpus (docs, wikis, support articles)
Works with up-to-date data (add to the index, not retrain)
Auditable: you can show the user the source
Cheaper: no training compute, API calls only
Use when: "The model needs to know facts from our data"

Fine-tuning

Model weights are adjusted on your training examples
At inference, no extra context needed — behavior is baked in
Best for: teaching a style, format, or behavior — not recall
Data must be collected, cleaned, and formatted (expensive)
Stale: retraining needed when data changes
Harder to audit: why did it say that?
Use when: "The model needs to respond differently — always structured JSON, always terse, always formal medical language"

Common PM mistake

Proposing fine-tuning to improve factual accuracy. Fine-tuning adjusts behavior and style — it does not reliably improve factual recall. A model fine-tuned on your documentation will still hallucinate; it will just hallucinate in your brand voice. Use RAG for facts, fine-tuning for format/behavior.

Decision framework

Question	RAG	Fine-tuning
Does the model need to know facts from your data?	✓ Yes	✗ No
Does the data change frequently?	✓ Yes	✗ Painful
Do users need to see sources/citations?	✓ Natural fit	✗ Hard
Do you need a consistent output format/persona?	Possible via system prompt	✓ Better
Do you have 1,000+ curated examples?	Not needed	✓ Required
Can you afford retraining costs?	Not applicable	$ to $$$$

In practice

Most production AI features combine both: a fine-tuned model (for format and behavior) with RAG (for current facts). But if you must pick one to start, RAG is almost always the right first step because it is faster to build, easier to debug, and does not require labeled training data.

Ready to apply these concepts?

Practice → Week 1 Next: Prompting →

Test your understanding

What is a token, and why does token count matter for a PM?v

A token is roughly 3-4 characters. Token count sets the cost of every API call and determines how much text fits in the context window. PMs track tokens to budget cost, design UX within limits, and explain why the model 'forgets' earlier messages.

What is the difference between inference and training? Which one do you pay for per request?v

Training is the one-time process of learning weights from data. Inference is running the trained model to produce output. You pay for inference — every user request is billed by input + output tokens.

A user asks 'what was our revenue last quarter?' The model answers confidently but incorrectly. Is this hallucination? What is the root cause?v

Yes — the model produced plausible-sounding text not grounded in fact. Root cause: the model has no access to your actual revenue data. The fix is grounding: RAG (retrieve real numbers before answering) or tool calling (query your database live).

What is the difference between an embedding and a token?v

A token is a discrete text unit. An embedding is a dense numeric vector representing the meaning of text — used for similarity search in RAG. Tokens are input units; embeddings are semantic representations.

When would you recommend RAG over fine-tuning? Give a concrete example.v

RAG: when the model lacks your facts (e.g., internal docs, current product catalog). Fine-tuning: when you need to change behavior or format (e.g., brand voice). Fine-tuning does not reliably implant facts — it teaches style.

What is a context window limit and how does it constrain product design?v

The context window is the max tokens the model sees at once. It constrains conversation history length, how many RAG documents can be injected, and how long user documents can be. Longer windows cost more. Design must decide what to include and what to drop.

What does temperature control, and when would you set it near zero vs near one?v

Temperature controls output randomness. Near zero: deterministic — good for SQL generation, factual Q&A, structured output. Near one: varied and creative — good for brainstorming, drafting, generating diverse options.