Chatbots

Chapter 6 · Phase 5 · practice in Week 9 and Week 10

What you will know after this chapter

How multi-turn conversation context works and what it costs · UX patterns (streaming, error states, fallbacks) · what to log and why · rate limiting and cost management · an incident playbook for when the chatbot misbehaves

1. Multi-turn conversations — context management

A chatbot is stateless on the model's side. Every API call is independent. To maintain conversation history, your application must send all previous messages with every new request.

How multi-turn context is maintained

Turn 1 — API call contains:

[system: "You are a support bot..."]
[user: "How do I reset my password?"]

Turn 2 — API call contains:

[system: "You are a support bot..."]
[user: "How do I reset my password?"]
[assistant: "Click Forgot Password on the login page..."]
[user: "I don't see that option on mobile"]

Token accumulation problem

Each turn adds more tokens to the context. A 20-turn conversation may consume 5,000+ tokens of context before a single word of response is generated. Long conversations become expensive and eventually hit the context limit.

Context management strategies

Strategy	How it works	Trade-off
Sliding window	Keep only the last N turns. Older turns drop off.	Simple. Model loses early context — may "forget" something said at turn 1.
Summarization	Periodically summarize old turns into a compact summary. Replace old messages with the summary.	Preserves key facts. Summary quality affects recall.
Selective memory	Extract and store key entities (name, issue type) from the conversation. Inject as structured context.	Precise. Requires entity extraction pipeline.
Session reset	Start fresh context after task completion. Each task is independent.	Clean, but requires detecting task boundaries.

2. UX patterns for LLM chatbots

Streaming responses

LLMs generate tokens one at a time. Without streaming, the user waits for the full response before seeing anything — often 3–10 seconds. With streaming, tokens appear as generated, making the experience feel much faster even if total time is unchanged.

How it works: instead of waiting for the complete response before sending anything back to the user, the server sends tokens as they are generated — word by word, essentially. The user's UI renders each token as it arrives, creating the typewriter effect you see in ChatGPT. Technically this is done over a Server-Sent Events (SSE) or WebSocket connection, but as a PM you need to know one thing: always ship streaming for any LLM response that takes more than 2 seconds. A 6-second streaming response feels acceptable. The same 6-second response without streaming feels broken.

PM rule

Time to first token is the UX metric that matters, not total generation time. Streaming can halve perceived latency without changing actual model speed at all. It is almost always worth the engineering effort.

Error states and fallbacks

Graceful API errors

Never show raw API errors to users ("openai.error.RateLimitError"). Map all errors to user-friendly messages: "I'm having trouble right now — please try again in a moment."

Timeout handling

Set a max response timeout (e.g., 30s). If exceeded, return a partial response or a fallback: "I started generating a response but it's taking too long. Here's what I have so far..."

Out-of-scope detection

Define what your bot should not answer. Detect off-topic queries (classifier or LLM-as-judge) and redirect: "I'm a support assistant — I can't help with that, but you can contact our billing team at..."

Confidence threshold

For RAG-backed chatbots, if retrieval recall is low (no high-similarity chunks found), respond with reduced confidence: "I don't have specific information on that — here's what's closest in our docs, but please verify."

3. What to log — observability for LLM chatbots

Without logging, you cannot debug incidents, measure quality, retrain, or audit. Log everything.

Field	Why it matters
`session_id`	Link all turns in a conversation for debugging
`user_id`	Attribute cost and behavior to specific users
`timestamp`	Correlate with deployments, incidents, A/B test periods
`model_version`	Track behavior changes across model updates
`full_prompt`	Reproduce any call exactly for debugging
`full_response`	Audit content, train reward models, detect policy violations
`input_tokens` / `output_tokens`	Cost attribution, quota enforcement
`latency_ms`	SLA monitoring, p95/p99 alerting
`retrieved_chunk_ids`	Debug RAG retrieval quality
`user_feedback`	Thumbs up/down for quality measurement
`finish_reason`	Detect truncations (`max_tokens` hit) or content filter triggers

4. Rate limiting and cost management

Per-user quota enforcement

Every user's token consumption should be tracked against a daily cap that corresponds to their plan tier. The logic works as follows: before making any API call, look up how many tokens this user has already consumed today. Add the tokens you are about to use. If the total exceeds their plan limit, block the request and surface a friendly message rather than letting the call go through and incurring unexpected cost.

Reasonable daily limits by tier: Free users — 10,000 tokens/day (enough for ~20 short conversations). Pro users — 100,000 tokens/day. Enterprise — negotiate based on contract. These numbers vary by product but the principle is the same: every plan has a ceiling and hitting it should be a deliberate, communicated experience, not a surprise error.

PM decision

Define quota limits before launch, not after your first surprise API bill. Token quotas are a pricing and trust lever — too restrictive and users feel blocked; too loose and a single abuser can drain your monthly budget overnight.

Cost-aware context window management

As a conversation grows, the context you send on each turn grows with it — every previous message gets included. Left unchecked, a 30-turn conversation might include 8,000 tokens of history even before the user's new question. The solution is a trimming strategy: always keep the system message (it defines the bot's behavior), then walk backwards through recent turns keeping them in the budget, dropping the oldest turns first when you run out of space.

This means the bot may "forget" things said 20 turns ago. Design your UX with this in mind — if something is important, the user should be able to pin it, or your system should extract key facts and inject them explicitly rather than relying on raw message history.

5. Incident playbook

When a chatbot misbehaves in production, you need a pre-defined response process. Improvising during an incident wastes time and increases damage.

P0 — Active harm

Trigger: Model generating dangerous, illegal, or severely harmful content.
Response: (1) Kill switch — disable the chatbot immediately, show static maintenance message. (2) Preserve logs for investigation. (3) Page on-call. (4) Draft public statement if user-facing.

P1 — Systematic errors

Trigger: Model giving wrong answers consistently (e.g., wrong prices, wrong instructions).
Response: (1) Add a disclaimer banner. (2) Identify root cause (prompt regression? model update? stale knowledge base?). (3) Hotfix or rollback prompt version. (4) Re-run evals to verify fix.

P2 — Degraded quality

Trigger: User satisfaction score drops, thumbs-down rate increases, but no safety issue.
Response: (1) Sample recent conversations to identify pattern. (2) Check if model version changed. (3) Run eval suite. (4) Iterate on prompt. No rush to kill switch.

P3 — Latency regression

Trigger: p95 latency exceeds SLA (e.g., > 5 seconds).
Response: (1) Check provider status page. (2) Check if prompt length grew (token cost). (3) Consider smaller model or caching. No user-visible quality impact.

← RAG

Practice → Week 9 Next: Prototypes →

Test your understanding

What is the context window management problem in multi-turn chatbots?v

Each new turn appends to conversation history sent to the model. Eventually the history exceeds the context window limit. You must decide what to drop or compress without losing the conversation thread.

Name two strategies for managing conversation history when approaching the context limit.v

(1) Sliding window: keep only the last N turns, drop oldest. Simple but loses early context. (2) Summary compression: periodically ask the model to summarize earlier turns and replace them with the summary. More complex but preserves key context in a smaller footprint.

What should a chatbot do when its confidence is low?v

Say so explicitly: 'I am not certain about this — you may want to verify with [source].' Or escalate to a human agent. Confident wrong answers erode user trust permanently once discovered.

Why is streaming important for chatbot UX?v

Streaming sends tokens as they are generated instead of waiting for the full response. This reduces perceived latency — users see the first word in 0.5s instead of waiting 5s. Users rate streaming experiences as faster and less frustrating even if total generation time is the same.

What metrics would you track to know if your chatbot is working well?v

Deflection rate, resolution rate, escalation rate, thumbs-up/down per conversation, session abandonment rate, and latency p50/p95. Also track token cost per resolved conversation.

A chatbot goes offline at 2am. What does a good incident response plan include?v

Automatic alerting when error rate exceeds a threshold. On-call runbook: (1) check provider status page, (2) check error logs, (3) route to fallback (static FAQ or human queue), (4) communicate status to users. A fallback path is mandatory — never depend on a single LLM endpoint with no backup.