Prototypes
Chapter 7 · Phase 6 · practice in Week 11 and Week 12
1. The reproducibility problem
An LLM prototype that "works great in the demo" but fails during the stakeholder review, or produces different output every run, is worse than useless — it erodes trust in the technology. Reproducibility is not optional.
2. What to pin for reproducibility
gpt-4o-2024-08-06 not gpt-4o. Aliases point to whatever the provider considers "current" — and it changes without warning. A model version bump can change behavior significantly.temperature=0 (deterministic mode). The model will still vary slightly across provider infrastructure, but output will be far more consistent. Use temperature > 0 only for creative tasks where variation is desirable.seed parameter. When specified with temperature=0, the same seed produces identical output. Set seed=42 (or any fixed value) for full reproducibility in demos and evals.v1.0, commit to git, and include the version in your demo notes. If a demo goes wrong, you can check out the exact prompt that was used.gpt-4o-2024-08-06), the temperature setting (0 for demos), the seed value if used, the prompt version tag, and the max output token limit. Log this at the top of every output you share. When a demo goes wrong two weeks later, you will be grateful you can reproduce the exact conditions.
3. Token cost estimation before you build
Build a cost model before writing a single API call. This shapes your architecture decisions and prevents budget surprises. The math is simple — you just need to commit to doing it.
Worked example — RAG-backed support chatbot
Assume you are using GPT-4o at approximately $5 per million input tokens and $15 per million output tokens (2025 pricing). Each conversation turn has these components:
| Component | Tokens | Notes |
|---|---|---|
| System prompt | ~500 | Static instructions, sent every call |
| Retrieved RAG chunks | ~1,500 | 3 chunks × 500 tokens each |
| User question | ~50 | Typical short question |
| Model response (output) | ~300 | A paragraph-length answer |
Total input per call: 2,050 tokens. Total output: 300 tokens. Cost per call: (2,050 × $5 / 1,000,000) + (300 × $15 / 1,000,000) = $0.0153 per conversation turn. At 1,000 queries per day, that is ~$460/month. At 10,000 queries per day, ~$4,600/month. These numbers should inform your pricing, tier limits, and model selection before you write a single line of code.
2. How many tokens of context am I injecting per call? (RAG chunks, history)
3. What is my expected output length?
4. At target query volume, what is the monthly cost?
5. Is there a cheaper model that achieves the same quality for this task?
Model cost vs quality tradeoffs (2025)
| Task | Try first | Upgrade to if needed |
|---|---|---|
| Simple classification, routing | GPT-4o mini / Claude Haiku | GPT-4o |
| Summarization | GPT-4o mini | GPT-4o |
| RAG question answering | GPT-4o mini | GPT-4o |
| Complex reasoning, analysis | GPT-4o | o3, Claude 3 Opus |
| Code generation | GPT-4o | o3, Claude 3.7 Sonnet |
| Long document processing | Gemini 1.5 Pro | Claude 3.5 Sonnet (200K) |
4. Demo script structure
A demo without a script becomes an improvised performance. Write the script before the demo.
State the problem (30 sec)
"Today, support agents take 4 minutes to find the right article for a customer question. We want to cut that to 30 seconds."
Show the baseline (1 min)
Show the current experience — the 4-minute manual lookup. This is the before state. Do not skip this; without it, the demo has no impact.
Run the prototype on your curated inputs (3–5 min)
Use 3 specific inputs you have tested and know work well. Show the best case, a medium case, and one graceful failure ("I don't have information on that").
Quantify what you saw (30 sec)
"On these 3 cases, we went from 4 minutes to under 30 seconds. On our eval set of 50 real tickets, we hit 87% accuracy." Use your eval data.
State the next step and open risks (1 min)
What is the next unknown to de-risk? "We need to test on adversarial inputs and integrate with the ticketing API." Name the risks proactively — it shows rigor.
5. Prototype to production handoff
A prototype proves feasibility. To hand off to engineering, you need to provide the right artifacts.
- Pinned system prompt — the exact version, in version control
- Eval suite — 50–100 test cases with expected outputs and current scores
- Token cost model — spreadsheet showing cost at target scale
- Failure mode documentation — known cases where the prototype fails and why
- Latency baseline — p50/p95 latency from your testing
- Architecture diagram — what components the production version needs (API, DB, vector store, logging)
- Model pinned version — exact version string, not an alias
Test your understanding
Further reading & watching
- BlogChip Huyen — Cost & Latency Analysis (Part I)Concrete cost breakdown for LLM prototypes — token pricing, inference costs, and buy-vs-build tradeoff. Read before budgeting a prototype.
- BlogChip Huyen — Prompting vs Fine-tuning (Part I)Helps decide your prototype architecture: when prompting is enough vs when fine-tuning is needed.
- ToolStreamlit — Build ML apps fastThe fastest way to turn a Python prototype into a shareable web UI. No front-end skill required.
- ToolGradio — Instant UI for AI demosSimilar to Streamlit. Used widely for research demos and rapid prototyping.
- EssayPaul Graham — Do Things That Don't ScaleThe canonical essay on why manual, prototype-grade approaches are how great products start.