FAANG Interview Prep — AI PM & TPM

Start with a one-sentence analogy: "An LLM is a very sophisticated autocomplete — trained on so much text that it learned patterns of language, reasoning, and knowledge." Then explain the three constraints that matter for product: (1) Context limits — the model only knows what you put in the prompt; it has no persistent memory. (2) Hallucination — it produces plausible-sounding text whether or not it's true. (3) Cost and latency — every word in and out costs money and time. Tie each constraint to a product implication: "This is why we can't just dump our entire knowledge base into the prompt — that's why we need RAG."

Diagnose first: Sample 20–50 affected conversations. Classify the hallucinations — are they factual errors, invented citations, wrong numbers, or off-scope responses? Each type has a different root cause. Common causes by type: Factual errors → model's training data is wrong or outdated (fix: RAG). Wrong numbers → model generating statistics from pattern-matching (fix: SQL tool calling). Invented citations → model completing a plausible citation pattern (fix: constrain to retrieved sources only). Mitigations: (1) Grounding via RAG — model must answer from retrieved documents only. (2) Tool use — for any fact that must be precise, give the model a tool to look it up live. (3) Self-critique prompt — ask the model to flag uncertain claims. (4) Output validation — parse the response and verify cited sources actually say what the model claims.

RAG when: the model needs to know facts from your corpus (docs, tickets, wikis), the data changes frequently, you need to show citations, or you're prototyping fast. Fine-tuning when: you need to change how the model responds — its format, tone, persona, or output structure — and you have hundreds or thousands of labeled examples. The key insight to land: "Fine-tuning teaches behavior, not facts. If your problem is that the model doesn't know your documentation, fine-tuning won't reliably fix that — it will just make it hallucinate more confidently. Use RAG for facts. Fine-tuning for format and style."

• Is the eval set representative? 90% on 20 curated examples is meaningless. Is it 500 randomly sampled real user queries?
• What does failure look like? Is the 10% wrong in a safe, minor way or in a dangerous, trust-destroying way?
• What's the accuracy on adversarial / edge-case inputs? Malicious prompts, off-topic questions, ambiguous queries?
• How was ground truth defined? Who labeled the correct answers, and are labelers consistent?
• Does 90% translate to user satisfaction? A technically correct answer that is too long, too formal, or unhelpful is still a failure.
• What is the regression baseline? What did users do before this feature? Is 90% better or worse?

• Clarify (1 min): Who are the users (end users or internal support agents)? What's the primary goal (deflection rate or resolution quality)? What existing support stack do we have?
• Scope the problem (1 min): Pick the highest-leverage starting point. "I'll focus on tier-1 FAQ deflection for end users, which handles ~60% of ticket volume."
• Architecture decisions (2 min): RAG over the knowledge base (not fine-tuning — we need citation support and the KB changes weekly). Streaming for UX. Escalation path to human agent when confidence is low.
• Top 3 risks (2 min): (1) Hallucinated answers erode trust. Mitigation: answer only from KB, cite sources. (2) Bot handles sensitive billing issues badly. Mitigation: route billing queries to human. (3) KB is stale. Mitigation: auto-flag articles older than 90 days.
• Metrics (2 min): Deflection rate, answer accuracy (human-reviewed sample), escalation rate, user satisfaction score post-interaction. Launch threshold: deflection ≥ 40% with accuracy ≥ 90% on eval set.

• Layer 1 — Offline evals (before any traffic): 100–500 test cases with known correct answers. Measure exact-match accuracy for structured outputs, LLM-as-judge for open-ended outputs. Gate: must pass before any user sees the feature.
• Layer 2 — Shadow mode: Run the new model in parallel with the existing experience. Compare outputs without serving them to users. Catch regressions silently.
• Layer 3 — Human review sample: For every eval metric, have domain experts review a 50-item random sample to validate that your automated scoring is calibrated correctly.
• Layer 4 — Live metrics at 1%: Thumbs up/down rate, escalation rate, session abandonment, latency p95. Alert thresholds must be defined before launch, not discovered post-launch.
• Layer 5 — Adversarial testing: A red team tries to jailbreak, confuse, or extract sensitive information. Run before any public traffic.

Bad: Showing AI output as authoritative fact with no indication it could be wrong ("Your account balance is $2,340"). Bad: showing a raw confidence number ("87.3% confident") that users cannot interpret. Bad: hiding the AI nature of the response entirely.

Good: Framing output as a suggestion, not a fact ("Based on your usage, it looks like..."). Good: surfacing the source ("From your plan documentation:"). Good: graceful degradation ("I'm not certain about this — you may want to verify with support"). Good: showing what the AI couldn't answer clearly ("I couldn't find information about X in your plan").

Key principle: Uncertainty UX is a trust calibration problem. You want users to trust the system appropriately — neither over-trusting (leads to acting on wrong information) nor under-trusting (ignores genuinely useful output).

Step 1 — Diagnose the competitor's feature: Is their 3-week version actually good, or is it a demo? Ship speed is meaningless if quality is poor. Analyze their feature honestly.

Step 2 — Find the 80/20: What is the smallest version of our feature that serves the core user need? Could we ship a constrained MVP in 4–6 weeks that covers 80% of cases?

Step 3 — Define the non-negotiables: What quality bar must we hit before shipping anything? For AI features, that usually means: eval accuracy ≥ X%, no safety violations, clear disclosure to users. These are not negotiable even under competitive pressure.

Step 4 — Communicate the trade-off clearly: "We can ship a limited version in 6 weeks that handles these 3 use cases, or a full version in 3 months. Here is what we give up in each scenario." Let leadership make the call with full information.

• 1. Is the signal real? Check if the feedback UI changed (a UI change can bias thumbs-down rate). Check if the sample size is statistically meaningful. Check if the drop happened across all platforms or one surface.
• 2. Did anything change? Model version update, prompt change, upstream data change, new traffic source (different user segment arrived). Check deployment logs against the timestamp of the drop.
• 3. Which query types degraded? Segment by query category, user plan, device, language. A drop in one segment with others flat means the issue is specific, not systemic.
• 4. Sample the bad responses: Read 20 low-rated conversations from after the drop. What pattern do you see — wrong facts, unhelpful tone, wrong language, irrelevant retrieval?
• 5. Run your eval suite: If you have offline evals, run them on the current prompt + model combination. Compare to the last passing run.

Bad north stars: Suggestions shown (supply metric, not demand). Suggestions clicked (could be clicked and immediately deleted). DAU using Copilot (engagement without quality).

Better: Accepted-and-retained suggestion rate — the fraction of AI suggestions that the user accepted AND did not delete within the next 60 seconds. This captures that the suggestion was useful enough to keep, not just curiosity-clicked.

Even better, tied to user outcome: Documents completed per user per week for Copilot users vs non-users. This ties the feature to the actual job-to-be-done (finishing writing tasks faster).

Key principle: Your north star should be something that goes up when the AI is genuinely helping users, and stays flat or goes down when it's a toy or a distraction.

Diagnose first: The ratio of cost-per-user doubled. That means either (a) each user is making more queries, (b) each query is consuming more tokens, or (c) a small number of power users are driving disproportionate cost. Check all three.

If average tokens per query grew: Did context window size grow (more history, larger RAG chunks)? Did output length grow (prompt change, new feature)? Did you add a new expensive step (e.g., a second LLM call for self-critique)?

Product fixes: (1) Smarter context management — trim conversation history aggressively. (2) Per-user quotas — cap daily token consumption by plan tier. (3) Model tiering — route simple queries to a cheaper model (GPT-4o mini vs GPT-4o). (4) Caching — if the same question is asked repeatedly (FAQ patterns), cache the answer. (5) Chunk size optimization — smaller, more precise retrieval chunks reduce noise tokens injected into the prompt.

Indexing pipeline (offline): Documents → chunker (splits text into 256–1024 token segments) → embedding model (converts each chunk to a dense vector) → vector database (stores vectors + original text).

Query pipeline (online): User question → same embedding model → vector similarity search → top-K most relevant chunks → injected into LLM prompt with the question → LLM generates answer citing retrieved chunks.

Failure points to name: (1) Wrong retrieval — the most similar vector isn't the most relevant chunk. Usually a chunking or embedding quality problem. (2) Stale index — document was updated but index wasn't re-run. (3) Chunk boundary problem — the answer spans two chunks; neither alone is sufficient. (4) Hallucination despite context — model ignores the retrieved context and generates from training weights. (5) Lost-in-the-middle — quality degrades for content in the middle of a long context; most important chunks should be first or last.

• Check provider status page first. LLM API providers have outages and degraded performance windows. 80% of latency spikes are provider-side.
• Check if input token count grew. Did a recent change increase prompt size? Larger prompts take longer to process and longer to generate responses for. Compare average input tokens before and after the spike.
• Check if a new LLM call was added. Did a recent deploy add a second model call (e.g., self-critique, reranking)? Two calls in series doubles latency floor.
• Check if it's all users or a segment. If only enterprise users (who have longer conversation histories) are affected, it's a context window size problem.
• Rollback plan: Have a feature flag that can switch the model version or revert to the previous prompt in under 10 minutes. If the cause is unclear and users are impacted, roll back first, investigate second.

• What problem does fine-tuning solve that prompting cannot? If the answer is "improve factual accuracy," fine-tuning is the wrong tool. If the answer is "consistent JSON output format" or "adopt our brand voice," it may be appropriate.
• How many labeled examples do you have? Fine-tuning needs hundreds to thousands of high-quality (input, ideal output) pairs. Who creates and reviews them?
• How will you evaluate the fine-tuned model against the base model? What's the eval suite? What metrics define success?
• What is the data governance plan? What proprietary data is used? Does it contain PII? What are the data processing agreements with the model provider?
• What is the ongoing maintenance cost? When the base model updates, will you need to re-fine-tune? Who owns that work?
• Have you tried a strong system prompt + few-shot examples first? Fine-tuning is rarely the first solution. Can the behavior be achieved at inference time instead?

• Principle of least privilege for tools: Only give the LLM tools it needs. If it's a Q&A bot, it should have no write access to databases, no ability to send messages.
• Input sanitization: Strip or escape known injection patterns ("ignore previous instructions," "you are now DAN") before sending to the model. Not foolproof but raises the bar.
• Output validation: If the model generates a tool call, validate the arguments against an allowlist. A user cannot inject a command that produces a call to a tool that doesn't exist or with arguments outside expected types.
• Separate user input from instructions: Clearly delimit what is system instruction vs user content in the prompt. Use structured formats (XML tags, JSON) rather than raw text insertion.
• Red team before launch: Have a team actively try to jailbreak the system. This is not optional for any externally-facing AI feature.

State the framework you'll use upfront: "I'll score each on impact, confidence, and effort." Then ask two clarifying questions: (1) What is the company's current AI maturity — are we building AI foundations or shipping to millions of users? (2) Which customer segment is the primary focus?

Impact: Which feature creates the most user or business value? A text-to-SQL tool that cuts analyst report time by 80% may have higher ROI than a chatbot with moderate deflection. Quantify where possible.

Confidence: Which feature has the clearest path to quality? Chatbots are deceptively hard to do well (hallucinations, off-topic, multi-turn complexity). Text-to-SQL has a tighter, more measurable success criterion.

Effort and risk: Which feature requires the least new infrastructure? Reuse of existing RAG pipeline, logging, evals infrastructure reduces cost and time.

Land on a recommendation with a rationale, not a tie: "I'd start with text-to-SQL because it has a clear success metric (query accuracy), builds on infrastructure we already need, and generates immediate analyst productivity wins that are easy to measure."

• Start with API: When you are exploring, prototyping, or don't yet know if the use case has product-market fit. API lets you validate in weeks without infrastructure investment.
• Stay with API at scale: When quality of third-party models is sufficient, data privacy requirements are met, and cost is manageable. Most products stay here permanently.
• Consider in-house when: (1) Cost at scale becomes prohibitive and you have the volume to justify training/hosting. (2) Data cannot leave your infrastructure (regulated industries, PII). (3) Your specific domain has such unique patterns that general models consistently underperform and fine-tuning is not enough. (4) You need real-time latency (< 100ms) that API calls cannot achieve.
• Key insight: "In-house" rarely means training from scratch — it means fine-tuning or running open-source models (Llama, Mistral) on your infrastructure. The decision is API vs self-hosted open-source vs full custom training, in increasing order of cost and control.

Situation: Set the context — what feature, what team, what the goal was.

Task: What was your specific role and what did you own?

Action: What did you build? What evaluation did you run before launch? Be specific — "we had 50 eval cases with 88% accuracy" is better than "we tested it thoroughly." This is where interviewers look for eval rigor.

Result (the failure part): What went wrong? Be precise about the failure mode — "hallucination rate in production was 3× higher than on our eval set because our eval set wasn't representative of long-tail queries." Vague failures ("it didn't work as well as expected") are weak.

Learning: What changed? Did you build better evals? Add grounding? Implement human-in-the-loop? This is the part interviewers remember most. Show systematic improvement, not just a fix.

The most compelling versions of this story have: (1) Genuine internal enthusiasm for the feature — otherwise killing it isn't hard. (2) Clear, specific data that contradicted the enthusiasm — not vibes, not "I had a bad feeling," but numbers. (3) A crisp framing of the data to stakeholders — you translated technical results into business consequences. (4) An alternative you proposed — you didn't just kill, you redirected energy productively.

Common AI-specific angles for this story: eval accuracy was too low to maintain user trust, the cost per query made unit economics impossible, a user study showed the AI answer was ignored or mistrusted 60% of the time, or the failure mode was high-severity (wrong medical/financial information) with no mitigation path.

The most effective approach: lead with the consequence, not the mechanism. Don't say "the model doesn't know post-training-cutoff data." Say "if a customer asks about a product we launched last month, the AI will give wrong information — and that erodes trust faster than just not having an AI feature."

Then offer the solution in business terms: "The fix is a retrieval system that fetches current product information before answering — it adds 3 weeks but means we can keep the AI accurate even as the product catalog changes."

Show in your story that you anticipated the skepticism and prepared: you had data, examples of the failure mode, and a clear trade-off framing ready before the meeting.