Learn
Learn hubFoundationsPromptingSQL & dataVisualizationRAGChatbotsPrototypesAgents & launch FAANG prep
Weekly practice
0102030405060708091011121314

Visualization

Chapter 4 · Phase 3 · practice in Week 6

What you will know after this chapter
The hypothesis-first workflow · which chart type to use for which question · how to validate AI system behavior with plots · common chart mistakes that hide the truth · Python/pandas patterns for PM analysis

1. Hypothesis-first visualization

The most common PM mistake with data is opening a BI tool, generating many charts, and then finding a story. This is data fishing — you will always find something that looks interesting, and you will have no way to know if it is real or artifact.

The correct workflow:

1

Write the hypothesis first

"I believe that Pro users who use the RAG feature in their first week retain at 90-day rates 20% higher than those who do not." Be specific about the effect direction and magnitude.

2

Choose the chart type that would confirm or deny it

For this hypothesis: a retention curve (cohort analysis) comparing two groups. The chart either shows the gap or doesn't.

3

Generate the plot

Run the query, create the chart. Look for the pattern you hypothesized. Check if it exists.

4

State what the chart proves or disproves

Write one sentence under every chart: "This chart shows X, which [confirms/does not confirm] the hypothesis because Y." If you cannot write this sentence, the chart is decorative, not analytical.

2. Chart type selection guide

Question typeChart typeExample
How does X change over time?Line chartDaily active users over 90 days
How does X compare across categories?Bar chartRAG query latency by model version
What is the distribution of X?HistogramDistribution of response latencies
What is the distribution of a count?Box plotToken usage per user per day, p25/median/p75/p99
Are X and Y correlated?Scatter plotChunk size vs retrieval accuracy
How does a cohort retain over time?Cohort / retention chartWeek 1 users: % still active at Week 2, 4, 8
Where do users drop off in a flow?Funnel chartPrompt sent → result received → user upvoted
What fraction of X is each category?Stacked bar or areaBreakdown of error types over time

3. Visualizing AI system behavior

For AI products specifically, you need charts that diagnose model behavior — not just business metrics. Here are the three most important ones to build for any LLM feature.

📊

Latency distribution chart

Load your inference logs into a spreadsheet or BI tool. Plot a histogram of response times in milliseconds. Draw vertical lines at p95 and p99 so you can instantly see your tail latency. Add a second panel showing median latency over time (by hour or day) to catch regressions. If your p99 is 3× your p50, you have a reliability problem worth investigating even if the average looks fine.

💰

Token usage per user (cost outlier chart)

Sum input + output tokens per user, then sort descending. Plot the top 20 users as a bar chart. Calculate what percentage of total token cost comes from the top 10% of users — this number is almost always shockingly high (often 60–80%). Use this to find abuse, set per-user quotas, and justify tiered pricing. A healthy cost distribution should be relatively flat; a spiky one means you have no guardrails.

🎯

RAG retrieval recall over time

For each week, calculate what fraction of queries had the correct answer-containing chunk in the top-K results. Plot this as a line with a target threshold line at 85%. A declining recall line tells you your index is going stale, your chunk strategy is degrading on new content, or a recent embedding model change broke retrieval. This chart is your early warning system for RAG quality decay.

4. Common chart mistakes that hide the truth

Mistakes to avoid

5. The one-sentence chart rule

Every chart you put in a doc, deck, or PR should have a title that states the conclusion, not just the data source:

❌ Weak title

  • "RAG latency by week"
  • "User retention chart"
  • "Token usage over time"

✓ Strong title (states the finding)

  • "RAG p99 latency exceeded SLA in weeks 3 and 7"
  • "Pro users retain 2× better at 90 days than Free"
  • "Token cost grew 3× while DAU grew 1.5× — efficiency regression"
← SQL & data
Practice → Week 6 Next: RAG →

Test your understanding

What is a hypothesis-driven chart, and how does it differ from exploratory analysis?v
A hypothesis-driven chart starts with a specific claim and builds one chart that proves or disproves it. Exploratory analysis scans data broadly to discover patterns. PM deliverables should be hypothesis-driven; data science exploration precedes them.
When would you use a bar chart vs a line chart?v
Bar chart: comparing discrete categories at a single point in time. Line chart: showing change over a continuous time axis. Key rule: lines imply continuity and trend; bars imply comparison of separate groups.
What are two ways a chart can be misleading with accurate data?v
(1) Truncated y-axis: starting at 95 instead of 0 makes a 2% change look like a cliff. (2) Cherry-picked time window: choosing the date range that only shows the favorable trend. Always show the full relevant context.
A stakeholder asks to 'see all the data' in a dashboard. Why is this often wrong?v
Showing too many metrics forces the viewer to decide what matters — and they may focus on the wrong thing. Every chart should support a stated decision. A dashboard full of numbers is a data dump, not a communication tool.
What is the PM's primary responsibility in a data visualization project?v
Define the decision the visualization must support and specify the audience. The PM owns the question; the analyst owns the answer. A PM who says 'show me everything' has abdicated this responsibility.

Further reading & watching