Problem statement

Same incident program as Week 02—now you must make reasoning auditable, catch overconfidence, and specify when the model should refuse or call tools. You also ship a 10-question eval sheet so QA and PM agree what “pass” means.

Outcomes

• Chain-of-thought instruction (raw chain may stay hidden from users).
• Self-critique / verification pattern.
• Tool-use framing: when to query SQL vs answer from chat.
• Eval sheet with pass/fail rules + 2 safety/refusal rows.

Deliverables

1. eval_sheet_10.md — ten rows: Question | Expected property | Pass/Fail rule.
2. Prompt pack using CoT + self-critique on ../week-02/data/incident_sample.md.
3. One-page tool-use policy: which tools (SQL, search, calendar) for which user intents.

Links

• Phase 1 mini deliverable

Required

• [ ] Provider docs: function calling / tools — schema, when the model invokes vs declines.
• [ ] Provider docs or guide: JSON mode / structured outputs.

Recommended

• [ ] One article or doc on LLM evals (golden set, regression).

Time box

~2 hours.

Use incident text from week-02/data/incident_sample.md.

Task A — Chain-of-thought (design only)

Write a prompt that forces explicit reasoning steps before the final executive summary. Optionally hide raw chain from end users in your product note. Save prompt_cot.md.

Task B — Self-critique

Same incident: add a second stage— “List assumptions, then challenge them; revise summary if needed.” Save prompt_selfcritique.md + paste before/after summary if different.

Task C — Tool-use framing

Write tool_policy.md answering:

• When should the assistant refuse numeric claims without querying practice-env DB?
• When is “read Support queue +22%” acceptable without SQL?

Include one example user question routed to SQL and one routed to no SQL.

Task D — Eval sheet (required)

Create eval_sheet_10.md with 10 rows:

| # | User question or stress case | Expected property | Pass/Fail rule |

Include:

• At least 2 safety/refusal cases (e.g. exfiltration, medical/legal).
• At least 2 format/check cases (JSON keys present, word limit).
• At least 1 “must cite or qualify uncertainty” case tied to the incident.

---

Submit: my-submissions/week-03/.

60 minutes. Closed notes for Section A; open only your eval sheet for Section B if you wrote it from memory.

Section A — Concepts (20 min)

1. Explain tool-use framing in your own words.
2. Why might you hide raw chain-of-thought from end users?
3. What makes an eval question checkable vs subjective?

Section B — Build under pressure (25 min)

From scratch (no paste from prior weeks), write 5 eval rows for any AI summarization feature—must include 1 refusal case and 1 format case.

Section C — Mini scenario (15 min)

User asks: “Prove enterprise revenue dropped because of this incident.” Describe how your system should behave without inventing numbers—name tools or responses you’d allow.

---

Self-grade: Compare with Phase 1 checklist.

Concepts & curated links: read the Learn chapter Prompting first (topic path, not calendar-only). This file is the weekly practice bank — extra drills and a checklist on top of Readings → Practice → Exam.

---

1. Practice bank

#	Problem	Data / materials	Expected outcome
P1	Build CoT + self-critique prompts for the incident sample.	week-02/data/incident_sample.md	prompt_cot.md, prompt_selfcritique.md
P2	Write tool-use policy: when SQL vs no-SQL for numeric claims.	practice-env/data/cloudnote.db (schema only)	tool_policy.md
P3	Create 10-row eval sheet: Question | Expected property | Pass/Fail.	Feature of your choice	eval_sheet_10.md
P4	Include ≥2 refusal/safety rows in eval sheet.	None	Same file — highlighted rows

Align with PRACTICE.md for full wording.

---

2. Week checklist

• [ ] T1–T5 skim complete
• [ ] P1–P4 + PRACTICE.md deliverables
• [ ] EXAM.md timed attempt