Lesson 4: Verify like an engineer, review like a skeptic

Lesson 4: Verify like an engineer, review like a skeptic

Objective

You complete the full evidence loop — verify both directions, exercise all three scenario skins end to end, review the final diff for scope creep and side effects — and write a three-line verification record that says what is proven and what is not.

Why this lesson exists

“Tests pass” is the beginning of verification, not the end. The suite pins nine behaviors. It does not tell you whether the agent edited files it should not have, whether the tool leaves junk behind on real data, or whether the report numbers are right for your domain. That judgment is the part that stays human.

The lesson

Run the whole loop, in order:

python verify.py starter          # your finished implementation: 9/9
python verify.py solution         # reviewed reference: 9/9
git diff --stat                   # (or your tool's equivalent) what actually changed?

Then run the tool like a user, once per skin (each writes report.json inside its scenario folder — check .gitignore handles it, then inspect the output):

python starter/report_tool.py scenario/excel-report
python starter/report_tool.py scenario/data-monitor
python starter/report_tool.py scenario/api-tool

Check three things in each report.json: total = valid + invalid, every errors[i].reason names a real row you can find in the data file, and one group total you recompute by hand.

Now the skeptic’s diff review. For every change the agent made beyond the contract lines, ask: which test forced this? If the answer is “none”, it is scope creep — revert it and rerun the suite. Also check for side effects the tests cannot see: files created outside scenario/, network calls (there should be zero), and behavior on an empty data file (try it — what happens?).

Exercise

Write the verification record. Three lines, honest:

Verified: <commands run, date, result>
Not verified: <what the suite does not cover — e.g. encoding of real exports, huge files>
Known limits: <what would make this break — e.g. schema change in the source system>

Compare it with REVIEW.md at the course root — same discipline, maintainer version.

Checkpoint

You pass when: all three skins run clean, your hand-check of one group total matches, the diff review found (and reverted) at least one piece of unforced scope — or you can defend why every change was forced — and your three-line record exists.

Expected evidence

The verification record, the git diff --stat output, and one hand-checked group total. This is the artifact you would attach to a real change.