Lesson 4: Review like an attacker

Lesson 4: Review like an attacker

Objective

Both suites pass, both skins replay fully blocked, and you hold a written record of what was reviewed, what a scanner did and did not catch, and what nobody has reviewed yet.

Why this lesson exists

blocked=9/9 is easy to over-read. It means nine specific attacks are blocked. A real attacker tries a tenth. Security review is the habit of asking what the tests did not try — and writing the answer down instead of assuming it.

The lesson

Run the maintainer pair and both replays yourself:

python verify.py starter
python verify.py solution
python starter/attack_replay.py scenario/team-notes
python starter/attack_replay.py scenario/clinic-notes

The suite only replays team-notes. clinic-notes uses different payloads — x' OR 1=1 --, an injected owner instead of an injected query, scans/../../secrets.txt, an empty key. If your fixes are mechanisms, it prints blocked=8/8 without any change. If it does not, one of your fixes was a filter.

Then do the review the suite cannot do:

1. Run a static scanner, then read what it missed. If you have Ruff (the repository’s development dependencies include it), run its flake8-bandit rules on your fix:

   ruff check --isolated --select S starter/notes_app.py
   

   Your fix should report nothing. For comparison, on the original starter Ruff 0.16.10 reports S608 (SQL built from strings), S324 (MD5, twice), and S301 (pickle.loads) — and nothing for the path join or the "dev-secret-change-me" fallback. If you committed a baseline, run the same command on it to see this yourself. A scanner is a fast first pass that catches known patterns; it is not a review.

2. Attack one thing the suite never tried. Pick one: a search query with Unicode case (É vs é), an attachment name that is a directory, a session token with a valid signature over a JSON list, a password that is 10 000 characters long. Predict, run, and write down the result.

3. Read the diff for what it removed. git diff the starter against your baseline. Confirm pickle and md5 are gone and that no except Exception swallows a SecurityError.

Exercise

Write the security evidence record for this change:

• Verified: the commands you ran, both replays, the scanner output.
• Not verified: at least — authentication and authorization as a whole, how the session token is transported (cookie flags, TLS), rate limiting on login, key storage and rotation, dependency vulnerabilities, and every attack not in attacks.json.
• Known limits: anything you found in step 2 that the contract never promised, and the password-hash choice (PBKDF2 rather than Argon2id).

Checkpoint

Run python verify.py. This checkpoint’s code prints once the starter suite passes; the maintainer pair above must both be green for the gate to mean what it says. You pass the lesson when you can answer:

1. Did clinic-notes pass without changes? If not, which fix was a filter?
2. Which two flaw classes did the scanner miss on the original starter, and why would a pattern-based tool miss them?
3. What did your extra attack do, and is it on your “not verified” list?

Expected evidence

Both green runs, both replay outputs, the scanner output, and your three-line evidence record.