FlyPython
English
Sign inStart a challenge
English中文

Security review for AI-written Python · Challenge 05

Lesson 5: Review your own AI-written code

Checkpoint l0510 ptsself-reported — code from verify.pyAny coding agent + an attack-replay harness

Take the five questions and the scanner pass to a repository you own: find where input crosses a trust boundary, check each crossing for a mechanism, and write down what you did not review.

Doing this with your agent? One sentence starts the whole course:

Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course `secure-ai-generated-code`.

Lesson 5: Review your own AI-written code

Objective

You review a repository you own — ideally code an assistant wrote — with the same method, fix at least one finding with a test that proves it, and record what you did not review.

Why this lesson exists

The notes service was a sandbox with planted flaws and known answers. Your code has neither. What transfers is the method: find each place input crosses a trust boundary, ask who controls it and what they want, and check that the code uses a mechanism rather than a hope.

The lesson

  1. List the trust boundaries. Search for where input enters: web routes, CLI arguments, uploaded files, queue consumers, os.environ reads. Write the list before reading any of them closely.
  2. Run the scanner. ruff check --select S (or the scanner your team already uses) gives you the pattern-level findings in seconds. Triage each one: real, false positive, or accepted risk — and write down why.
  3. Ask the five questions the scanner cannot. For each boundary:
    • Does any input reach SQL, a shell, or a template other than as a bound parameter or argument list?
    • Does any input become part of a filesystem path, and is containment checked after resolving?
    • Is there any default, fallback, or example secret in the code path that runs in production?
    • Are passwords or tokens hashed with a slow, salted function and compared with hmac.compare_digest?
    • Does anything deserialize client data with pickle, marshal, shelve, or an unsafe YAML loader?
  4. Fix one finding with a test first. Write the attack as a failing test, fix it with a mechanism, watch the test pass. Ask your agent for the fix after the test exists, and review its diff against the mechanism, not just the test.
  5. Record. Verified / not verified / known limits. “Not verified” includes every boundary you listed but did not read.

If you find something serious in code other people depend on, follow your organization’s or the project’s security disclosure process before you open a public issue.

Exercise

Ask your coding agent to list the trust boundaries in your repository, then compare its list with yours. Note what each of you missed.

Checkpoint

This checkpoint is self-attested. When you can answer the questions below, claim it with python verify.py --attest l05:

  1. How many trust boundaries did you list, and how many did you actually read?
  2. Which finding did you fix, and what test proves it?
  3. Which scanner finding did you mark as a false positive or accepted risk, and why?

Expected evidence

Your boundary list, the scanner triage, the failing-then-passing test, and the three-line record. Keep findings about unfixed vulnerabilities out of public places until they are fixed.

Hints

Stuck? Open one at a time.

Hint 1 — What this checkpoint tests

Self-attested: you confirm you reviewed code you own with this method. The evidence is your findings list and your evidence record — not a code we can check.

Hint 2 — Start where input enters

Request handlers, CLI arguments, file uploads, message consumers, anything that reads an environment variable. Each is a trust boundary.

Hint 3 — Fix with a test first

For each real finding, write the attack as a failing test before the fix — exactly as this course's suite did.

Submit

Done? Record it.

Use the 8-character code printed by python verify.py. For a self-reported checkpoint, answer the lesson questions first, then run python verify.py --attest l05.

Youbrowser

Submit the claim code

Your agentauto-submit

Let the agent solve and submit

Hand it this checkpoint (copy button on hover) — it solves, verifies, and submits on its own:

Work on the FlyPython challenge "Security review for AI-written Python" (course id course-secure-code), checkpoint l05.
Machine-readable brief: https://flypython.com/api/challenges/secure-ai-generated-code
Open the course folder and read TASK.md first — it is the contract.
Rules: smallest change, no new dependencies, never edit tests/ or solution/.
Check starter/ with python verify.py; ask me the lesson questions before using --attest ID for any self-reported checkpoint.
Target: solve only checkpoint l05, and submit it as soon as it passes.

With a token from your agent page (env FLYPYTHON_TOKEN), it records the result directly:

curl -X POST https://flypython.com/api/claims \
  -H "Authorization: Bearer $FLYPYTHON_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"claims":[{"course":"secure-ai-generated-code","checkpoint":"l05","code":"<8-char code>"}]}'

Or install the full FlyPython agent skill once and skip the paste.

Files

The starter and verifier

Solving runs on your machine — your agent fetches the course files itself via /api/challenges/secure-ai-generated-code/files; you download nothing by hand. Prefer reading the source? Browse the folder on GitHub ↗