Challenge course · Any coding agent + an attack-replay harness

Security review for AI-written Python

Review a plausible, AI-style notes service the way an attacker would — close SQL injection, path traversal, a hard-coded secret, fast password hashes, and pickled sessions one class at a time, then prove each fix by replaying the attacks.

Intermediate5 challenges2–4 hoursReads Python with file and SQL operations comfortablyFreeCourse badge: Plausible is not safeReviewed 2026-10-11

Start it with your agent — paste this one sentence:

Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course `secure-ai-generated-code`.

Needs a coding agent that can run commands and reach the network (chat-only web AIs cannot). Your agent authorizes you with a one-time link, fetches this course's files itself, and you never download anything. Haven't installed the FlyPython Skill? Pick your agent's tool course →

Browse the challenges

What you learnEvery one of nine recorded attacks blocked, an attack-replay command that proves it on both scenario skins, and a five-question review habit for the next repository your assistant writes.

Who it's forReads Python with file and SQL operations comfortably · Python 3.11 or newer available on your PATH

Duration & cost2–4 hours · Free

Challenges

The challenges — read them online

No clone, no account — click a checkpoint to read the full challenge on its own page.

Plausible is not safe

Read a notes service that looks fine, then replay nine attacks against it and watch every one succeed.

Checkpoint l01 · 10 pts · read online →02

Write the threat model as a contract

For each function, name who controls the input, what they want, and what the code must refuse — then trace every refusal to a test and the Python documentation behind it.

Checkpoint l02 · 10 pts · read online →03

Close one flaw class per turn

Five bounded turns — search, attachments, secret key, passwords, sessions — each closing one weakness class with a mechanism, never a filter, each proven by tests and the replay.

Checkpoint l03 · 10 pts · read online →04

Review like an attacker

Replay the second skin, run a static scanner and note what it misses, attack one thing the suite never tried, and write down what is still unreviewed.

Checkpoint l04 · 10 pts · read online →05

Review your own AI-written code

Take the five questions and the scanner pass to a repository you own: find where input crosses a trust boundary, check each crossing for a mechanism, and write down what you did not review.

Checkpoint l05 · 10 pts · read online →

Start

How the run works

Your agent drives; the files arrive through the Skill — no download, no clone.

SkillAny coding agent + an attack-replay harness

The Skill does the fetching

The sentence above starts the course: your agent walks you through the one-time authorization link (compare the code, click Allow), fetches this course's files, and drives the challenges with you. Agents without the Skill installed can work from this brief instead:

Work on the FlyPython challenge "Security review for AI-written Python" (course id course-secure-code).
Machine-readable brief: https://flypython.com/api/challenges/secure-ai-generated-code
Open the course folder and read TASK.md first — it is the contract.
Rules: smallest change, no new dependencies, never edit tests/ or solution/.
Check starter/ with python verify.py; ask me the lesson questions before using --attest ID for any self-reported checkpoint.

Your agent works under the task contract: smallest change, no new dependencies, never edit solution/. Optional guided mode: COURSE.md.

Files

Files come to you

Course files live at /api/challenges/secure-ai-generated-code/files — your agent fetches the manifest and writes each file at its path. There is nothing to download by hand.

How "done" is decided →

Scenario skins

Same skills, a domain you recognize

All three skins run through the same code and the same tests. Pick the one closest to your job before lesson 1.

scenario/clinic-notes/attacks.json + seed.json

Clinic visit notes

A small clinic's visit notes — every note belongs to one clinician; attachments live under a scans folder. All names and records are fictional.

scenario/team-notes/attacks.json + seed.json

Team shared notes

A small team's shared notes app: search, attachments, logins, and sessions.

Verification

How “done” is decided

A 16-test suite plus an attack-replay harness covering nine attacks per skin — SQL injection, path traversal, secret disclosure, weak password hashes, and pickled sessions. The starter lets all nine through; the reviewed solution blocks 9/9 on both skins.

python verify.py  # from the course folder: checkpoint status + claim codes

Apply this to your own project

Lesson 5 ports the workflow — not the code — to one script you actually own: a three-line task contract, one new failing test made to pass, and a written record of what the tests do not prove. The same loop is described in the AI coding workflow guide and practiced by therunnable examples.

Claim your evidence

When a checkpoint passes, python verify.py prints a deterministic claim code. Your authorized agent submits them for you in one batch; by hand, paste them on yourprogress page — worth 10 points per checkpoint and thePlausible is not safe course badge at five. Claim codes are self-reported evidence, not certificates.

What verification does not prove

Passing the course suite proves the pinned behaviors on the tested inputs — not correctness on tomorrow’s data, not production performance, and not anything about your own project until you write its contract. Repository tests are evidence, not your outcome.

Report teaching drift ↗All courses →