Lesson 1: Plausible is not safe

Lesson 1: Plausible is not safe

Objective

The course files are in your working folder, fetched by your agent. You have read the starter without running it, written down what you think is wrong, and then replayed the attacks — and you can say what an attacker gets from each of the five functions. No code is changed in this lesson.

Step 1 — Let the agent fetch this course (you download nothing)

Open your coding agent in an empty working folder and paste this one sentence:

Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course secure-ai-generated-code.

The agent gives you an authorization link with a short code. Open it, log in, make sure the code matches what the agent showed you, and click Allow — then tell the agent “done”. It writes this course’s files to courses/secure-ai-generated-code/.

Prerequisite: these courses need a coding agent that can run commands and reach the network. A chat-only web AI cannot do them.

Safety note: the attacks in this course touch only the course’s own files and temporary folders. One test deliberately unpickles a harmless payload that creates a directory in a temporary folder — that is the evidence, not an accident.

Why this lesson exists

AI-written code is usually readable, consistent, and documented. Those are the signals reviewers use to decide code is fine — and none of them say anything about what a hostile input can make it do. The starter works for every friendly input. That is exactly why it is dangerous.

Warm-up (5 minutes, no agent)

Open starter/notes_app.py and read the five functions after init_db. For each one, write one line: what input does it trust, and what could go wrong? Do not run anything yet.

The lesson

Ask the agent to survey without changing anything:

“Read COURSE.md — it is the teaching contract for this folder. Then read TASK.md and starter/notes_app.py. Do not change any file. For each function, tell me which input it trusts that it should not.”

Then run the learner’s command and the attack replay:

python verify.py
python starter/attack_replay.py scenario/team-notes

The replay prints blocked=0/9. Read the [OPEN] lines — each is a real consequence:

Attack	What the attacker got
t01, t02 (sql)	Bob’s and Carol’s notes, including “Salary review”, from a search as Alice
t03, t04 (path)	the contents of secrets.txt, a file outside attachments/
t05, t06 (secret-key)	a service that started with a key everyone can read in the source
t07 (fast-hash)	a stored hash that one MD5 call reproduces — and a leaked MD5 that still logs in
t08 (forged-session)	an admin session the attacker signed with their own key
t09 (pickle-session)	code execution: the payload created a directory on load

The suite has eleven failing tests that pin these behaviors. The maintainer command prints them:

python verify.py starter --expect-failure

Five tests already pass: a normal search works and is owner-scoped, a normal attachment reads, a password verifies against its own hash, a session round-trips, and the replay prints usage. Every friendly path works.

Exercise

Compare your warm-up notes with the replay. Which flaws did you spot by reading, and which did you miss? Then pick the flaw you missed and ask the agent to explain the attack in two sentences — not to fix it.

Checkpoint

Run python verify.py. When you can answer these without looking, claim the checkpoint with python verify.py --attest l01:

1. How does query %' OR owner != ' turn Alice’s search into everyone’s notes? Write out the SQL the starter actually runs.
2. Why does t09 count as code execution and not just “bad data”?
3. Which flaw did you miss in the warm-up, and what about the code made it look fine?

Expected evidence

Your warm-up notes, the replay output, and your three answers.