Challenge course · Any coding agent (Claude Code 2.x / Codex CLI 0.x / Cursor)

From “it runs” to “it ships”: verifying AI-generated code

Build the release-evidence machine your AI-written project is missing — a ship check that parses what actually ran, refuses zero-test projects, and writes an honest delivery record.

Intermediate5 challenges1–2 hoursOwns an AI-written Python projectFreeCourse badge: It runs is not evidenceReviewed 2026-09-12English + 中文

Start it with your agent — paste this one sentence:

Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course `verifying-ai-generated-code`.

Needs a coding agent that can run commands and reach the network (chat-only web AIs cannot). Your agent authorizes you with a one-time link, fetches this course's files itself, and you never download anything. Haven't installed the FlyPython Skill? Pick your agent's tool course →

Browse the challenges 中文版 →

What you learnship_check.py: one command that turns “the demo worked” into a written delivery record — command, exit code, parsed test count, a verdict that refuses projects with no tests, and an explicit unverified list.

Who it's forOwns an AI-written Python project · Python 3.11 or newer available on your PATH

Duration & cost1–2 hours · Free

Challenges

The challenges — read them online

No clone, no account — click a checkpoint to read the full challenge on its own page.

“It runs” is not evidence

Watch the starter bless a project with no tests and one with a failing test — then want better.

Checkpoint l01 · 10 pts · read online →02

Define what a release record must prove

Command, exit code, parsed count and outcome, an unverified list, and a verdict that refuses zero-test projects.

Checkpoint l02 · 10 pts · read online →03

Build the ship check, test by test

Parsing, verdicts, the unverified list, atomic writes, exit codes — one failing group at a time.

Checkpoint l03 · 10 pts · read online →04

Read the record like a skeptic

Green, red, and no-tests mean different things — prove you can tell them apart.

Checkpoint l04 · 10 pts · read online →05

Gate your next release

Take the ship check to a project you own and make the record part of shipping.

Checkpoint l05 · 10 pts · read online →

Start

How the run works

Your agent drives; the files arrive through the Skill — no download, no clone.

SkillAny coding agent (Claude Code 2.x / Codex CLI 0.x / Cursor)

The Skill does the fetching

The sentence above starts the course: your agent walks you through the one-time authorization link (compare the code, click Allow), fetches this course's files, and drives the challenges with you. Agents without the Skill installed can work from this brief instead:

Work on the FlyPython challenge "From “it runs” to “it ships”: verifying AI-generated code" (course id course-verify-ship).
Machine-readable brief: https://flypython.com/api/challenges/verifying-ai-generated-code
Open the course folder and read TASK.md first — it is the contract.
Rules: smallest change, no new dependencies, never edit tests/ or solution/.
Check with python verify.py until its gates pass, then report each claim code to me.

Your agent works under the task contract: smallest change, no new dependencies, never edit solution/. Optional guided mode: COURSE.md.

Files

Files come to you

Course files live at /api/challenges/verifying-ai-generated-code/files — your agent fetches the manifest and writes each file at its path. There is nothing to download by hand.

How "done" is decided →

Scenario skins

Same skills, a domain you recognize

All three skins run through the same code and the same tests. Pick the one closest to your job before lesson 1.

scenario/green-project/ship.json + tests/

Two passing tests

A tiny project with two passing tests and a declared unverified list.

scenario/red-project/ship.json + tests/

One wrong expectation

The regression under study: one deliberately wrong test expectation.

scenario/no-tests/empty tests/

Zero tests

The state that must never pass: no tests at all.

Verification

How “done” is decided

A 10-test suite pins parsed counts, the no-tests verdict, the unverified list, atomic record writes, and CLI exit codes; the starter reproduces seven named failures, the solution passes 10/10.

python verify.py  # from the course folder: checkpoint status + claim codes

Apply this to your own project

Lesson 5 ports the workflow — not the code — to one script you actually own: a three-line task contract, one new failing test made to pass, and a written record of what the tests do not prove. The same loop is described in the AI coding workflow guide and practiced by therunnable examples.

Claim your evidence

When a checkpoint passes, python verify.py prints a deterministic claim code. Your authorized agent submits them for you in one batch; by hand, paste them on yourprogress page — worth 10 points per checkpoint and theIt runs is not evidence course badge at five. Claim codes are self-reported evidence, not certificates.

What verification does not prove

Passing the course suite proves the pinned behaviors on the tested inputs — not correctness on tomorrow’s data, not production performance, and not anything about your own project until you write its contract. Repository tests are evidence, not your outcome.

Report teaching drift ↗All courses →