FlyPython
Sign inStart a challenge

From “it runs” to “it ships”: verifying AI-generated code · Challenge 01 · 中文版

"It runs" is not evidence

Checkpoint l0110 ptsself-reported — code from verify.pyAny coding agent (Claude Code 2.x / Codex CLI 0.x / Cursor)

Watch the starter bless a project with no tests and a project with a failing one — then want better.

Doing this with your agent? One sentence starts the whole course:

Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course `verifying-ai-generated-code`.

“It runs” is not evidence

Objective

Watch the starter bless a project with no tests and a project with a failing one — then want better.

The lesson

Run python starter/ship_check.py scenario/green-project and then .../no-tests — both exit 0. A green check that cannot fail is decoration. Run the verifier to see the seven behaviors you will build:

Exercise

  • Map each expected failure to a real gap: parsed counts, no-tests verdict, unverified list, atomic writes, exit codes
  • Open scenario/red-project/tests and find the wrong expectation

Checkpoint

The lesson’s commands run, and you can answer these in your own words (the agent asks; you answer — that is the gate):

  1. Which command did this lesson run, and what did it decide?
  2. What failed at the start, and why — in your own words?
  3. Before trusting a similar change next time, what would you check first?

Expected evidence

The command transcript and your answers.

Hints

Stuck? Open one at a time.

Hint 1 — What this checkpoint tests

That you watched the starter bless what it should refuse: run it on scenario/no-tests and scenario/red-project and see 'passed' where a human sees failure.

Hint 2 — Name the blind spots

The starter checks exit codes only — a zero-test project 'passes' and a failing suite can look fine. Write down the two failure modes before fixing anything.

Hint 3 — When you are done

You can say why exit-code-only verification lies, in one sentence.

Submit

Done? Record it.

The 8-character code verify.py progress printed for this checkpoint — from the browser, or straight from your agent.

Youbrowser

Submit the claim code

Your agentauto-submit

Let the agent solve and submit

Hand it this checkpoint (copy button on hover) — it solves, verifies, and submits on its own:

Work on the FlyPython challenge "From “it runs” to “it ships”: verifying AI-generated code" (course id course-verify-ship), checkpoint l01.
Machine-readable brief: https://flypython.com/api/challenges/verifying-ai-generated-code
Open the course folder and read TASK.md first — it is the contract.
Rules: smallest change, no new dependencies, never edit tests/ or solution/.
Check with python verify.py until its gates pass, then report each claim code to me.
Target: solve only checkpoint l01, and submit it as soon as it passes.

With a token from your agent page (env FLYPYTHON_TOKEN), it records the result directly:

curl -X POST https://flypython.com/api/claims \
  -H "Authorization: Bearer $FLYPYTHON_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"claims":[{"course":"verifying-ai-generated-code","checkpoint":"l01","code":"<8-char code>"}]}'

Or install the full FlyPython agent skill once and skip the paste.

Files

The starter and verifier

Solving runs on your machine — your agent fetches the course files itself via /api/challenges/verifying-ai-generated-code/files; you download nothing by hand. Prefer reading the source? Browse the folder on GitHub ↗