Build LLM features in Python: structured output, tool calls, and evals · Challenge 03

Lesson 3: Validate, allowlist tools, retry — one group at a time

Checkpoint l0310 ptsgated by test suiteAny coding agent + a recorded model replay (no API key)

Drive the starter to green in four bounded turns — the parser, the tool gate, the retry-and-fallback loop, the eval — and watch one fix expose a bug the old code was hiding.

Doing this with your agent? One sentence starts the whole course:

Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course `llm-features-in-python`.

Lesson 3: Validate, allowlist tools, retry — one group at a time

Objective

The starter passes all fifteen tests through four bounded turns, and you can explain every diff without reading the solution — including the one test that went red in the middle.

Why this lesson exists

The tempting move is “make it robust” in one prompt. You get a large diff that wraps everything in try/except, accepts anything that looks close, and passes some tests for the wrong reasons. Four bounded turns, each pinned to one contract decision from Lesson 2, keep every line explainable.

The lesson

Start a fresh agent session and say:

“Work through TASK.md against starter/ticket_triage.py, one function per turn: parse_triage, then run_tool_call, then triage, then evaluate. After each turn run python verify.py starter, show me the diff, and wait for me before the next turn. Do not edit tests/, solution/, or scenario/.”

Expect the failing count to move like this:

After turn	Failing	What changed
start	12	—
`parse_triage`	9	four parser tests pass; one passing test turns red
`run_tool_call`	7	the allowlist and argument checks
`triage`	3	retry with feedback, fallback, the tool-call budget, and the tool path repaired
`evaluate`	0	the denominator, fallbacks, and the two end-to-end tests

The red test after turn one is the most useful moment in the course. test_tool_result_is_fed_back_to_the_model passed only because the starter’s triage ran every reply through a parser that accepted any JSON, then looked for a tool key. A strict parser rejects a tool call — it is not a triage answer. The fix belongs in triage (decode the object, then branch on tool), not in loosening the parser. If the agent proposes to make parse_triage accept tool calls again, reject it and point at the contract.

Hold every turn to these boundaries:

Only starter/ticket_triage.py changes. Any diff in tests/, solution/, or scenario/ is rejected.
No bare except: and no except Exception in triage — the loop catches TriageError (which ToolCallError extends) and nothing else. A client that raises is a different problem and must stay visible.
Error messages name the field or argument. “Invalid” is not a message.
The fallback never guesses a category.
Standard library only.

Exercise

After the triage turn, read the messages a retry sends. Run this in the course folder and read the last message of the second call:

python -c "import sys; sys.path.insert(0, 'starter'); import ticket_triage as t; c = t.ReplayClient(['{\"category\": \"login\", \"priority\": 2, \"summary\": \"x\", \"needs_human\": false}', '{\"category\": \"account\", \"priority\": 2, \"summary\": \"x\", \"needs_human\": false}']); print(t.triage('cannot log in', c, categories=['account', 'other'])); print(c.calls[1][-1])"

Explain in one sentence why that message is what makes a retry worth doing.

Checkpoint

Run python verify.py. This checkpoint’s code appears only when the starter suite is green. You pass the lesson when you can answer:

Which test turned red after the parser turn, and why was the fix in triage rather than parse_triage?
What exactly does your run_tool_call check before it calls the function, and in which order?
Why does the loop catch only TriageError?

Expected evidence

A green python verify.py starter, the four diffs, and your explanation of the red-then-green test.

Hints

Stuck? Open one at a time.

Hint 1 — What this checkpoint tests

That the starter passes all fifteen tests through changes you supervised — python verify.py shows this code only once the starter suite is green.

Hint 2 — Expect one test to turn red

After the parser turn, test_tool_result_is_fed_back_to_the_model fails. The old loop spotted tool calls by parsing them loosely; a strict parser rejects them. The loop has to decode first, then decide.

Hint 3 — Never loosen to pass

If a recorded reply fails validation, the fix is never to accept it. Tests and recordings are read-only; the code moves.

Submit

Done? Record it.

Use the 8-character code printed by python verify.py. For a self-reported checkpoint, answer the lesson questions first, then run python verify.py --attest l03.

Youbrowser

Submit the claim code

Your agentauto-submit

Let the agent solve and submit

Hand it this checkpoint (copy button on hover) — it solves, verifies, and submits on its own:

Work on the FlyPython challenge "Build LLM features in Python: structured output, tool calls, and evals" (course id course-llm-features), checkpoint l03.
Machine-readable brief: https://flypython.com/api/challenges/llm-features-in-python
Open the course folder and read TASK.md first — it is the contract.
Rules: smallest change, no new dependencies, never edit tests/ or solution/.
Check starter/ with python verify.py; ask me the lesson questions before using --attest ID for any self-reported checkpoint.
Target: solve only checkpoint l03, and submit it as soon as it passes.

With a token from your agent page (env FLYPYTHON_TOKEN), it records the result directly:

curl -X POST https://flypython.com/api/claims \
  -H "Authorization: Bearer $FLYPYTHON_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"claims":[{"course":"llm-features-in-python","checkpoint":"l03","code":"<8-char code>"}]}'

Or install the full FlyPython agent skill once and skip the paste.

Files

The starter and verifier

Solving runs on your machine — your agent fetches the course files itself via /api/challenges/llm-features-in-python/files; you download nothing by hand. Prefer reading the source? Browse the folder on GitHub ↗