Build LLM features in Python: structured output, tool calls, and evals · Challenge 01
Fetch the course through the FlyPython Skill, run the starter's eval, and catch the number that cannot be true — then map every failing test to the real-world reply that breaks it.
Doing this with your agent? One sentence starts the whole course:
Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course `llm-features-in-python`.The course files are in your working folder, fetched by your agent. You have run the starter’s eval, found the number in its output that cannot be true, and can say which real-world model reply each failing test stands for. No code is changed in this lesson.
Open your coding agent in an empty working folder and paste this one sentence:
Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course
llm-features-in-python.
The agent gives you an authorization link with a short code. Open it, log
in, make sure the code matches what the agent showed you, and click
Allow — then tell the agent “done”. It writes this course’s files to
courses/llm-features-in-python/.
Prerequisite: these courses need a coding agent that can run commands and reach the network. A chat-only web AI cannot do them. The exercise itself needs no API key and makes no network calls.
The first version of almost every model-powered feature looks like the
starter: send a prompt, json.loads the reply, use the fields. It works in
the demo because the demo reply was well-formed. In a product, the model
eventually wraps its JSON in a Markdown fence, invents a category, says
"priority": "high", asks for a tool you never built, or answers in prose.
Every one of those is a normal reply — the code that handles it is the
feature.
Ask the agent to survey without changing anything:
“Read COURSE.md — it is the teaching contract for this folder. Then read TASK.md and starter/ticket_triage.py. Do not change any file. Tell me which behaviors the tests expect that the starter does not have.”
Then run the learner’s command — your single check for the rest of the course:
python verify.py
Now run the starter’s eval on two skins:
python starter/ticket_triage.py scenario/it-helpdesk
python starter/ticket_triage.py scenario/shop-support
Read the first line of the it-helpdesk output: accuracy=1.000 correct=5/6. Five out of six is not 100%. The starter’s evaluate skips
every case that crashed and divides by the survivors — so the harder the
model fails, the better the eval looks.
The suite has twelve failing tests. Grouped by the reply that breaks them:
| Failing tests | The reply that breaks the starter |
|---|---|
test_parse_strips_a_markdown_fence | ```json { … } ``` instead of bare JSON |
test_parse_rejects_an_unknown_category_naming_the_field, test_parse_rejects_bad_priorities, test_parse_rejects_missing_extra_and_mistyped_fields | "category": "login", "priority": "high", an extra "confidence", prose, a JSON list — all accepted or crashing |
test_unregistered_tool_is_refused, test_tool_arguments_are_validated_before_the_call | {"tool": "issue_refund", …} or {"asset_tag": 301} — the model choosing what your code runs |
test_invalid_reply_is_retried_with_the_error_fed_back, test_repeated_invalid_replies_fall_back_to_a_human, test_tool_call_budget_ends_in_the_fallback | one bad reply ends the request instead of getting a second chance — or a safe fallback — and a model that keeps calling tools is never stopped |
test_accuracy_divides_by_every_case_and_counts_fallbacks | the eval quietly drops crashed cases |
test_shop_support_eval_is_reported_below_threshold, test_it_helpdesk_eval_passes_its_threshold | end-to-end: the eval line you just read |
Three tests already pass — a valid reply parses, a tool result is sent back to the model, and a missing argument prints usage. The starter is not all wrong; it is wrong exactly where replies stop being polite.
The maintainer command prints the failing names exactly:
python verify.py starter --expect-failure
Pick the skin you most identify with, open its recordings.json next to
its cases.json, and for each case write one line: what the starter does
with that reply (accepts, crashes, or gets it wrong) and what it should
do. Check yourself by asking the agent to confirm — not to fix.
Run python verify.py. When you can answer these without looking, claim
the checkpoint with python verify.py --attest l01:
it-helpdesk eval print accuracy=1.000 with
correct=5/6, and which of the two numbers is honest?The default command’s output, the two eval lines, and your per-case notes for one skin. Lesson 4 compares them against the fixed eval.
Hints
That you can reproduce the failure before fixing anything: python verify.py must show l03 and l04 as [open], and you can explain what the starter's eval output gets wrong.
On it-helpdesk the starter prints accuracy=1.000 correct=5/6. Both numbers cannot be right. Find which one is lying and why.
You can name, for one skin, which recorded reply each failing test stands for. Then claim the checkpoint with python verify.py --attest l01.
Submit
Use the 8-character code printed by python verify.py. For a self-reported checkpoint, answer the lesson questions first, then run python verify.py --attest l01.
An account is only needed to keep points and badges — reading and solving stay free.
Hand it this checkpoint (copy button on hover) — it solves, verifies, and submits on its own:
Work on the FlyPython challenge "Build LLM features in Python: structured output, tool calls, and evals" (course id course-llm-features), checkpoint l01.
Machine-readable brief: https://flypython.com/api/challenges/llm-features-in-python
Open the course folder and read TASK.md first — it is the contract.
Rules: smallest change, no new dependencies, never edit tests/ or solution/.
Check starter/ with python verify.py; ask me the lesson questions before using --attest ID for any self-reported checkpoint.
Target: solve only checkpoint l01, and submit it as soon as it passes.With a token from your agent page (env FLYPYTHON_TOKEN), it records the result directly:
curl -X POST https://flypython.com/api/claims \
-H "Authorization: Bearer $FLYPYTHON_TOKEN" \
-H "Content-Type: application/json" \
-d '{"claims":[{"course":"llm-features-in-python","checkpoint":"l01","code":"<8-char code>"}]}'Or install the full FlyPython agent skill once and skip the paste.
Solving runs on your machine — your agent fetches the course files itself via /api/challenges/llm-features-in-python/files; you download nothing by hand. Prefer reading the source? Browse the folder on GitHub ↗