Challenge course · Any coding agent + a recorded model replay (no API key)

Build LLM features in Python: structured output, tool calls, and evals

Turn a demo-day model call into a feature you can ship — validate structured output, allowlist tool calls, retry with the error fed back, fall back to a human instead of guessing, and measure it with an offline eval that divides by every case.

Intermediate5 challenges2–4 hoursComfortable with functions, dictionaries, and exceptionsFreeCourse badge: Model output is untrusted inputReviewed 2026-10-11

Start it with your agent — paste this one sentence:

Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course `llm-features-in-python`.

Needs a coding agent that can run commands and reach the network (chat-only web AIs cannot). Your agent authorizes you with a one-time link, fetches this course's files itself, and you never download anything. Haven't installed the FlyPython Skill? Pick your agent's tool course →

Browse the challenges

What you learnA working ticket-triage feature (ticket_triage.py) that treats every model reply as untrusted input — validated output contract, tool-call allowlist, retry with the error fed back, human fallback — plus an offline eval with a pass/fail threshold you can replay on every change.

Who it's forComfortable with functions, dictionaries, and exceptions · Python 3.11 or newer available on your PATH

Duration & cost2–4 hours · Free

Challenges

The challenges — read them online

No clone, no account — click a checkpoint to read the full challenge on its own page.

A demo reply is not a feature

Run the starter's eval, catch the number that cannot be true, and map every failing test to the real-world reply that breaks it.

Checkpoint l01 · 10 pts · read online →02

Write the output contract

Read TASK.md as the boundary between your code and the model — what a reply must look like, which tools it may run, and what happens when it does not comply.

Checkpoint l02 · 10 pts · read online →03

Validate, allowlist tools, retry — one group at a time

Drive the starter to green in four bounded turns — parser, tool gate, retry-and-fallback loop, eval — and watch one fix expose a bug the old code was hiding.

Checkpoint l03 · 10 pts · read online →04

Read the eval honestly

Run all three skins, accept the one that fails its threshold, and write down what a handful of recorded replies can and cannot tell you.

Checkpoint l04 · 10 pts · read online →05

Apply it to your own LLM feature

Put a real model behind the same seam: a callable client, an output contract, an allowlist, a fallback, and an eval you record once and replay on every change.

Checkpoint l05 · 10 pts · read online →

Start

How the run works

Your agent drives; the files arrive through the Skill — no download, no clone.

SkillAny coding agent + a recorded model replay (no API key)

The Skill does the fetching

The sentence above starts the course: your agent walks you through the one-time authorization link (compare the code, click Allow), fetches this course's files, and drives the challenges with you. Agents without the Skill installed can work from this brief instead:

Work on the FlyPython challenge "Build LLM features in Python: structured output, tool calls, and evals" (course id course-llm-features).
Machine-readable brief: https://flypython.com/api/challenges/llm-features-in-python
Open the course folder and read TASK.md first — it is the contract.
Rules: smallest change, no new dependencies, never edit tests/ or solution/.
Check starter/ with python verify.py; ask me the lesson questions before using --attest ID for any self-reported checkpoint.

Your agent works under the task contract: smallest change, no new dependencies, never edit solution/. Optional guided mode: COURSE.md.

Files

Files come to you

Course files live at /api/challenges/llm-features-in-python/files — your agent fetches the manifest and writes each file at its path. There is nothing to download by hand.

How "done" is decided →

Scenario skins

Same skills, a domain you recognize

All three skins run through the same code and the same tests. Pick the one closest to your job before lesson 1.

scenario/it-helpdesk/recordings.json + devices.json

IT helpdesk with a device lookup tool

Classify internal tickets into five categories; the model may look up a device by asset tag first.

scenario/saas-feedback/recordings.json

SaaS feedback inbox

Classify in-app feedback with no tools at all — the model answers from the ticket text alone.

scenario/shop-support/recordings.json + orders.json

Shop support with an order lookup tool

Classify a shop's support mail; the model may look up an order before answering.

Verification

How “done” is decided

A 15-test suite pins the output contract, the tool-call allowlist, retry with error feedback, the human fallback, and an eval that divides by every case. The starter reproduces the named failures; the reviewed solution passes 15/15 against recorded replies — no network, no API key.

python verify.py  # from the course folder: checkpoint status + claim codes

Apply this to your own project

Lesson 5 ports the workflow — not the code — to one script you actually own: a three-line task contract, one new failing test made to pass, and a written record of what the tests do not prove. The same loop is described in the AI coding workflow guide and practiced by therunnable examples.

Claim your evidence

When a checkpoint passes, python verify.py prints a deterministic claim code. Your authorized agent submits them for you in one batch; by hand, paste them on yourprogress page — worth 10 points per checkpoint and theModel output is untrusted input course badge at five. Claim codes are self-reported evidence, not certificates.

What verification does not prove

Passing the course suite proves the pinned behaviors on the tested inputs — not correctness on tomorrow’s data, not production performance, and not anything about your own project until you write its contract. Repository tests are evidence, not your outcome.

Report teaching drift ↗All courses →