Build LLM features in Python: structured output, tool calls, and evals · Challenge 02

Lesson 2: Write the output contract

Checkpoint l0210 ptsself-reported — code from verify.pyAny coding agent + a recorded model replay (no API key)

The prompt asks; the contract checks. Read TASK.md as the boundary between your code and the model — what a reply must look like, which tools it may run, and what happens when it does not comply.

Doing this with your agent? One sentence starts the whole course:

Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course `llm-features-in-python`.

Lesson 2: Write the output contract

Objective

You can read TASK.md as a set of testable statements about model replies, trace each statement to a test in tests/test_ticket_triage.py, and explain the three decisions the contract makes for you: what a valid reply is, which tool calls may run, and what happens when the model does not comply.

Why this lesson exists

Open build_messages in the starter. It already tells the model the exact JSON fields, the allowed categories, the priority range, and how to call a tool. Many teams stop there — the prompt says what we want, so the reply will be what we want. The prompt is a request. The contract is what your code enforces when the request is ignored, and the tests are how you know it does.

Provider APIs often offer a JSON or schema-constrained output mode. Where you use one, it reduces malformed replies; it does not tell you that the category is right, that a tool argument is safe to act on, or that your eval is honest. The contract in this course still applies.

The lesson

Read TASK.md with the tests open beside it. The contract makes three decisions:

1. What a valid reply is. Exactly four fields, each with a type and a range. Anything else is rejected with an error that names the field.

Contract line	Test
“strip … one surrounding Markdown code fence”	`test_parse_strips_a_markdown_fence`
“`category` is one of categories”	`test_parse_rejects_an_unknown_category_naming_the_field`
“`priority` is an `int` from 1 to 4 (`bool` and `float` are rejected)”	`test_parse_rejects_bad_priorities`
“a missing field or an extra field raises `TriageError` naming that field”	`test_parse_rejects_missing_extra_and_mistyped_fields`

Why name the field? Because the error message is going back to the model in Lesson 3. “Invalid reply” gives it nothing to correct; “category must be one of billing, shipping, …; got ‘login’” tells it exactly what to change.

2. Which tool calls may run. The tools dict is the allowlist. An unregistered name is refused before anything runs, and arguments are checked against declared names and types before the function is called. The recorded reply {"tool": "issue_refund", …} in shop-support is the case to remember: a model that can name any function can make your code do anything that function does.

3. What happens on failure. One retry with the error fed back, then a fallback that sends the ticket to a human with no category. The fallback never guesses — a confident wrong category is worse than an honest “needs a person”.

Exercise

Take a model call from a project you know — or invent one — and write its output contract in the same shape: the exact fields with types and ranges, the tools it may call with their argument types, and the failure policy (how many retries, what the fallback returns). Then mark each sentence: could a test assert it without reading your mind?

Checkpoint

When you can answer these, claim the checkpoint with python verify.py --attest l02:

build_messages already states the format. Why does parse_triage still have to check it?
Why must a rejected tool call never reach the tool function — what could issue_refund have done?
Why does the fallback return category: None instead of the most likely category?

Expected evidence

Your drafted output contract with each sentence marked testable or not.

Hints

Stuck? Open one at a time.

Hint 1 — What this checkpoint tests

That you can read TASK.md as testable statements and trace each to a test — and say why the format request in build_messages is not enough on its own.

Hint 2 — A prompt is a request, not a guarantee

build_messages tells the model the exact format. The model usually follows it. 'Usually' is the reason parse_triage exists.

Hint 3 — A tool call is the model choosing what your code runs

The tool registry is an allowlist and the arguments are untrusted input — the same rule you would apply to a web form.

Submit

Done? Record it.

Use the 8-character code printed by python verify.py. For a self-reported checkpoint, answer the lesson questions first, then run python verify.py --attest l02.

Youbrowser

Submit the claim code

Your agentauto-submit

Let the agent solve and submit

Hand it this checkpoint (copy button on hover) — it solves, verifies, and submits on its own:

Work on the FlyPython challenge "Build LLM features in Python: structured output, tool calls, and evals" (course id course-llm-features), checkpoint l02.
Machine-readable brief: https://flypython.com/api/challenges/llm-features-in-python
Open the course folder and read TASK.md first — it is the contract.
Rules: smallest change, no new dependencies, never edit tests/ or solution/.
Check starter/ with python verify.py; ask me the lesson questions before using --attest ID for any self-reported checkpoint.
Target: solve only checkpoint l02, and submit it as soon as it passes.

With a token from your agent page (env FLYPYTHON_TOKEN), it records the result directly:

curl -X POST https://flypython.com/api/claims \
  -H "Authorization: Bearer $FLYPYTHON_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"claims":[{"course":"llm-features-in-python","checkpoint":"l02","code":"<8-char code>"}]}'

Or install the full FlyPython agent skill once and skip the paste.

Files

The starter and verifier

Solving runs on your machine — your agent fetches the course files itself via /api/challenges/llm-features-in-python/files; you download nothing by hand. Prefer reading the source? Browse the folder on GitHub ↗