Build LLM features in Python: structured output, tool calls, and evals · Challenge 04

Lesson 4: Read the eval honestly

Checkpoint l0410 ptsgated by test suiteAny coding agent + a recorded model replay (no API key)

A green suite proves the contract; the eval measures the feature. Run all three skins, accept the one that fails its threshold, and write down what a handful of recorded replies can and cannot tell you.

Doing this with your agent? One sentence starts the whole course:

Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course `llm-features-in-python`.

Lesson 4: Read the eval honestly

Objective

Both suites pass, you have run the eval on every skin, and you hold a written record of what the numbers prove and what they do not.

Why this lesson exists

In Lesson 1 the starter’s eval said 100% on a skin where one case in six had crashed. That is the most common way model features get shipped too early: not a bad model, an eval that hides its failures. Now that the eval divides by every case, the numbers can go down — and that is the point.

The lesson

Run the maintainer pair in a terminal yourself:

python verify.py starter
python verify.py solution

Then run your eval on every skin:

python starter/ticket_triage.py scenario/shop-support
python starter/ticket_triage.py scenario/it-helpdesk
python starter/ticket_triage.py scenario/saas-feedback

You should see:

Skin	Result	Exit
`shop-support`	`accuracy=0.750 correct=6/8 fallbacks=1`	1
`it-helpdesk`	`accuracy=0.833 correct=5/6 fallbacks=1`	0
`saas-feedback`	`accuracy=0.800 correct=4/5 fallbacks=0`	0

Compare it-helpdesk with Lesson 1: the starter said 1.000, your eval says 0.833. Same recorded replies, honest denominator.

Now read each failure line as a reviewer:

shop-support fails its threshold — correctly. Case s08 got a valid but wrong category; s07 fell back to a human after two prose replies. Nothing in your code is wrong. The feature, on this data, is below the bar you set. Try --min-accuracy 0.7 and watch it “pass” — then write down why that is not a fix.
A fallback is safe but not free. h06 fell back because both replies broke the 120-character summary limit. Count fallbacks separately: a feature that falls back on half the tickets is safe and useless.
Look at what the eval does not score. Correctness here is category + needs_human. priority and summary are validated for shape but never compared to a label. Decide whether that is acceptable for your feature.
Check the edges the suite skips. What happens if cases.json is empty? If recordings.json has fewer replies than the loop asks for? (ReplayClient raises — your loop should let it.) Run one and write down what happened.

Exercise

Write the three-line evidence record for this feature:

Verified: the commands you ran and what they proved.
Not verified: at least — live model behavior, traffic you have not recorded, unscored fields, and anything the recordings do not imitate.
Known limits: the threshold each skin missed or barely met, and the fallback rate.

Checkpoint

Run python verify.py. This checkpoint’s code prints once the starter suite passes; the maintainer pair above must both be green for the gate to mean what it says. You pass the lesson when you can answer:

Why is shop-support exiting 1 the correct behavior of your code?
What does accuracy=0.833 on six hand-written cases tell you about a live model on live tickets?
Which field would you add to the eval’s scoring first, and why?

Expected evidence

Both green runs, the three eval lines, and your three-line evidence record.

Hints

Stuck? Open one at a time.

Hint 1 — What this checkpoint tests

That starter AND solution both pass — the maintainer pair verify.py starter / verify.py solution must both be green, and python verify.py prints this code once your starter suite passes.

Hint 2 — A failing eval can be the correct result

shop-support reports 0.750 against a 0.800 threshold and exits 1. Your code is right; the feature on that data is not good enough yet. Lowering the threshold is not a fix.

Hint 3 — When you are done

You have a three-line evidence record — verified / not verified / known limits — that a teammate could act on.

Submit

Done? Record it.

Use the 8-character code printed by python verify.py. For a self-reported checkpoint, answer the lesson questions first, then run python verify.py --attest l04.

Youbrowser

Submit the claim code

Your agentauto-submit

Let the agent solve and submit

Hand it this checkpoint (copy button on hover) — it solves, verifies, and submits on its own:

Work on the FlyPython challenge "Build LLM features in Python: structured output, tool calls, and evals" (course id course-llm-features), checkpoint l04.
Machine-readable brief: https://flypython.com/api/challenges/llm-features-in-python
Open the course folder and read TASK.md first — it is the contract.
Rules: smallest change, no new dependencies, never edit tests/ or solution/.
Check starter/ with python verify.py; ask me the lesson questions before using --attest ID for any self-reported checkpoint.
Target: solve only checkpoint l04, and submit it as soon as it passes.

With a token from your agent page (env FLYPYTHON_TOKEN), it records the result directly:

curl -X POST https://flypython.com/api/claims \
  -H "Authorization: Bearer $FLYPYTHON_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"claims":[{"course":"llm-features-in-python","checkpoint":"l04","code":"<8-char code>"}]}'

Or install the full FlyPython agent skill once and skip the paste.

Files

The starter and verifier

Solving runs on your machine — your agent fetches the course files itself via /api/challenges/llm-features-in-python/files; you download nothing by hand. Prefer reading the source? Browse the folder on GitHub ↗