Lesson 5: Apply it to your own LLM feature

Lesson 5: Apply it to your own LLM feature

Objective

You apply the pattern — output contract, validated parsing, tool allowlist, retry and fallback, an honest offline eval — to a model call in a project you own, and record what the eval said.

Why this lesson exists

The course used hand-written replies so it could run anywhere, offline. Your feature talks to a real model. The pattern transfers because the only thing the code ever depended on was a callable that takes messages and returns text.

The lesson

1. Wrap your provider behind the seam. Write a function live(messages) -> str that calls your provider’s API and returns the reply text. Read the API key from an environment variable; never put it in code or in a recording.

2. Write the contract first. Exact fields, types, ranges; the tools and their argument types; the retry count and the fallback. Lesson 2’s exercise is your draft.

3. Record once, replay on every change. Wrap the live client so each reply is saved, run your labelled cases through it once, then evaluate from the recordings on every later change — no key, no cost, the same answer every time:

   import json
   from pathlib import Path
   
   
   class RecordingClient:
       """Wrap a live client and keep every reply so an eval can replay it."""
   
       def __init__(self, live):
           self.live = live
           self.replies = []
   
       def __call__(self, messages):
           text = self.live(messages)
           self.replies.append(text)
           return text
   
   
   def save_recordings(recorded: dict[str, list[str]], path: Path) -> None:
       path.write_text(
           json.dumps({"replies": recorded}, ensure_ascii=False, indent=2) + "\n",
           encoding="utf-8",
       )
   

   Re-record deliberately — when you change the prompt, the model, or the provider — and compare the new eval line with the old one.

4. Gate on the eval. Pick a threshold before you look at the result, and make the eval command exit non-zero below it — the same shape as ticket_triage.py.

5. Record what it does not prove. Write the three lines: verified / not verified / known limits. “Not verified” always includes live traffic you have not recorded.

If your feature does not fit — the output is free text with no fields to validate, or there are too few cases to label — write down where the pattern stopped working. That judgment is part of the skill.

Exercise

Label at least ten real or realistic inputs for your feature before you record anything. Note which ones you expect to fall back.

Checkpoint

This checkpoint is self-attested. When you can answer the questions below, claim it with python verify.py --attest l05:

1. What does your live client return, and where does its key come from?
2. What was your first eval line, and did it pass the threshold you chose in advance?
3. What is on your “not verified” list that a stakeholder would assume was covered?

Expected evidence

Your output contract, your labelled cases, the first eval line, and the three-line record. Keep recordings with customer data out of public repositories.