Build safe Python automation that can be retried

Write the run contract first

Describe the automation as data flowing through named states. Include a stable run ID, immutable input references, normalized output location, pending through failed states, timestamps, the last successful checkpoint, and a concise error category.

“The command exited” is not a completion definition. Name the artifact and the checks that prove it is usable.

Separate discovery, decision, and mutation

Read external state first, calculate the intended changes, then execute them. This makes a dry run possible and keeps tests away from destructive side effects.

The agent-loop example uses a narrow tool allowlist and records an action trace. A production automation should add the same visibility around every mutation: what was requested, what changed, and how the result was verified.

Make writes atomic and repeatable

When creating a local artifact, write into a temporary file in the destination filesystem, validate it, then replace the final path atomically. Never treat a .partial file as a completed output.

For external APIs, use a stable idempotency key when the provider supports one. Otherwise persist enough state to detect whether the requested effect already happened before retrying.

Do not retry every failure. Retry only errors classified as transient, such as a timeout or an explicit server-overload response. Invalid input, authentication failure, and contract violations need intervention rather than backoff.

Bound retries and time

A safe policy defines maximum attempts, per-operation and total deadlines, exponential backoff with jitter, retryable error classes, and state persisted between attempts.

Iteration limits and time limits solve different problems. Use both when a dependency can block.

Verify business completion

Verification should match the output. Parse the generated file, check record counts and identifiers, open produced media, or fetch the remote object by its final ID. Store the result with the run.

Test at least the first run, exact rerun, interruption after a checkpoint, transient recovery, permanent failure without retry, and invalid output rejected before publication.

Sources

The file-handling guidance follows Python’s pathlib and tempfile documentation. Operational records should use logging or an equivalent structured logger.

Next, decide whether any step genuinely needs model judgment in Build a testable Python agent loop without an API key.

Verification record

About the author

Practical Python guides researched, tested, and maintained by the FlyPython editorial team. Editorial standards and contact details →