← Blog

Your scheduled AI agents fail silently. Here's how to catch it

· 3 min read · CronObserver team

Last Thursday our 9 p.m. summary agent didn't send its report. No error in the inbox, no red dashboard, no stack trace. We noticed the next morning, when someone asked where the summary was.

That is the normal way scheduled AI agents fail. Not with a crash you see, but with an absence nobody is watching for.

Four ways an agent fails without telling you

Agents that run on a schedule (Claude Code routines, Codex jobs, n8n or Make flows, LangGraph or CrewAI pipelines behind a cron) have failure modes that traditional jobs mostly didn't:

  1. It never ran. The host rebooted, the OAuth token was revoked, the plan quota ran out before the trigger fired. Nothing executed, so nothing logged.
  2. It ran and failed. A 429 from the model provider, a context window overflow, a tool that returned HTML instead of JSON. The process exits, and unless someone reads that log, it's gone.
  3. It ran and got stuck. The agent loops on a tool call, retries forever, or waits on a prompt nobody will answer. It is "running" for six hours.
  4. It ran "fine" and delivered garbage. Zero documents processed, an empty summary, a report about the wrong day.

Your error tracker catches none of the first three reliably, because the problem is not an exception. The problem is that the thing you expected didn't happen.

Watch for the absence, not the error

The fix is old and boring, which is why it works: a heartbeat, also called a dead man's switch.

  1. Tell a monitor when the agent should finish (for example, every day by 21:15).
  2. At the very end of a successful run, the agent pings a URL.
  3. If the ping doesn't arrive in time, you get an alert.

The key is to ping only on success and only at the end. Then cases 1, 2 and 3 all look the same to the monitor: no ping. You don't need to predict how the agent will break.

Adding it to an agent in one line

In CronObserver, each monitor has a check-in URL. Add it after the agent command, chained with && so it only runs if the agent exited cleanly:

# crontab: daily summary agent at 21:00 UTC
0 21 * * * /opt/agents/run-summary.sh && curl -fsS --retry 3 -X POST https://cronobserver.com/checkin/<token>

In Python, put the ping after the work, outside any try that swallows errors:

import json, urllib.request

def ping(token: str, payload: dict) -> None:
    req = urllib.request.Request(
        f"https://cronobserver.com/checkin/{token}",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    urllib.request.urlopen(req, timeout=10)

result = run_agent()          # raises on failure: no ping, so you get an alert
ping(TOKEN, {
    "status": "success",
    "processed": result.items,     # how many things the agent handled
    "durationMs": result.elapsed_ms,
    "model": result.model,
})

The JSON payload is optional and stored with each check-in, so when something looks off you can see what the last good run actually did.

Pick the grace period carefully

Agents are less predictable than a pg_dump. A run that takes 4 minutes today may take 12 tomorrow because the provider is slow. Set the schedule to when the agent starts, and the grace period to the slowest normal run plus a margin. Too tight and you get false alarms; too loose and you learn about the failure hours late. Start generous and tighten after a week of real data.

What about case 4, the garbage output?

A heartbeat proves the agent finished. It doesn't prove the output is good. Two cheap guards help today:

  • Make "empty" a failure. If the agent processed zero items or produced an empty report, exit non-zero before the ping. No ping, alert.
  • Send what it did in the payload. Item counts and duration in each check-in give you a history to compare against when something feels wrong.

The checklist

  • One monitor per scheduled agent, not one per server.
  • Ping at the end, only on success (&&, or after the work in code).
  • Grace period = slowest normal run + margin.
  • Empty output exits non-zero.
  • Alert goes to a person who can act: email on any plan; Slack, Teams, Telegram or a webhook on the paid plans.

Your agents will fail. The question is whether you hear about it at 21:15 or the next morning.

Monitor your first agent today. New accounts get 14 days of the Team plan, no credit card. Create your monitor →