An automation audit is a check of every job you believe runs on a schedule against evidence that it actually ran, because the file where you declare a schedule and the system that executes it are two different things. In July I audited my own AI assistant and found ten declared jobs with zero runs between them.
The file nobody read
I run a personal AI assistant on my own machine. It is a folder of instruction files, scripts and logs that drafts my newsletters, refreshes a private dashboard every morning and runs my engagement routines on LinkedIn and X. Each recurring routine had an entry in a file called scheduled_tasks.json, with a cron expression next to it. Daily at 07:30. Twice a day for replies on X. A set of jobs for a music channel pipeline.
On 27 July I sat down to find out why the numbers on my dashboard never seemed to change, and found that nothing on the machine read scheduled_tasks.json, which was documentation that looked exactly like configuration. The operating system's real scheduler, the crontab, held a single line: the Monday newsletter job. The other ten declared jobs had never fired once, including the dashboard refresh, both LinkedIn engagement runs, both X reply runs and all three music pipeline runs. The two blog jobs, the Sunday draft and the Monday publish, were missing from both places. I had removed them from the file on 20 July and they did not come back as real scheduler entries until 10 August.
The proof was sitting on disk. Only two dashboard snapshots existed, dated 24 July and 27 July, and I had run both by hand. The history file that should have gained a row every day jumped straight from the 24th to the 27th.
The one job that was registered had its own problem. Its script appended output to a file inside a logs folder, and that folder did not exist. The redirect failed before the job started, so every Monday since I set it up, the newsletter job died without writing a single line anywhere. A job that fails before it can log looks identical to a job that was never scheduled.
I fixed all of it that day. I wrote a real runner script, added real crontab entries, created the log folder inside the script, and fed the command empty input, because cron gives a job no keyboard and the command-line tool I use will otherwise sit waiting for one. I also found the dashboard server sending no cache headers, so my browser had been showing a page stamped "as of 24 July" while the file on disk carried 27 July. Three separate faults, and every one of them looked like the same symptom: stale numbers. A full run of the new dashboard job then finished cleanly in six minutes and rebuilt the page.
Two weeks later it happened again, in a different shape. On 10 August I opened the dashboard log and found every run since 28 July was exactly three lines long: started, "Credit balance is too low", completed with exit code 1. My runner scripts load a settings file that includes an API key. When that key is present, the command-line tool bills the API account instead of my subscription, and the API account had a zero balance. For 13 days the dashboard had been stale again, and the newsletter log held one line of output in its entire life.
I wrote a fix that removed the key before the tool started, and I tested it in a stripped environment with almost nothing loaded. It printed "OK". Ten minutes later the real noon newsletter job fired and failed with a new message: "Not logged in". A job launched by cron cannot reach the login my interactive sessions use, so that API key had been the only way those jobs could authenticate at all. My test had cleared the environment variables and still run inside my own logged-in session. Cron's context is different, and the test could not see the difference. I reverted the change within the hour.
The real fix is a billing action on my side, and it is still open as I write this. The logs show the same message on 13, 14, 20 and 21 September. This essay is being drafted in a session I started myself on a Tuesday, because both blog jobs, the Sunday draft and the Monday publish, died on that exact line.
The four proofs an automation ran
Every one of those failures would have been caught by asking four questions in order. I now ask them of every scheduled job, and I would ask them of any agent a team claims is "running in production".
DECLARED --> REGISTERED --> STARTED --> PRODUCED --> RAN IN CONTEXT
(a file) (the scheduler) (a log) (an artefact) (the real runner)
| | | |
Proof 1 Proof 2 Proof 3 Proof 4Proof 1: Registered, where the executor reads it. A schedule is real only in the system that fires it. For me that is crontab -l. For a team it might be a workflow tool, a cloud scheduler or an orchestration console. Open that system and read the list back. Compare it line by line with whatever document, slide or config file the team treats as the source. My gap was ten declared jobs against one registered one, and no amount of reading the JSON file would ever have shown it.
Summary: read the scheduler itself, then the document.
Proof 2: Started, with a log that exists before the job does. A job has to be able to say it started. That means the log destination is created by the job itself, or by setup that runs first, never assumed. My newsletter job could not write its first line because the folder was missing, so the absence of a log looked like the absence of a job. Put a timestamped "started" line at the very top of every runner, before anything that could fail.
Summary: if a job cannot log its own start, you cannot tell broken from never scheduled.
Proof 3: Produced an artefact, checked where the reader sees it. A start line proves only that the job began. Each job should leave something with a timestamp: a file, a record, a draft, a row. Then check that artefact at the point where a person actually consumes it. My dashboard file on disk was fresh while the browser showed a three-day-old copy. A check against the file would have passed. A check against the screen caught it.
Summary: the newest output, dated, in the place someone reads it.
Proof 4: Ran in the real context. The fix that passed in my stripped test failed in cron within ten minutes. Scheduled jobs run with a different user context, a different environment, no terminal and often different file permissions. I had already learned in late July that a cron job on my Mac could not reliably read a script stored inside my Documents folder, which is why the runners now live in a separate bin folder. The only test that counts is a real scheduled run: set the job to fire five minutes from now in the actual scheduler and watch the log.
Summary: a green result from a simulation tells you about the simulation.
The order matters. Each proof assumes the one before it. There is no point reading a log for a job that was never registered, and no point checking an artefact for a job that never started.
None of the four needs special tooling. My first audit took one sitting on 27 July. The faults it exposed had been live since the day each job was declared, and the next one ran for 13 days before I found it in the dashboard's own log file.
Why this matters more once agents are involved
When a scheduled script hits a missing file, it usually says so. An agent job has more places to stop quietly: a login it cannot reach, a billing account it did not know it was using, a folder that is not there. My worst failures were silent in the most literal sense: a three-line log that nobody opened, and a dashboard that kept rendering its last good numbers.
If I were reviewing a team's agents, I would ask for one table with five columns: job name, where it is registered, last start time, newest artefact with its date, and the identity it authenticates as. The fifth column is the one I would expect to come back blank, because it only matters once the builder has left the room and the scheduler takes over.
The second reason is billing and identity. My jobs broke on an authentication path I did not know I was using. The interactive tool on my screen and the scheduled tool at 06:00 were the same program authenticating two different ways. Any team moving agents from a person's laptop into a scheduler inherits that split. In my case, every job that worked on screen had been running as me, inside a session I had logged into.
The third reason is that an AI system keeps producing activity even when its scheduled output has stopped. Most of my assistant's work runs in sessions I start myself, and those kept producing drafts and posts, so the system felt busy. The scheduled half of the system had been dead for weeks, and I noticed only because a number on a tile never moved.
Three things to do this week
- List every scheduled job next to its newest output. Write down each routine you believe runs on its own, and beside it the path, record or link of the most recent thing it produced, with a date. Any job where you cannot fill in the second column goes to the top of the list.
- Diff your list against the scheduler itself. Open the real executor, whether that is
crontab -l, your workflow tool or your cloud console, and read every entry back. Mark jobs that exist only in documentation and jobs that exist only in the scheduler. Expect at least one surprise in each direction. - Watch one job run for real. Pick the job you trust least and schedule it to fire five minutes from now in the actual scheduler. Sit with the log open while it runs. If it fails, you have found the gap in five minutes instead of 13 days.
What to read next
While you are here, the back catalogue has more on this:
- The 50-Case Test That Decides If Your Enterprise Agent Ships
- The Loop Stack, or How to Stop an AI Agent From Gaming Its Own Goal
- The Four Constraints That Decide Whether AI Output Is Usable
Footer
Anees Merchant writes one essay every Tuesday about enterprise AI, agentic systems, and the human side of the work. He is the author of Merchants of AI, a TEDx speaker, and a doctoral researcher in Human-AI Communication at the Swiss School of Business and Management.
The newsletter version, with extra commentary, goes out separately on Mondays. Subscribe at /newsletter.
See you Tuesday.