How do you know your automation is really working?

A job ran every morning for seven weeks and reported success every morning. It was reading the wrong table the whole time. Roughly half a million words sat unread on a disk about eighteen inches from the person who owned them, while a log file said the pipeline was working.

This is our own failure, dated and measured. The shape of it shows up in a drip campaign, a lead router and a transaction checklist as readily as it showed up here.

What we found on our own machine

The job was simple. Every night it opened the local history database belonging to a dictation app and built a record of how one person speaks, so that anything drafted later would sound like them rather than like a machine. It ran on a schedule. It reported success.

It was reading a table called RemoteNotifications, which it had found by itself. That table held 18 rows, and later 19. The table it was supposed to read, called History, held 4,295 rows.

The log window runs 2026-06-20 to 2026-08-08. Twelve separate runs across that period found the wrong table. Before 2026-08-01 at 21:21:52 UTC there are zero lines anywhere in that log recording anything ingested at all. Not a small number. Zero.

The first correct run, once the table was named by hand, pulled 4,203 rows and 537,023 words. Half a million words of one person speaking, sitting on their own disk, unread, for as long as the log covers.

Every line in that log was true

This is the part worth slowing down for. The log was not lying. Nothing in it was inaccurate. The job started when it was supposed to start. It opened the database it was pointed at. It read a table. It found no new rows in that table, because that table almost never changes. It exited zero.

The script treated no new rows as a normal successful ending, which is a reasonable thing to do, because most nights on a healthy pipeline genuinely have nothing new. A source that never changes produces the identical result to a source that has nothing to report. From outside, on the evidence available in the log, those two situations are indistinguishable.

No error. No warning. No alarm on an empty result. Seven weeks of a green light on a job that had never once done what it existed to do.

The failure hiding inside the failure

When we went back through the log to work out how long this had been going on, we found a second problem underneath the first. The log records no runs at all between 2026-06-26 and 2026-07-28. Thirty one days. For a month the job produced neither data nor a line saying it had run.

Whether it never fired, or fired every night and failed to write a line, we do not know. That information no longer exists. This is the harder of the two failures to catch, because a wrong result at least gives you something to read. An absent line gives you nothing, and a person scanning a log sees a June run, a July run, and no reason to count the days between them.

Four rules we now run on every scheduled job

1. 01Never let a job auto-discover what it should be reading. Name it. Discovery is a convenience that costs you the one thing you needed to be certain about. A job that finds its own source will, on the day something is renamed, quietly find a different one and carry on.
2. 02Assert a floor. Decide what a sensible minimum looks like before you switch it on. If the pull comes back under that number, the job fails rather than succeeds. A failure you can see beats a success you cannot check.
3. 03Report counts on every run, including the empty ones. A line that reads zero new rows out of 4,295 total is a completely different message from a line that reads zero new rows out of 18, and only one of those tells you where you are.
4. 04Never exit successfully on an empty pull without also stating the total the source held. This is the specific rule that would have caught ours in the first week, at zero cost, with one extra number on one line.

The same shape, three more times

Once you have seen this pattern you find it everywhere. Three more, all observed, all with dates.

• Twelve of seventeen test cases passed while the test harness was completely broken. All twelve were cases that expected the system to block something, and blocking is what a dead script produces. A corpse refuses everything. Only the five cases that expected a pass exposed the problem. A test suite made mostly of refusals cannot tell a careful gate from a broken one.
• A plugin installer printed a green success checkmark on a plugin that could not load. Confirmed by test on 2026-08-10. The cause was unquoted colons in a metadata field. The metadata failed to parse, and every component in the plugin loaded with empty settings, silently, while the installer reported that everything had gone in.
• A publish reported failure while succeeding. The verification step ran three seconds after the push, checked six paths that had never existed, got 404 on all six, and called the whole thing a failure. The change had landed. The check was looking in the wrong place at the wrong moment and delivered its verdict with full confidence.

That third one is the inverse of the other two and teaches the same lesson. A check that runs against the wrong thing, or too early, delivers its verdict with the same confidence either way.

Now put this in a real estate business

None of the four below are client cases. They are illustrations, chosen because they have the identical shape to the failure above.

• A drip campaign reporting sent while the list query returns zero. The platform sent everything it was given. It was given nobody. The dashboard shows a completed send, and a send to nobody completes faster than a send to four hundred people.
• A lead router with a rule that stopped matching after a field was renamed. The rule looks for a value that no longer exists. It matches nothing, routes nothing, and errors on nothing, because finding no matches is a legitimate outcome for a filter.
• A transaction checklist that auto-completes because a document was uploaded to the wrong slot. A file arrived. The rule watching that slot fired. The item went green. Nobody has read the file, and the item that would have prompted somebody to read it is now closed.
• A review request automation that has been firing at an empty segment since March. Same shape as ours. The segment definition drifted, the audience became zero, and a send to zero recipients reports the same green as a send to two hundred.

In every one of these the symptom is the same. Nothing errors, and the only sign is somebody going quiet.

That is what makes this category expensive. The quiet gets blamed on the market or on a cold database, because those explanations are available and a silently dead automation is not. Meanwhile the tool reports healthy numbers, and healthy numbers are what people check.

What this discipline does not do

The one sentence to carry out of this: a success response is not evidence. Everything else here is that sentence with dates attached. Read the artifact back from the system that holds it, say what you checked and when you checked it, and treat a green light as a question rather than an answer.