Phase 14 — The orchestrator, and the binary nobody knew was load-bearing
31 AUG AT 11:46 AM

Phase 14 — The orchestrator, and the binary nobody knew was load-bearing

0 LOVES 3 VIEWS
It held the only writer that demotes a runtime state back to offline, and it had been serving production for fifteen days from a closed session's scratch directory.

The only writer that demotes

A binary called agent-worker had been serving production for fifteen days from a closed session’s scratch directory under the system temp folder. One cleanup and it would have been gone.

⭐ It was not “a process in the wrong place”. It contains the only writer anywhere that demotes a runtime state back to offline. An enumeration of all sixteen writers established the whole lifecycle for a bridge:

  • Created — written offline.
  • Promoted once, by the agent’s own heartbeat, to idle.
  • Demoted on staleness — and this is the only path back.

Without the third step, the second is terminal. idle outlives the process it describes, forever.

⭐ And that is the mechanism of this program’s founding incident. A bridge read idle for two days while dead. Until the writers were enumerated, nobody knew why — and the answer is almost certainly that the demoter was not running then either.

Pushing Tin exists because of a symptom whose cause was a binary nobody knew was load-bearing.

It now has a make target, deploys into the same manifest as the bridge binaries, and appears in both runbooks.

A cutover that fails while looking like it worked

⭐ The finding that outweighed the deploy target. The skill API binds with a fatal, so when two orchestrators race, the new one dies and the old one keeps serving. A wrong-order cutover therefore fails while looking like it succeeded — with no signal at all until the temp directory is eventually cleaned and the real server vanishes.

So both runbooks now verify by which pid holds the port, never by “the port is up”. Those are different questions, and only one of them distinguishes your new process from the ghost of the old one.

The same shape appeared again in the collision between the development and production orchestrators, which share a default port. The measured detail is what makes it worth writing down: the worker completes config load, seeds its skills, and announces every subsystem by name before the listener fails. The log reads as a healthy start right up to the final line, so it presents as a broken build rather than a port conflict — and nothing in the output mentions production at all.

The staleness banner was built from a stamp that already existed and nothing read — no new writer, no schema change. Its threshold is read from the row’s own expiry rather than frozen in Go, because the interval is database-managed and a hardcoded constant would be exactly the defect an earlier phase spent itself removing.

⚠ Two defects in this phase’s own files were fixed before it closed, on the principle that a phase does not close leaving defects in the thing it just built. One was a runbook whose summary contradicted its own section 216 lines later — and a reader trusting the summary would get precisely the failure that section exists to prevent.

Pushing Tin — managing a bridge fleet from inside the product
Pushing Tin — managing a bridge fleet from inside the product
Aug 29, 2026 Pushing Tin
← Back to Pushing Tin