So today I discovered that there’s a cron job that holds non-reproducible state that died, and now our system is fucked.
The cron job doesn’t live inside any source control. This morning it entered a terminal state, and because it overwrites its state there’s no way to revert it.
I’m currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.
He does charge a consulting fee to “fix” these issues
Almost all of them are dumb shit like this, where something is built in super hacky and dumbass ways.
It’s his kill switch and he forgot to check in.
Super hacky and dumb? Sign me up 😂
Me running all my services in tmux