So today I discovered that there’s a cron job that holds non-reproducible state that died, and now our system is fucked.

The cron job doesn’t live inside any source control. This morning it entered a terminal state, and because it overwrites its state there’s no way to revert it.

I’m currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.

  • wise_pancake@lemmy.caOP
    0·
    2 months ago

    What’s extra frustrating is the previous guy did create a git repo of these types of hacks, but this one doesn’t live in it for no discernible reason.

  • arotrios@lemmy.worldEnglish
    0·
    2 months ago

    This is almost exactly what happened to me on Monday, resulting in a fifteen hour day.

    My particular jenga piece was an Access query that none of my predecessors had deigned to document or even tell me about… but was critical to run monthly or you had obsolete data embedded deep within multi-million dollar reports.

    Thank god I don’t work on salary anymore, or I’d be really upset.

  • MonkderVierte@lemmy.ml
    0·
    2 months ago

    Only tangentially related, but “What a elegant house of cards” is an insult i’m going to use someday.

  • grrgyle@slrpnk.net
    0·
    2 months ago

    I have also mixed up crontab -l with crontab -r. 😔

    Let this be a lesson to start versioning your crontabs.

  • Godort@lemm.ee
    0·
    2 months ago

    Time to restore a whole machine backup to a VM with no network connectivity, and manually pull the command?

    • wise_pancake@lemmy.caOP
      0·
      2 months ago

      I was able to do that

      Turns out there was a second bug which triggered this one, and a bug I found in this script that I thought was responsible was happening silently for months.

      Now three bugs are squashed

  • LainTrain@lemmy.dbzer0.com
    0·
    2 months ago

    Cron job that evals some base64 encoded string which is actually downloading a script from a personal GitHub repo of an IT guy who left…