YOU KNOW that moment when you get the creeping sensation on the back of your neck because you know that something you've done has gone very, very, excruciatingly, undeniably wrong?
That's what must have happened to one of the system administrators at GitLab, as it appears that at some point, the company servers have been relieved of about 300GB of live production data, with the folder already down to 4.5GB before the fault was spotted, according to The Register.
GitLab is currently down while the company tries frantically to get everything back. You can watch their progress live on this page.
GitLab launched in 2014 with backing from Khosla Ventures and Y-Combinator as an alternative to services like Github. Github has itself had major outages in the past, once with five days offline thanks to DDoS attacks.
"But where were the backups?" we hear you cry. Well herein lies the rub with the backups seemingly unrestorable, at least while the site stayed online and so the only solution appears to have been to pull the plug while the fix is ongoing.
A Google Doc explains that of five techniques for ensuring data is backed up, "none are working reliably or set up in the first place". Oh my, heads are gonna roll.
The database covers issues and merge requests but repositories and wikis are safe - otherwise the affected code would be much bigger.
It seemed at first that things were just sluggish, but then it was spotted that the data was being chewed up like Pacman on an ectoplasmic binge.
At the time of writing, GitLab says it's 68 per cent done with restoring the database. For the most part, users have been very understanding, with comments like "mankind is not perfect" and "thanks for your work and transparency" buy in the great scheme of being at home to Mr Cock-Up, this is right up there with interviewing a taxi driver on BBC News about Apple Music.
The look of horror on Guy Goma's face is probably the same one the GitLab sysadmin had when he realised what had happened. µ
You're not the voice, try and understand it
Not 'Appy bunnies
News reaches us, per Plex