Today, we are sharing a sysadmin story that is worth a read. Exactly a month ago on Saturday 8 July 2023, user accounts started disappearing from the Vivaldi Social, our Mastodon instance.
What was going on, how did this happen, and what were the consequences?
That Saturday, Vivaldi sysadmin Thomas Pike got home after a bike ride (perhaps, he was away for his choir practice. Yes, this sysadmin is also an Opera singer!) and noticed that something was not right when he discovered his home timeline on Vivaldi Social was empty. This got him curious. He checked with a few more users and realized that some accounts had been deleted, and then recreated as a completely new account when the user logged back in.
This is where the investigation started. And there was no looking back! Thomas did a great job of getting to the bottom of the issue and eventually, after very long hours, restored the data.
Thomas has documented the entire weekend that went into solving this issue. We are very impressed with this meticulously written blog post and recommend you to read it ➡ https://thomasp.vivaldi.net/2023/07/28/what-happened-to-vivaldi-social/
And there’s also a TL;DR (too long; didn’t read) section at the end.
Here is a quick incident summary of what happened with Vivaldi Social.
- Saturday 15:15 – Vivaldi Social is sent an account rename message from an external instance. Rogue account merge operation starts in response to this.
- Saturday 15:25 – First signs of the incident are noticed.
- Saturday 17:20 – Account merge operation is stopped after restarting docker containers. 198 accounts in total were deleted/merged between 15:15 and 17:20.
- Sunday 13:00 – Probable root cause is identified.
- Sunday 14:25 – The root cause is confirmed.
- Sunday 21:55 – Data restoration commences.
- Sunday 23:27 – Data restoration completed.
- Monday 10:40 – 6 affected accounts with symbols in username are fixed.
- Monday 11:05 – Lost web settings data is restored.
- Tuesday 15:31 – Incorrect counter values are fixed.
- Tuesday 16:01 – 4 accounts with invalid data are fixed.
This incident reminds us that sysadmins are unsung heroes whose importance cannot be overstated. They are the masterminds behind the scenes, tirelessly working to keep technology systems running smoothly and seamlessly. Without their expertise, there would be constant disruptions, downtimes, and vulnerabilities.
On behalf of all the users whose accounts were affected, here’s a note of thanks to Thomas and everyone who pitched in to resolve the issue.
If you enjoy reading stories by sysadmins, we are confident that this was worth a read!
Do you have any interesting stories, where sysadmins saved the day?
If so, drop it below or on Thomas’s blog.
Want to try a new, reliable social network? Do give Vivaldi Social a try. Our goal to start Vivaldi Social on Mastodon was to offer better alternatives for people to communicate in an algorithm-free environment with no surveillance capitalism, devoid of tracking or dataprofiling. Read more.
