More luck than anything really. It was probably because it had 6 months left and the fact that reading and writing felt slow. Everything else behaved normally and buying a new disk was an educated guess that turned out to be the correct choice.
6 months ahead of scheduled swap, it didn't die, it just started reading and writing really sluggishly, making the whole server behave really weird. Disk smart statistics looked healthy and disk self tests passed with flying colors. Anyway, had to swap it early and do a re-install of the OS.
The rest of my cluster temporarily took over running some pods and only saw downtime for a few pods that were dependent on some disks in the failing server.
I guess the incident has restarted my interest in distributed storage.
I feel like this wouldn't reduce costs, since the load is the same, but just moved to a different daemon, in this case nginx. I for one, pay for bandwidth on my VPS, so the cost for me would be the same.
One thought I've had, is to use a slow loris technique combined with a small pool of connections and an ai poisoner, to keep the scraper occupied for as long as possible, without using a lot of bandwidth.
Man, I know that feeling. One thing that helped me better deal with issues like this, was to have a changelog. Basically I write down what a setting was, what I changed it to and a reason. If something goes wrong, I can at least undo what changes I've made and see if it helps. It's not perfect, but it might shave some hours off a RCA.
Yeah, im in disbelief. It was just weeks ago when they reported a website was spamming the parliament with emails. Now fucking Denmark is trying again, WTF Denmark!!!
Absolutely agree. I have been thinking of starting a selfhost guide that takes you through the different ways to selfhost and the basic concepts of it, but gave up because I'm a shit writer and my experiences are mostly docker, k8s and Terraform/OpenTofu.
I run a pretty barebone Archlinux with several distroboxes. My main motivation for this setup is that I work on a lot of different projects that all have very different setups. Running them in distroboxes make sure I can just drop the box, once the project is finished, and all code and data is just wiped, without having any impact on my main setup.
Cool, is there beer? Python probably.