$5 per month for a VPS? What could possibly go wrong? Well lots actually. As the saying goes you get what you pay for, so when I decided to create my first droplet with Digital Ocean I wasn’t expecting a whole lot. Some users on HackerNews had rave reviews about them and me being frugal was always looking for a cheaper VPS host. So I created a droplet and copied this site over from AWS to them.
Things were working swell for a very a while, then my internet at home went on the fritz. While using a hotspot to make my last post I also decided to login to my droplet and do an apt update. And that’s when the problems arose.
See, I wanted to configure a proxy that I could use from any hotspot in Paris, since I don’t trust hotspots and I need the internet for banking and accounts (see DNS spoofing) I wanted a connection I could trust. So I installed some proxy onto my droplet. For $5 more I should have bought another. And then the connection died. No http, ssh, nor ping. I could no longer reach my droplet, my website was gone, and I was trying to fix this over a wireless hotspot that jumped APs every few minutes. No problem, I told myself I’ll log into the web console and fix it. The web console never worked on my Mac. So I did the same from a PC running Firefox and that just showed me a black screen.
Rebooting the server showed the server was stuck at init-bottom. This is the point in the boot sequence where grub should exit and the kernel is supposed to take over, and it wasn’t happening. Back at the droplet control panel I tried a few different kernels and stopped at the same point. I sent a ticket to Digital Ocean and the support team suggested I boot using the DO-Recovery kernel. That recovery kernel dropped me into a busybox shell but it didn’t have the tools I needed to access a borked LVM volume and it didn’t have network drivers. So not only could I not access the partition, but even if I did I wouldn’t be able to do a whole lot.
I opened another ticket to get a LiveCD boot (any distro). The support team came through again and got me into a Ubuntu LiveCD shell where I was able to open the volume with this site. Then I realized I had no network access and raised it with DO who straightened it out almost immediately.
With my volume mounted and reachable from the network I created another droplet with the same configuration as the broken server then using SCP I copied over my files, databases, and configurations to the new droplet. Some file permissions had to be edited but everything is right where I left it. And that is where we are today.
So I’d like to thank Digital Ocean and their invaluable support team for helping me get things back on track!