Lessons From Rebuilding a Server

As scary as this statement is, I’m my own SysAdmin. This does not come from choice, mind you, but necessity. Sure, I could farm out server administration like many do, but I’ve never found the complete flexibility and power from such arrangements that having your very own server provides. So I make do.

And “making do” has so far meant that every now and then I need to wipe the thing clean and start from scratch. I’ve done this several times now, sometimes because the server was hacked and other times because I was an idiot and did something from which server wiping was the least painful of solutions. Or at least the most understandable to this accidental SysAdmin.

So recently (ok, like yesterday) I completed such a process yet again. Thankfully, this time I was able to take the time to do a full backup. And I mean full. Here are some of my “take-aways” from this project, which most real SysAdmins will laugh at for reasons ranging from “duh” to “you’ve got to be kidding”. But here goes, nonetheless:

1. Backup everything unique. Everything. Perhaps you are thinking “duh” at this point, but allow me to elaborate. This item covers things that perhaps don’t immediately occur to you. Besides all of your data files there are essential configuration bits that would be difficult to build back from scratch. For example, your web configuration files that are in the directory “sites-enabled” if you are using Apache2. In backing up that directory, I was dismayed to discover that one of them was a sym-link to “sites-available”, so I had to rebuild that one from the default settings. Keep in mind that the directory “mods-enabled” is likely only sym-links to “mods-available”.
2. Don’t forget database files. Typically you will backup your database files using the usual utilities, depending on whether you use MySQL, PostGres, or what have you. I cheated this time and simply copied the database files from where MySQL stored them and it worked fine, as far as I could tell, and was less work and more intuitive to me. YMMV.
3. Document everything you do. I was not very good at this before and I keep paying the price. So this time I was determined to document every step so I didn’t have to figure it out each time. You will be glad you did too.
4. Pay attention to dependencies. Most Unix software you install requires other bits to be installed first. Don’t gloss over these, as you will regret it if you don’t get them successfully installed first.
5. Consider taking this opportunity to change or upgrade things. I flirted with the idea of installing the latest operating system but then decided I could do the upgrade after getting back up and I didn’t want to complicate anything by having to deal with potential problems from the upgrade. However, I did take the opportunity to fix a couple things that had been nagging problems in my last install.
6. Focus on getting the most important sites/services up first. I had my most important sites back up within 24-hours, but my personal web site was down for over a week due to problems and intervening travel. I doubt my fans were too disappointed. It they were, they were very quiet about it.
7. Don’t forget to “harden” the system after you install it. There are various security things you can change to make it harder to hack into your server. Don’t forget to do those or else you may be doing this entire process again soon.

As an accidental SysAdmin I’m sure there are other things that real SysAdmins can point out, which I invite anyone to do in the comments below. I’m being selfish, as I’m hoping to learn how to do this better for the next time. Because I’m absolutely sure of one thing — there will be a next time.