May 17, 2021

Lessons From Rebuilding a Server

serverAs scary as this statement is, I’m my own SysAdmin. This does not come from choice, mind you, but necessity. Sure, I could farm out server administration like many do, but I’ve never found the complete flexibility and power from such arrangements that having your very own server provides. So I make do.

And “making do” has so far meant that every now and then I need to wipe the thing clean and start from scratch. I’ve done this several times now, sometimes because the server was hacked and other times because I was an idiot and did something from which server wiping was the least painful of solutions. Or at least the most understandable to this accidental SysAdmin.

So recently (ok, like yesterday) I completed such a process yet again. Thankfully, this time I was able to take the time to do a full backup. And I mean full. Here are some of my “take-aways” from this project, which most real SysAdmins will laugh at for reasons ranging from “duh” to “you’ve got to be kidding”. But here goes, nonetheless:

  1. Backup everything unique. Everything. Perhaps you are thinking “duh” at this point, but allow me to elaborate. This item covers things that perhaps don’t immediately occur to you. Besides all of your data files there are essential configuration bits that would be difficult to build back from scratch. For example, your web configuration files that are in the directory “sites-enabled” if you are using Apache2. In backing up that directory, I was dismayed to discover that one of them was a sym-link to “sites-available”, so I had to rebuild that one from the default settings. Keep in mind that the directory “mods-enabled” is likely only sym-links to “mods-available”.
  2. Don’t forget database files. Typically you will backup your database files using the usual utilities, depending on whether you use MySQL, PostGres, or what have you. I cheated this time and simply copied the database files from where MySQL stored them and it worked fine, as far as I could tell, and was less work and more intuitive to me. YMMV.
  3. Document everything you do. I was not very good at this before and I keep paying the price. So this time I was determined to document every step so I didn’t have to figure it out each time. You will be glad you did too.
  4. Pay attention to dependencies. Most Unix software you install requires other bits to be installed first. Don’t gloss over these, as you will regret it if you don’t get them successfully installed first.
  5. Consider taking this opportunity to change or upgrade things. I flirted with the idea of installing the latest operating system but then decided I could do the upgrade after getting back up and I didn’t want to complicate anything by having to deal with potential problems from the upgrade. However, I did take the opportunity to fix a couple things that had been nagging problems in my last install.
  6. Focus on getting the most important sites/services up first. I had my most important sites back up within 24-hours, but my personal web site was down for over a week due to problems and intervening travel. I doubt my fans were too disappointed. It they were, they were very quiet about it.
  7. Don’t forget to “harden” the system after you install it. There are various security things you can change to make it harder to hack into your server. Don’t forget to do those or else you may be doing this entire process again soon.

As an accidental SysAdmin I’m sure there are other things that real SysAdmins can point out, which I invite anyone to do in the comments below. I’m being selfish, as I’m hoping to learn how to do this better for the next time. Because I’m absolutely sure of one thing — there will be a next time.

Roy Tennant About Roy Tennant

Roy Tennant is a Senior Program Officer for OCLC Research. He is the owner of the Web4Lib and XML4Lib electronic discussions, and the creator and editor of Current Cites, a current awareness newsletter published every month since 1990. His books include "Technology in Libraries: Essays in Honor of Anne Grodzins Lipow" (2008), "Managing the Digital Library" (2004), "XML in Libraries" (2002), "Practical HTML: A Self-Paced Tutorial" (1996), and "Crossing the Internet Threshold: An Instructional Handbook" (1993). Roy wrote a monthly column on digital libraries for Library Journal for a decade and has written numerous articles in other professional journals. In 2003, he received the American Library Association's LITA/Library Hi Tech Award for Excellence in Communication for Continuing Education. Follow him on Twitter @rtennant.


  1. Steven Schwengel says:

    Yep, I’m with you on this one! As scary as it sounds, the benefit of having your own server is the admin privileges of root (Super User or ultimate Admin privileges as to who can do what and not getting shut out of installing some little unknown utility that won’t work under someone else’s restrictions).

    You don’t have to be beholden to work around installs like having to do phpize a php package or other work around install headaches. But, with the privileges of running your own server rest the responsibilities of possible failure that rests on your own shoulders as well. What is the old saying “with power, or root access, comes responsibility” :) Root on within reason … keep calm and plan on …