Mayhem

Don't know much about history

Don't know much biology

Don't know much about a science book

Don't know much about the French I took

Sam Cooke, Wonderful World

Don't know much about server administration apparently never made it to the final version.

I currently rent a virtual server from DigitalOcean for Cottonwood's home page. As Starting Over becomes more of a reality — meaning, I have actual posts to publish — I also need a host for the blog. As a company that ostensibly offers several web services from a common source, the obvious solution is to put both products on the same server. This also has the upside of being a great practice run for future multi-service platform offerings.

Not only is this a relatively straightforward task, there is actually a DigitalOcean guide specifically written for this purpose. Perfect.

I set about completing each task in the how-to guide. Modify the current configuration file? Sure! Moving existing (and currently operational) files to fit the recommended file scheme. Why not? It's all part of the guide.

After an hour of meticulous guide-following, problems began to surface. NGINX, the server program, threw an unexpected error. Fortunately, all it took was some google sleuthing and a few on-the-fly fixes to address what was obviously — obviously — a temporary setback.

The next problem was not so much an error but more of a discrepancy with the guide. The server was set to use a service management program, essentially a set of tools for easily starting and stopping other programs "automatically", that differed from the program used in the guide.

This is where things really started heading south. While certainly not the most complicated administrative task, I was completely unfamiliar with either program. I didn't even know my server used "Systemd". Or how that differed from Upstart. Or System V. Or what any of these differences entailed.

Still, I had the guide. I stuck to the plan and hammered out a few of the recommended changes. With a few minor adjustments as needed, of course. A few final touches and then...

Nothing.

Actually, worse than nothing. Not only did the new service not work, even the previously operational business site was now no longer displaying. Cue panic.

Those without "tech" experience probably don't realize how normal a state of affairs "broken" is. One of the most widely used development philosophies consists of:

1) Breaking the program - generally by introducing a new requirement that the program cannot handle
2) Implementing a duct tape and chicken wire solution
3) Tweaking the solution until it is no longer an embarrassment to share with your colleagues

Errors happen all the time and are actually very useful. The difference is that sane developers go through this process in an area that is separated from the "production" code that the outside world sees. In other words, they don't try to implement a new solution on the server that is currently serving their business's live website, much like you wouldn't try to replace your car's spark plugs while doing 80 miles per hour down the freeway.

I set about fixing my mistake. As the hours drifted by, I became more and more frustrated and actually considered a fresh install — essentially the nuclear option. I opted for more research instead. Fortunately, I was able to at least get the base site working again which provided a "pause" button to reset my mental capacities. And take a shower. Roughly five hours after starting, I was basically back to where I began minus a few brain cells.

With more knowledge and better clarity of mind, I was able to get the blog set up painlessly with approximately 45 minutes of work later that night.

Despite the frustration and "wasted" time, the end result was actually not bad:

  • The sites both work as originally intended.
  • I now know the basics of Systemd and am much more comfortable using it. Ditto for simple ownership/permission settings and database admin.
  • I learned a valuable lesson about messing with production code for the relatively low cost of some time and frustration.
  • Most importantly, I experienced first hand how disruptive and demoralizing an uncontrolled "mistake" can be. Even after stepping back from the computer, I had a bad case of brain haze all afternoon as my subconscious attempted to sort the morning's new learnings and innovate additional solutions. Part of the problem was a lack of structure in my day up to that point for several unrelated reasons, but it was a great example of how "mindspace management" is going to be a necessary skill without an external force providing the daily structure of meetings, deliverables, etc.

On a positive note, the feeling when everything finally worked WAS AWESOME. It would have been a trivial task for a tenured system administrator, but watching all the pieces come together for the first time was almost euphoric. At this point, I'll call that a win.

For those interested in the gruesome details, here are a few of the workarounds that were required: