Remember the days maybe ten years ago when we had a sysadmin for every 30 or 40 servers? Those were simpler times when each server in the data center was unique, even if just slightly, from its closest neighbor. A data center with 1000 servers had 30 operators who spent each of their days patching and monitoring and resolving operational issues with ‘their’ servers. Each server was a critical element of the big picture, and the failure of any one of these was typically reason to cancel dinner plans with your significant other.
I have to laugh thinking back on those days and wonder how we survived. We were so tightly wound-up in all of the intertwined technology that most sysadmins didn’t have the luxury to think into the future, and instead spent all of their time trying to keep their head above water and keep what they had running!
The first big milestone came when automated and scalable software solutions popped up that could provision devices according to templates, and then apply those templates to new and existing servers. These templates become ever more capable, and things like patch management became part of the template and automated process. No longer would each device require an entire afternoon of loading software or applying patches. We simply defined any number of servers as a specific template type, and then the provisioning toolsets would keep those servers matching those templates. 1 server or 1000 servers, made no difference. A single sysadmin could now handle 25 times the number of servers in their daily schedule. Instead of managing 30-40 individual servers in the old days, they could now manage in some cases 30-40 RACKS full of servers.
The second game-changer was virtualization. Virtualization essentially broke the 1 to 1 alignment between these physical boxes and the number of actually running servers. Provisioning and update automation combined with virtualization meant that a sysadmin could now able to manage literally thousands of ‘servers’ almost instantly with a high degree of confidence that they were optimized, secured and supportable regardless of what failures might occur. And best of all, since many server instances were now virtualized, the number of walks down to the data center was drastically reduced, and in fact sysadmins could really exist ANYWHERE on the planet as long as a network connection existed. Follow-the-Sun(or moon) strategies popped up that leveraged sysadmins in multiple locations.
Lastly, the decomposition of many volume applications (like web ecomm, search and social servers) using technologies (like Hadoop) allowed these applications to automatically expand or contract to any number of servers without operational intervention which also allowed for failures of any given server to have no perceived effect on the business itself.. With relative ease, now 1000 servers could be tasked for web traffic in the middle of the night, and 10,000 servers could be active in the middle of the day, without any human intervention needed. The applications were resilient, essentially self-healing and users DID NOT need to know or care where their transactions were actually being serviced. Today, most web-centric data centers will have literally dozens or hundreds of physical servers off-line for maintenance due to hardware failure at any point in time, and the process to replace failed hardware device(s) in this type of configuration is considered a normal monthly maintenance task, rather than an urgent business-impacting one.
So, where did those SysAdmins go? Are they working at Starbucks (one of my favorite places on earth) or did they become teachers? Nope! They are still practicing their chosen craft, but doing so at a much higher level, with much more impact and in a fashion that increases their ability to innovate. Their job satisfaction has increased dramatically as their role transitioned from tedium to technology. In most cases these professionals have been given the opportunity to contribute in a much more meaningful fashion, and they feel more tightly connected to the organization’s success. Systems Administrators can now look for innovative ways to support the business in a fashion that either decreases operations costs, or increases business value.