The Art of Consolidation - Part I - Planning Ahead

Posted By Gabe E. Nydick

Friday at 12:11PM EDT

No Comments

Categories:

Infrastructure, Technology

With all of the talk these days about consolidation and Xen as the keystone to that process for affordable, DIY environments, there’s something lacking from the tips, tricks, and how-to’s on getting started. I see a large focus on setting up cluster file systems, SANs, and other shared storage solutions mixed with heartbeat to provide a level of resiliency; but small shops trying to deal with the added layers of complexity of HA storage also often don’t realize what they’re getting themselves into.

With the added hardware efficiency virtualization brings, the eager and well intentioned systems person can and will now run more servers than before on less hardware.

• That 3rd DNS server can now be setup…
• Just one more image server…
• I’ll clone everything just to have hot and/or cold standbys…

Approaching a new, consolidated, virtualized environment requires taking some steps back and rethinking the basics before adding on the complexity virtualization itself imposes, let alone, those of exotic and esoteric storage systems.

What are those basics? Let’s talk about the problems that plague your every day systems administrators.  As those of us who’ve been through the start-up mill know, systems are usually put in place initially by non-systems people, then handed off to inexperienced systems people, only to be inherited by experienced technologists, at some point, usually at a point where the systems have grown to an unmanageable number and the effects felt are what prompts the hire.

Here are just a few of the many problems…

• User authentication is not centralized - Every time someone is hired or terminated, user accounts need to be managed on every system that user is going to touch. I personally have come into environments where that means hundreds, if not thousands of machines need to be logged into just to grant or revoke access.
• User identification is not consistent - Again, through organic growth, the UID/GID of each individual user/group is different on many machines, making the process of clean-up in the future quite difficult as well as doing many maintenance tasks like having to do an emergency re-purposing of a system to cover another role.
• User authorization is not centralized - Just like with authentication, the underlying systems that grant or deny access, like PAM and your password, may have to manually updated on thousands of machines.
• Lack of standards - different operating systems - As much as we all love Linux and open source in general, that love is often a source of pain. Over the years, a company will accumulate RedHat and Slackware systems from the past, may switch to FreeBSD for a while, then jump to Ubuntu or OpenSuSE (my personal favorite). Now, differing configurations, kernels, user space utilities, threading models, memory and file system managers/schedulers, all contribute to so many unexpected problems for day to day operations. Aside from that, the knowledge domain needed to maintain these systems is too wide and results in either a staff that is less than knowledgeable enough or over-priced and under-tasked.

Those few points are enough to result in the following deficiencies

• Change management processes are difficult if not impossible to support because of too much manual work
• Lack of automated installation
• Standards are non-existent
• Lack of security in general
• Inability to audit: security, availability, manageability, and scalability

The fixes are simple and quite elegant in practice and I’ll be posting more on each of them. Upcoming posts will focus on…

• Single Sign On
• Getting Xen Working on the First Try
• Automated Installation
• Cloning Xen VMs
• and more…