One of the topics that has been near and dear to every technologist’s heart over the past dozen years is the skyrocketing consumption of raw storage. Storage space just grows. Everything we do requires storage. Everything we touch changes some kind of file and all those files must be stored over varying periods of time. To make this fact even more sensational, with the speed at which networks perform today, people don’t think twice about sending documents which are much bigger than just a few years ago. Today sending a 5MB Powerpoint to a dozen mail recipients doesn’t even raise an eyebrow! So the rate that we consume storage is also increasing at an alarming pace. Multi-media is everywhere, stored in the back-end to service the social and mobile styles of computing. Those are the basic roots of the problem.
So the industry has reacted by trying to solve the problem from a number of different angles. One simple tactical approach is to limit access, implement retention policies and enforce disk quotas. This essentially forces end-users to think about what they are storing and make-tradeoffs themselves and what to keep and what to trash. Users police their own behaviour and realize quickly that they own their own destiny. I would suggest that this is a band-aid to the problem and will only offset the dire need for a cure by a year or two.
A second and more strategic approach is to implement storage strategies that include smarter approaches to storage. While many products exist in the crowded storage segment, two of the most maturing technologies include compression and de-duplication. But switching over is easier said than done and their has always been a price to pay for this more strategic approach. Massaging data in real-time, without having to re-architect any of the upper layers is the magic here. When done properly, the data is essentially abstracted and then presented upstream as if it were still completely intact and residing individually on media. All makes good sense, but there has historically been a significant premium to pay for doing so.
The economics have always gotten in the way. The speed that this could be done has always been a challenge at the price-point that makes doing so compelling. This is where the world is changing. The processing performance for the required CPUs has greatly increased, the availability of large arrays of really fast solid-state storage has become commonplace, the software to do the massaging magic has become increasingly smarter, and the willingness for customers to think differently has never been better.
So how specifically does the MATH come in to this story? Well, it turns out that the storage makers only recently realized that they can credibly apply “unit costing models” on the PERCEIVED STORAGE rather than the RAW STORAGE. Just a few years ago the cost of a 100GB enterprise magnetic hard disk was running almost $5 per MB. That disk would store 100GB of raw data (and the specifics of how the data was stored didn’t need to be part of the discussion). Today when compression and de-dupe are applied, that same 100GB drive might be able to effectively store 6-8 TIMES that original amount. Assuming that the cost of the drive itself stayed the same, that would mean that an effective rate of 50-60 cents per MB. In this simple example, the 100GB drive could be viewed as if it were a 600-800GB drive!
With such a significant change to the very economics of storage, storage vendors have responded by introducing more intelligent storage, storage that is based upon solid-state devices, and storage that is much more reliable in nature. The best part, and due mostly to this “New Math” (remember that term from the 1970’s?), the cost for bringing the same perceived amount of this much FASTER, really SMART storage to the end-user is LESS that the price that they were willing to pay for slower and less robust RAW magnetic media alone. It’s true! Scalable, ultra-reliable, ultra-fast solid-state storage can be had at the same price point that raw magnetic drives offer today. That is really important for the storage industry as it allows absolute price points to remain the same, and end-users win by having more capabilities at that same price-point. Vendors stay healthy, end-users receive growing value.