Lee Hutchinson Ars Technica | Published a day ago
Moore's Law gets all the press. It's easy to present even to non-technical readers, and the way it's most often expressed is something like, "computers double in speed every year," though that's a bastardization of the axiom, which actually states that the transistor count of integrated circuits tends to double every eighteen months or so. This formulation does succinctly capture how fast computers have gotten in so short a time.
But integrated circuit density hasn't been the only computing tech which has shown extremely rapid progress over the past thirty years. Consider magnetic storage. Modern hard drives are precisely manufactured miracles, products of billions of dollars and decades of research into magnetism and quantum mechanics, squeezing ludicrously large amounts of data into ludicrously tiny spaces. A hard drive with about three terabytes of capacity can be had for less than $150 today; a PC equipped with two or three of these would have more on-board storage than most large enterprises had in aggregate even a decade ago.
That kind of inexpensive capacity has revolutionized the way people keep and use data, both at home and at work. From complex storage- and compute-intensive tasks like oil and gas upstream processing all the way down to editing a home vacation video, the ability to store and manipulate increasingly voluminous data actually drives serious innovation.
A PCM .wav file of a three-minute song might be 30 or 40 MB, while an mp3 of the same song might be 3 MB. If all you've got is a 100 MB hard disk to work with, even the ten-fold decrease in file size doesn't make keeping an mp3 collection practical. But in 1995-6, when mp3s began to flow back and forth on the USENET and FTP and websites, the format suddenly became an extremely attractive way to collect music. With most folks having a handful of gigabytes, having an entire music collection available on a computer suddenly became possible. Copious local storage paved the way for entire new industries—think of early iPods and the iTunes store.
(Despite the sea change currently underway to solid state disks, there's every indication that spinning magnetic storage will continue to be with us for years to come—Seagate already sells a 4 TB external 3.5-inch hard drive.)
But data in and of itself doesn't do much. In this era of YouTube and Facebook, data is useless without a means to transport it from one place to another, a way to share it with others. Unfortunately, data transport has not been an area where the brain-smashing innovation race in integrated circuits and magnetic storage has been mirrored. Just as the density explosion in storage has opened the door to people and companies doing tremendous new things with computers, the lack of commensurate scaling in commonly available telecommunications has greatly hamstrung people and industries.
This has led to one of the most annoying paradoxes of the Internet age. On one hand, the massive increase in storage density encourages creation at all layers, from the biggest of businesses leveraging the ability to analyze and mine targeted data out of petabytes of raw information down to the adorable old couple trying to make a video on their home computer (both of these things can eat up a lot of hard drive space!).
On the other hand, the asymmetrical nature of most broadband solutions available to consumers in the US and Europe and a stagnation in their speed encourages only consumption at the "lower" levels of that stack. Companies that need both the ability to transmit and receive data over distance can usually afford to pay for symmetrical high-speed network links, while consumers (at least in the US) typically can pick from two choices for Internet access—DSL or cable. Both access methods typically provide plenty of download bandwidth for Netflixing and iTunesing and YouTubeing, but comparatively tiny upload bandwidth for sending data (most DSL and cable Internet plans have upload speeds that are less than 25 percent of the download speeds).
This asymmetry of access leads us to a strange place, where most folks have the ability to store and create more amazing things than ever before, while at the same time they lack the ability to quickly and easily share any of those things with each other.
Storage through the decades
Let's take a look back at a typical consumer's (i.e., my) hard disk drive at five-year intervals over the past three decades:
The chart on the top shows what appears to be a roughly linear progression of capacity in storage, but the vertical axes actually increase in powers of ten, showing almost a true exponential growth curve. To better appreciate the growth, consult the chart on the bottom, which uses fixed vertical axes. Starting at 5 MB in 1981, we jump quickly to 20 MB in five years, then to 120 MB in five more years, then to 2 GB, to 40 GB, to 500 GB, and finally to 3 TB.
The capacity increases get truly staggering near the end. This alone doesn't tell us much, since it's an accepted bit of conventional wisdom that hard drives will always grow larger over time. The important thing about this increase in storage capacity isn't that it has let us store more data, but that having fast random access to more data lets us do more with that data.
No one keeps files for the sake of keeping files. Even digital hoarders who have every single episode of Doctor Who ever created don't simply have those files to have files—they have those files because they comprise a collection of data with value as a collection. An Excel or Word document has no value in and of itself; rather, documents gain value when they are consulted or shared, when the data that they contain is compared against or integrated with other data and transformed into new data. With this in mind, the exponential increase in storage density coupled with a decrease in how much it costs per-gigabyte to purchase that storage has driven a huge shift in what it means to create and innovate.
In the early '80s, microcomputers like the IBM PC and the Apple II were used largely for traditional computing tasks—crunching numbers or programming applications that would be used for crunching numbers. (Well, that and games.) A 5MB ProFile hard drive for an Apple Lisa cost about $3,500 when it first became available for that system in 1981, but the price could be justified by the fast and random access that ProFile offered to what at the time was a significant amount of data (equivalent to dozens of double-sided floppy disks). Because of the cost, though, home computers didn't typically have internal hard disk drives until the late 1980s. As their storage capacities increased, so did the things that computers were typically used to do.
Serious music production at home, a trend that arguably got started with module tracking on the Amiga platform and spread rapidly in the early 1990s to the PC, relied on bigger hard drives to hold the digital samples that made up the music files. Accelerating storage density pushed this kind of music production out of the basement and into the home studio, today letting artists like Pomplamoose record and produce entire albums on personal computers without relying on time in hugely expensive recording studios. Entry barriers into a recording career are being quickly demolished and a tidal wave of skilled and talented artists are appearing, doing things that would have been impossible even a few years ago.
Terabytes of fast disks available in most home computers also enable software like iMovie and Windows Movie Maker to exist and thrive, bringing movie-creation tools to the average home user. Not everyone is a talented auteur with a Citizen Kane waiting inside —in fact, most folks tend to create videos of their babies farting or themselves farting or animals farting—but folks like Neill Blomkamp or Bruce Branit have also risen above the noise to create some amazing things.
This sort of storage-driven innovation holds true for the enterprise just as much as it does for the home—the fact that I've been linking to YouTube videos above is telling. Companies like Google (which owns YouTube) wouldn't exist without huge amounts of storage. Google began out of a need to locate and organize information, and it manages and stores petabytes and petabytes of data. Even more to the point, it does so in a way that underlines the commoditzation of information technology in general—most big enterprises these days store data in large monolithic storage arrays, but Google keeps all of its data on throwaway servers with consumer-grade hard disks, relying on redundancy and smart design to route around failed components. As storage densities continue to increase and the transformational things that people can do with that storage scales, more and more enterprises will shift in Google's direction.
Advanced trending and data analytics can also be done now with more data available—for example, T-Mobile was able to take the huge amount of network and subscriber data they store, sift it, and come up with insights into why customers leave (disclaimer: the linked article mentions EMC, and I'm an EMC employee). Having more data at hand lets us make connections or create new things that simply wouldn't have been possible without massive storage.
There's a darker side to all of this shiny creativity, though. It can still be too hard to share it with the world.
Storage enough at last?
There's a famous episode of The Twilight Zone named "Time Enough at Last," wherein a bookish fellow played by Burgess Meredith laments that there's not enough time to indulge in reading. He decides to take his lunch break in the vault of the bank where he works, and while he's down there, the Commies nuke everything. He emerges from the vault stunned at the ruins around him and, finding no one alive but himself, eventually decides to take his own life. Instead, though, he spies a library in the distance; he approaches it and, finding that it's stocked full of undamaged books, realizes that his prayers have been answered—he finally has time, time enough at last to read all he wants! The hook, though—spoiler alert from 1959!—is that as he sits down to read his first book, he breaks his glasses. The episode closes with Burgess howling about the unfairness of it all, amidst stacks of books that he will now be forever unable to read.
This parallels the situation most consumers (at least within the US and much of Europe) find themselves today. We have, finally, storage enough at last to do just about anything we want. We can create and keep terabytes of digital stuff, but the value of that stuff is in what you can do with it, and a home movie doesn't do you any good sitting on your hard drive—its value comes from sharing it. Increasingly, that sharing takes place over the Internet.
And sharing is a problem. Let's take a look at the typical amount of inbound and outbound network bandwidth a US home might have had for the same years in which we look at hard drive capacity. Internet connections to the home weren't terribly common before about 1994, but those of us old enough to experience the days before USENET and the Web will remember local BBS systems and fledgling "information services" like Prodigy and Compuserve, all of which served many of the same purposes then that the Internet does now—connecting with others and exchanging files and messages.
From 300bps in 1981, we travel through 1200bps and the halcyon days of local BBSs at 14.4Kbps, through the brief 56K era and into broadband, going from 1Mb to 3Mb to the 10Mb (and beyond) connections common today. The same kind of exponential-ish increase in speeds is evident, especially in the break between 1996 and 2001 when the widespread shift to broadband connectivity began to pull folks out of needing to use modems over POTS and onto DSL and cable Internet access. However, with that came huge gulfs between the speed at which a person could download things to his computer versus the speed at which he could upload things from it to others.
To be sure, upload/download asymmetry has been around for a long time. Back in my own BBS days, I used to look with envy on the guys with USRobotics Courier HST Dual Standard modems, which could talk to other Courier HST modems at a ridiculously fast (for the time) 16.8Kbps and at the same time have a 450bps back-channel available for control traffic. Similarly, most folks who lived through the 56K days remember that 56K was really only one way—traffic going out from your modem was still limited to 33.8Kbps (and in reality, most folks only saw 22-26 Kbps). But nowhere is the asymmetry as great as with modern cable and DSL connections. The practical reason behind the split between up and down speeds is one of frequency utilization—there's only so many hertz available to carry a signal, and so more of the available frequency space is allocated to carry information to the consumer than from the consumer.
Unfortunately, this common asymmetric split now affects a good chunk of the world's Internet users.
You can't take it with you
Most folks carry what feels like significant chunks of our lives on our hard drives—we express ourselves through the things we create, and we keep things that have powerful meaning for us. A hard drive crash where data is lost can lead to not just technical or financial problems, but actual personal anguish—our drives are surrogates for our memories, and if they die we can lose parts of ourselves.
And yet, with all the innovation that has arrived as a consequence of more and faster data storage, most lack the ability to easily move that data around. We can download multi-gigabyte high definition movies from iTunes or NetFlix in astonishingly short amounts of time, but how many hours does it take to upload a multi-gigabyte file?
Amount of time it would take to upload an entire hard drive at each point in time
Companies like Microsoft and Apple and others continue to roll out cloud sync services, which among other things suggest you put your important files "in the cloud" so they can be accessible anywhere—but why should that be necessary when they could just as easily be accessed in situ on your own computer? I have hundreds of movies that I've ripped from DVDs that I own just sitting on my NAS; why shouldn't I be able to access them wherever I am and stream them to myself at a friend's house as fast as NetFlix can? Why shouldn't I be able to get to any part of my digital identity from anywhere in the world?
Cloud backup services like Backblaze or Mozy have their place, too, providing off-site storage for important data in case of a disaster, but it can take weeks—literally weeks—to upload a few terabytes of data. Most people today don't bother with this kind of backup service, precisely because it takes so long to populate initially. Worse still, for people with metered Internet connections, that initial data copy will almost certainly count against your 50 or 150 or 250 GB monthly cap, which might lead to your Internet access being terminated without explanation by your ISP.
The lack of upstream bandwidth in a typical consumer broadband connection holds back some of the innovation that could come from huge amounts of storage. For instance, with a 10 Mb upload connection to go along with your 10 Mb download connection, content sharing sites like YouTube suddenly lose some of their relevance. "Digital locker" services like RapidShare and MegaUpload serve fewer purposes. An entire infrastructure of sites and services that have evolved as buffers and workarounds for tiny upload speeds are made redundant and can be done away with.
Do more with more
Even if our wings remain clipped by upload restrictions, we're still innovating through storage. We have more and faster storage available to us at home and at work, and the things we can do with that capacity continue to evolve. Parkinson's Law states that "work expands so as to fill the time available for its completion." A common corollary to that law is that "data expands to fill the space available for storage." Put even more simply, as the ghostly voice said to Kevin Costner in Field of Dreams, "If you build it, they will come."
Nothing exists in a vacuum, of course, and it's disingenuous to suggest that increases in storage capacity can alone be responsible for anything, much in the way that it's wrong to say an increase in CPU clock speeds alone is responsible for anything, or that the change from IDE to PCI buses in a PC was by itself responsible for anything. However, the rapid increase in storage capacities to which the average consumer has access has removed barriers and created big new opportunities.
