How storage has evolved to keep up with our growing needs

Whenever moving data stops, it needs to be stored. Crucially important to our daily lives, as David Shapton reports, storage is now highly evolved and almost unbelievably capable.

Storage never stands still. It's constantly improving: getting faster and bigger. It also invites questions because the main types of storage (hard disks, SSDs, memory cards and tape, to mention a few) tend to look the same, even though their storage capacities might have grown by a thousandfold. So you never quite know what you're looking at.

There's a kind of duality with storage. Digital storage stores bits. But it also stores text, video, and pictures of roses, the King of the United Kingdom, warthogs and kittens. Storage preserves music and poetry, engineering diagrams and medical procedures.

At one level, bits and bytes are neutral: without meaning in isolation. But in combination, they can store the wisdom and knowledge of an entire civilisation.

We live in an era of rapid and continuous change, caused in part by technology or at least facilitated by it. For a long time, storage has been a "rule of thumb" empirical guide to progress in computing and related fields. And progress has been staggering by any measure.

For example, in the mid-'90s, a 600 MB SCSI drive cost £2,700. That's a hard disk with a smaller capacity than a CD ROM. Around the turn of the century, a 300MB USB Thumb drive cost £380. Today, you can buy an SD card with 128GB storage - a thousand-fold increase in around twenty years - and it costs less than the 2001 device. Other types of storage have shown even greater gains.

Evolutionary niches

Storage comes in almost as many versions as species in a rainforest. Each has evolved and adapted for a specific role. In the same way that motor vehicles come in diverse forms like racing cars and articulated trucks, storage can be fast or slow, cheap or expensive, rugged or resolutely untransportable. And - still today and some way into the future - solid state or electro-mechanical.

You only have to look at how storage is employed to understand the diversity in storage types. For example, terabytes of tiny transactional data storage will be optimised differently than a repository for gigantic media files. "Online storage" for video editing and grading will be radically different to "offline" or "Archive" storage, where the priorities are capacity, low cost and longevity rather than outright speed.

Safety is another crucial factor. Note that safety isn't the same as security. For example, a secure drive might include encryption to render its contents unusable or inaccessible by anyone without the correct credentials. A safe drive will have a radically lower chance of losing data through corruption or failure than drives that might be faster, but with no protection.

Incredibly, tape is still an essential medium. LTO is still being developed and has a roadmap all the way to an incredible petabyte per tape. (A petabyte is a thousand terabytes, a million gigabytes.) Capacity typically doubles with each generation, and, luckily, so does the speed of access or retrieval. Readily available software has simplified previously arcane and frankly hostile user interfaces to make the tape experience less painful and more on a par with using a disk drive.

At the other end of the speed spectrum is the astonishingly fast NVMe format. Some storage appliances combine multiple NVMe drives in a single unit, protected by RAID 6, and paired with as many as eight Thunderbolt ports. Unfortunately, they're not cheap, but the price is almost irrelevant if you need something as fast as this.

Fast or massive storage would be useless without interconnects that match their performance. Early SCSI drives had huge parallel interfaces that were as thick as a hosepipe. But parallel interconnects have a hard limit on their bandwidth, as the faster the connection, the more likely that bits will arrive poorly timed. Serial connectors fix this problem and can go much faster.

Modern storage interconnects are easy to use and mostly "plug-and-play". The faster drives get, the more impressive the technology that powers these almost miraculously quick connections. One drawback is more expensive and typically shorter cables, but you can easily add fibre to extend cable lengths almost indefinitely.

Today, with abundant solid-state storage, it seems almost incredible that mechanical drives are still available. But there's a good reason: they're cheap and big. For fast and vast online storage, they are still the best choice.

External storage architectures have evolved into speedy, reliable and inexpensive topologies. You can choose from SAN, NAS and hybrid solutions involving a mix between local (on-prem) storage and the cloud.

Capacity and speed just are the start

As drives become increasingly sophisticated, it's worth remembering that drive capacity and speed don't tell the whole story. We tend to think of drives as "hardware", but that's increasingly inaccurate. Inside even the most uncomplicated drive, you'll find a spectrum of software to manage the internal workings, prevent errors, operate caches, and set up data buffers. Externally you can buy software that will orchestrate data transfers faster than relying on file systems like Windows and macOS. You can measure a storage system's raw speed, but you always have to look at it in the context of the actual work you're planning to do. Small files behave differently from big files. Transferring data slowly brings out different characteristics than when you move it quickly.

We live in a golden storage age where solid-state drives have transformed our laptops and data centres, but mechanical drives still have an essential role. Speedy interconnects give us new topologies and lets us externalise storage that would otherwise have to be inside our computers.

Fortunately - and remarkably - storage has kept up with the almost insane growth in data over the last few decades. For at least twenty years, if you replace your computer every three years, you'll have been able to copy your entire drive onto the new drive and still have plenty of room. Each time one storage technology runs out of steam, another one appears.

Today's storage is a testimony to ingenuity and an absolute necessity for modern living. And for creative people, our entire life's work will only exist in the future thanks to the genius behind today's almost miraculously capable storage devices.