November 17, 2007.
Hard drives are growing at a remarkable rate. What they haven't been doing is getting faster at the same rate as this size growth. This has led to the interesting phenomenon that it is taking longer and longer to read the entire contents of a large almost full hard disk. I wanted to know how much change there has been in the time it takes to read a large disk over the last decade and to see if I can use these figures to make some projections on how long it will take to read a full drive in the future.
To do this I gathered data on the size and average sustained transfer rate of a large and reasonably fast 3.5 inch ATA disk drive from each year going back to 1998. All data came from StorageReview.com as they have an extensive archive of disk drive reviews with performance benchmarks. The data was then used to calculate how long it would take to read each drive and this graph was produced.
The time taken has increased about 8 fold in decade. If hard disks are still the primary method of storage in 10 years time, and if the trend continues, we can expect it will take just over a day to read a large disk. This could have important impacts on the way we use hard drives. People running the Grandfather/Father/Child method of backups will need to modify their routine possibly so that a “child” done every second day instead of daily. Systems that run a full backup at night during low load times will find that backups spill over into peak load periods and will have make adjustments to compensate. Disk utilities like tools that undelete files could potentially take more than a day to get back a files accidentally put in the trash. It may be faster to redo the work or retrieve lost data from a backup than it is to use an undelete utility.
These predictions are based on the assumption the hard drives will still be around in a decade. On the face of it, this seems a safe assumption as disks have been the mainstay of data storage for over three decades. What makes the future less certain is that the cost per megabyte for competing storage technologies, particularly flash memory, has been improving at a remarkable rate and may displace hard disk before a decade is over.
As always, I like to show the data I used for my graphs. It's particularly important with this graph because there are several sources of error that I have introduced when selecting the representative drive for each year. Firstly, the selected drive may not have been the largest available in that year and the chosen drive may also not be the fastest for it's size in that year even though thats what I tried to select. Another error is in the calculation of the average transfer rate. This was done by visually inspecting the transfer chart and estimating the average. The error is likely to be far less than 10% but at least you know some error is in there. I also noticed that average transfer rate also works out to be consistently very close to 2/3rds the way (closer to the max figure) between the maximum and minimum transfer rates If you find a more appropriate drive are more accurate data then you can substitute it in and produce your own graph.
1998 Qantum Fireball Plus KA 18 gigs with average transfer rate 11.3MB/sec.
1999 Maxtor Diamond Plus 5120 20.4 gigs with average transfer rate 14MB/sec.
2000 IBM 75 GXP 75 gigs with average transfer rate 32MB/sec.
2001 Maxtor DiamondMax D540X 160 gigs with average transfer rate 30MB/sec.
2002 Western Digital WD2000BB 200 gigs with average transfer rate 48MB/sec.
2003 Western Digital Caviar WD2500JD 250 gigs with average transfer rate 48MB/sec.
2004 IBM Deskstar 7K400 400 gigs with average transfer rate 53MB/sec.
2005 Seagate Barracuda 7200.8 400 gigs with average transfer rate 58MB/sec.
2006 Western Digital RE2 WD5000YS 500 gigs with average transfer rate 63MB/sec.
2007 Seagate Barracuda ES.2 1 terabyte with average transfer rate 85MB/sec.