Storage Impermanence: Now It's Personal
About that failing hard drive I mentioned yesterday…
To the best of my recollection, and across the innumerable HDD-inclusive systems I’ve used over the years, I’ve only had a few hard drives fail on me:
- There was the MicroDrive whose arm got stuck partway between parked and active, and which I fixed via a vigorous tap to the case per IBM’s recommendation
- The 3.5" HDD that, in spite of being behind a robust surge protector, didn’t survive a premises power spike…along with the motherboard it was connected to…along with the floppy drive and two optical drives also connected to the motherboard…and the PC power supply that fed them all…
- The 2.5" HDD in a NAS, and
- The overheated 3.5" HDD in a NDAS
As of Sunday afternoon, you can add one more drive to the list, although for the life of me, I can’t get the error to repeat itself (which actually isn’t terribly surprising…keep reading). After doing a bunch of network writes to my first-generation Buffalo TeraStation, I happened to glance at it (in a corner of the living room) as I walked by from the office to the kitchen, and with alarm I saw blinking red error indicators both at the general Diagnostics and Drive 1 Status/Full LED positions. Since the drive was configured as a four-HDD RAID 5 setup, i.e. combining striping for performance with parity for redundancy, the NAS was still running. But I’d lost my extra-drive ’safety net’.
At this point, since the NAS was still accessible, I should have immediately backed its contents (which included several hundred dollars’ worth of Apple iTunes Store-sourced Grateful Dead Dick’s Picks albums that I hadn’t yet burned to CDs as backup) to an USB-tethered or another networked drive. But no…after logging into the TeraStation’s web server and confirming the RAID error status, but seeing no means of rebuilding the array as-is, I rebooted the NAS…forgetting that by default I had it configured to automatically shut down upon detection of a RAID error (why it didn’t do this the first time, I haven’t a clue).
The TeraStation rebooted…and shut down, conveniently still communicating via front panel LEDs that drive #1 was the culprit. Inside the NAS were four 160 GByte Western Digital Caviar 160 GByte 7,200 RPM PATA HDDs. Let me be abundantly clear upfront that I’m not at all being critical of WD. The NAS had been up nearly 100% since the summer of 2005 and, because the TeraStation didn’t support HDD auto-spin down after a period of non-access (one of my many criticisms of the unit), the HDDs were also up nearly 100% of that nearly 3 years. HDDs, like anything, are ultimately impermanent…that’s, after all, why I went with RAID 5.
My storage unit stash included an abundance of spare HDDs. Unfortunately Truckee, CA was in the midst of another massive storm, and there was no way I was going to risk life and limb (in what would likely be a fruitless attempt anyway, given the road conditions) going out in it to retrieve a replacement drive. So I spent the next 24 hours sweating over my crippled NAS, remembering all the invaluable data that was stored on it and wondering if I’d be able to revive it.
Monday afternoon found me at the dug-out storage unit, pawing through a plastic bin filled with 2.5" and 3.5" HDDs, optical drives and other mass storage miscellanea (see…I’m somewhat organized). Amidst the diversity of PATA and SATA HDDs, I found only a single drive (from Hitachi) that fit all the necessary qualifications to join my compromised RAID 5 cluster. I headed back home and proceeded to disassemble the TeraStation. Let me tell you…hot-swap, it’s not. The verbiage in the user manual is classic (bolded emphasis is mine, as are misspelling corrections):
If a STATUS/FULL LED is blinking, note the drive number before continuing. Use a clean, padded work area to disassemble your TeraStation. You’ll need a #2 Phillips screwdriver. You will be removing and replacing a total of 22 screws to replace a hard drive, so keep each screw that you remove carefully in a safe place. Be careful not to drop the TeraStation, or cut yourself on sharp interior metal parts. Buffalo Technology is not responsible for any damage that you do to yourself or your TeraStation while changing out a hard drive! Be careful.
Suffice it to say that the company wasn’t exaggerating much, but I still succeeded in negotiating the transplant surgery. When I re-booted the TeraStation, I got the same error LED pattern as before, but the system was thankfully smart enough to realize that it had a fresh drive and didn’t automatically shut down this time. And when I logged in to the web interface, here’s what greeted me:
To clarify: the first time I saw the above screen, all four drives’ boxes were checked and grayed out…therefore my confusion as to how to proceed (which the product documentation didn’t alleviate).
Continue reading with ‘RAID: Not A Perfect Panacea‘…
