When you run into trouble with a rebuild, specifically if the rebuild just stops cold (at some percentage), or drops a second drive during the rebuild process, this will be your most vulnerable time, because every instinct screams “try again”, and “trying again” is where you lose one drive and end up with no drives left. Here’s what makes a rebuild fail, three things you do that will make it even harder to fix, and the self-control you need to help an injured RAID array come back alive.
A rebuild tests full intensity on every sector of every surviving disk at sustained levels, on drives of the same age and mileage as the dead drive. It’s a stress test, scheduled for the worst possible week.
An unreadable sector on a survivor. A full rebuild of a RAID 5 requires reading back everything off every remaining drive. If there is just one bad sector on a surviving drive, many controllers will stop reading as soon as they hit that, or drop that drive during the rebuild process. Because modern disk sizes are so large, the probability of encountering at least one bad sector across all of the drives while doing a full rebuild is unacceptably high. That is why large rebuilds tend to fail so much more often than smaller ones did. The batch-twin trap. When identical drives were purchased together and used identically, it was likely that the first drive to die would be followed by its siblings. So when those siblings are asked to help rebuild the RAID set, it is like asking them to run through a marathon. The wrong-drive slip. In times of panic, people mistakenly pull a good drive instead of the dead one, or launch a hot-spare to start an automated rebuild against a dying survivor. Both of these “human error” reasons for failed rebuilds account for an alarming number of arrays that end up in our lab. Whatever reason caused the failure, one thing remains true: the data is still stored on each individual drive, until someone does something else that deletes it.
Force online. The controller removed a member because there was something wrong with it at some point. Forcing it back causes the ‘repair’ process to write bad data (or “rubbish”) into every stripe. Re-initialise or recreate the array. Every controller provides this option for cleaning up the slate, and will write new metadata (and possibly new parity) to all of the members directly onto your volume. When you’ve finished re-initialising your array, it will now be healthy, but your data will need to be recovered. Serial retries. Every time you rebuild, you’ll repeat the intense reading of all of the drives which have already proven they’re unable to handle it. Each subsequent attempt is taking away from your chances of recovering your data. The next time you feel like trying any of these, shut down the array instead.
The protocol for a degraded RAID system fits on a sticky note. Power the box down. A degraded RAID system will remain stable at rest, but will continue to deteriorate when under load. Photograph and label each drive in relation to its bay location prior to attempting anything else. Member order is critical during the reconstruction phase. Touch nothing in your controller’s settings. No configuration save, no repairs, no firmware upgrades during a crisis. Next comes the professional sequence: each member is imaged read-only (including the failed one), although many times the failed unit has 95% of its information available in readable form on less strenuous equipment and holds the stripes the survivors are missing, and then the RAID is rebuilt virtually using those images, allowing experimentation with different parameters without risk. Once your data is extracted from your virtual volume, do not gamble with your original disks. This is the essence of what we do for RAID recovery, whether you have high-end rack servers or low-end desktop NAS boxes. We provide a detailed explanation of how arrays fail in the RAID 10 guide.
In cases where the disk array contains data that is irreplaceable to you, and if no verified backup exists: do not let the rebuild process finish; instead, pause the process if the disk controller allows for it, and halt as much added load as possible. A slow rebuilding process means that the remaining disk that is working is doing its best to read sectors, which will stress it to the point of failure after it has executed maximum intensity reads for several cycles. When the disk array has been paused, but still degraded, there may still be a possibility of performing a recovery without damaging the second drive completely.
Forcing a member online is probably the worst button in RAID. The controller dropped (i.e. marked offline) that disk at some point in time, and the array has continued past that point. Once you force the disk back online, the controller may choose to use stale data as valid data, which would cause corruption in whatever data is currently being written to the disk by using the corrupted parity to validate its own repair of all other data. If the array is important enough to warrant recovery, shut down the system completely and create images of each member. Then you can perform virtual re-assembly without risk of data loss due to incorrect assembly.
Identical bench, identical methodology. Under a desk, a four-bay Synology or QNAP is architecturally a compact RAID server: mdadm or one of its vendor variants running on standard disks. Only send the labelled disks (the enclosures are unnecessary); each disk will be cloned; the array will then be rebuilt using the clone images and return to being available for file access. Additional details can be found on the NAS recovery page.
Free diagnostic within 48 hours on multi-disk arrays in the lab: imaging done before running array logic, quote written before doing any work.