Two disks down: there are only two correct moves
The first is to power the server off. The second is to call someone who recovers data. Everything in between — Force Online, reseating, initializing, rebuilding — can turn data that was still salvageable into data that is gone.
That is not a scare tactic. RAID5 tolerates exactly one disk failure. Lose the first and the array goes degraded but keeps reading and writing. Lose the second and the array goes offline, and every file on it becomes logically invisible. At that moment each surviving disk holds a unique fragment of the parity, and one wrong write destroys it permanently.
What happened here
On the afternoon of August 26, 2026, an injection mold shop in Shiqi, Panyu called. The server was beeping and the screen was full of red text. On the phone we told them to touch nothing and read us the model: a Dell PowerEdge R730 with a PERC H730 controller and four 2TB Western Digital RE4 enterprise drives (WD2000FYYZ) in RAID5. It held SolidWorks drawings, CNC programs and nearly three years of scanned orders, about 3.2TB in total.
iDRAC showed Slot 1 and Slot 3 offline, one after the other, with the virtual disk in Failed. The worse news came next. The shop's part-time IT person had clicked Force Online in the PERC interface the night before and pulled Slot 3 back into the array. The system "recovered". It ran for a morning. In the afternoon Slot 1 dropped again, and then the whole array became unrecognizable, with the boot message "No Foreign Configuration or PDs are not responding."
What Force Online did: Slot 3 was not dead. It was failing slowly and had been kicked out on a response timeout. Forcing it back let the array write using its data for a while, which updated the parity for one round. That disk's contents were now offset in time from the other two healthy disks.
Recovery: four steps, three days
Step 1: image every disk, write-protected (day 1). All four drives came out and were labeled. We mirrored them sector by sector with PC-3000 Data Extractor. Slot 3 produced a run of bad sectors around the 68% mark, and the mirror speed fell from 120MB/s to 2MB/s. We finished with reverse reading and skip reading to pull every readable sector, ending at about 97.4% integrity. That step took 14 hours, and it cannot be hurried, because hurrying loses sectors.
Step 2: establish disk order and stripe parameters (morning of day 2). The hard part of RAID5 recovery is not the algorithm but the parameters: disk order, stripe size (128KB here) and parity rotation (left-symmetric). We ran entropy analysis on the images and used the position of the NTFS MFT to work the order back. Force Online had broken the obvious mapping, so we tried four permutations. The third, 2-0-3-1, produced an intact MFT.
Step 3: virtual reassembly and parity repair (afternoon of day 2). The array was reassembled virtually at the image layer, with nothing written back to any original disk. For the 1,200-plus damaged blocks in the Slot 3 image, we rebuilt the missing data from the parity on the other three. This is where RAID5 earns its keep: any one disk's data can be computed from the rest.
Step 4: export and verify (day 3). The directory tree came first. The SolidWorks drawing folders were complete, and so were the scanned orders. Two 2019 CNC program folders had garbled file names from damaged MFT records, so we scanned by file-header signatures and recovered about 80% of them, broken down by file type. The export went to two new 4TB enterprise drives and took 11 hours. On site we opened 30 SolidWorks assemblies at random: 28 opened normally and two reported file damage in the bad-sector area. The CNC programs were checked for G-code integrity in Notepad.
The bill
| Item | Price | Notes |
|---|---|---|
| RAID5 logical rebuild, 4 disks / 3.2TB | RMB 3,800 | Imaging, parameter analysis, reassembly and export |
| Bad-sector imaging surcharge | RMB 600 | Slot 3 had a bad-sector rate near 2.6%, requiring reverse and skip reading |
| Target storage, 2 x 4TB enterprise drives | RMB 1,560 | Western Digital Ultrastar. You can supply your own instead |
| Total | RMB 5,960 | No recovery fee if the recovery fails |
The plant manager's own words: "If I had known it would cost five thousand, I would not have let Xiao Zhang press that button." Force Online did not add to the bill, but it pushed the job from a standard process of a day and a half to three days of trial and error on the parameters. With worse luck, a full round of parity would have been overwritten and could not have been computed back at all.
A checklist for the next RAID5 failure
- First disk down. Back up the critical data immediately, while the array is still online, then replace the disk and rebuild as soon as you can. A degraded array is the most dangerous window, because a second failure ends it. Before you swap, match the new drive's model, capacity and firmware as closely as possible.
- Second disk down. Stop. Write down the slot number of every disk and photograph them. Do not pull or reseat anything, do not force anything online, do not initialize, do not attempt a rebuild.
- Read the lights. On a Dell carrier, solid green is healthy, blinking green is rebuilding, solid amber is failed, and blinking amber is a predicted failure: the drive still works but is dying. When two are amber, the second is often not truly dead but timed out, and that is exactly the situation where Force Online does the most damage.
- Say the right things when you call. Controller model (PERC H730 or H330, P440 and so on), RAID level, drive count, capacity and model, the order in which disks dropped, and everything done since. That list determines the recovery plan.
Three things to tell anyone still running RAID5
First, RAID5 is not a backup. It is an availability scheme that keeps you running when one disk dies, and there is still only one copy of the data. Second, on an array with more than four disks, or with disks larger than 4TB, a rebuild is measured in tens of hours and the chance of losing another disk during it is not small, so consider RAID6 or RAID10. Third, and most practical: for drawings, programs and orders you cannot lose, add a disk for scheduled cold backups, or pay a few hundred RMB a year for cloud backup. Either is far cheaper than spending six thousand to recover.
If an array fails, power it down before you call 020-39029800. Someone answers around the clock. Read us the details from the list above and we can tell you how good the odds are.