RAID5 dropped two disks, do not force online_The real recovery process of a mold factory in Panyu

First, let's start with the conclusion: RAID5 drops two disks, you only have two correct actions

The first move isshutdownThe second action isFind someone to recover.. All the operations in between-Force Online, replug, initialize, rebuild-each of which may turn salvageable data into salvageable data.

This is not a scare. RAID5's fault tolerance is only one disk. When the first disk is dropped, the array enters a degraded state and can still read and write; when the second disk is dropped, the array is offline and all data is logically "invisible". At this time, each remaining disk in the array carries a unique piece of verification information. One wrong step is permanent destruction.

Second, the situation this time: the design data of the mold factory, lost two plates

On the afternoon of August 26, 2026, an injection mold factory in Panyu Shiqi called and said that the server "beeps and the screen is full of red characters". The phone asked them to stay still and report the model: Dell PowerEdge R730, PERC H730array card, 4 pieces of 2TB Western Digital RE4 enterprise disk (WD2000 FYZ) to do RAID5, loaded with SolidWorks drawings, CNC machining programs and nearly three years of order scanning parts, with a total data volume of about 3.2TB.

The status seen in iDRAC is: Slot 1 and Slot 3 were offline one after another, and the status of the virtual disk (VD) was Failed. The bad news came later-the IT part-time worker in the factory had clicked "Force Online" in the PERC management interface the night before, and forcibly pulled Slot 3 back to the array. At that time, the system was "recovered." After using it for one morning, Slot 1 fell off again in the afternoon, and then the entire array was completely unrecognizable, and the boot prompt was "No Foreign Configuration or PDs are not responding."

Consequences of forced online: Slot 3 disk is actually a slow failure (response timeout is kicked out, not completely dead). After forcibly pulling back array partially writes its data, parity is updated for one round. In other words, the data on this disk is staggered from the data on the other two good disks on the timeline.

III. Recovery process: four steps, three days

Step 1: Write the protected image completely (Day 1).All four disks were unlabeled and mirrored sector-by-sector with PC-3000 Data Extractor. Slot 3 saw a lot of bad sectors (read timeout) at about 68% of the disk, and the mirroring speed dropped from 120MB/s to 2MB/s. Finally, reverse reading + skip reading was used to extract all readable sectors, and the image integrity was about 97.4%. This step took 14 hours, and it couldn't be faster. If it was faster, the sectors would be lost.

Step 2: Determine tray order and strip parameters (morning of day 2).RAID5 recovery is not the most difficult algorithm, but the parameters: disk order (which block is disk 0), stripe size (this one is 128KB), check rotation direction (left symmetry). Use analysis software to do entropy analysis on the image file, combined with the file system (NTFS) MFT position to reverse the disk order.

Step 3: Virtual recombination and verification repair (afternoon of day 2).RAID5 is virtually reassembled at the mirror layer, and no original disk is written back. For more than 1200 damaged blocks in Slot 3 mirror, the check relationship of the remaining three disks is used to make up for them. The advantage of RAID5 is here, the data of any disk can be calculated by using other several blocks.

Step 4: Export Verification (Day 3).First scan the directory tree: SolidWorks drawings catalog is complete, order scan is complete, there are two old CNC program catalog in 2019 file name random code (MFT record damage), manual scanning according to file header characteristics, finally recovered by file type about 80%. Data exported to the customer's newly purchased 2 4TB enterprise disk for mirror backup, export took 11 hours. Open SolidWorks on site and randomly check 30 assemblies, 28 normally open, 2 report "file damage"(corresponding to the area where bad sectors are concentrated), CNC program uses Notepad to check G code integrity.

IV. Charges and Cost Account

The factory manager's original words were "If I had known earlier, I would not have let Xiao Zhang click that button." Forced online didn't cost much, but it pushed the recovery difficulty from "standard process one and a half days" to "parameter trial and error three days." If I was unlucky, the calibration block would not be returned after the entire round of rewriting, which meant that it was really gone.

V. RAID5 Self-inspection list after losing the plate

1. Drop the first plate:Backup critical data immediately (while array is still online), and then replace the disk as soon as possible to rebuild. During the reconstruction period, array performance degrades, and if another piece is lost, it will be completely destroyed. This time is the most dangerous window. Before changing the disk, confirm that the new disk model, capacity, and firmware version are as consistent as possible.
2. Drop the second plate:Stop. Write down the slot number of each disk and take photos. Do not unplug, do not force, do not initialize, do not attempt to rebuild.
3. Plate of light:Dell server tray indicator, solid green = OK, blinking green = rebuilding, solid amber = failure, blinking amber = predictive failure (PFA, disk still working but dying). Two amber pieces, second one is often not really dead, response timeout kicked-this is the most dangerous situation to force online.
4. Make it clear when repairing:Controller model (PERC H730/H330, P440, etc.), RAID level, number of disks, capacity model, order of disk loss, and what was done during that time. This information directly determines the recovery plan.

VI. Three sentences for enterprises still using RAID5

First, RAID5 is not a backup, it is only an availability scheme of "no downtime for a bad disk", and there is only one copy of data. Second, RAID5 with more than 4 disks or a single disk exceeding 4TB has a reconstruction time of 10 hours, and the probability of disk loss during reconstruction is not low, considering RAID6 or RAID10. Third, it is also the most practical: Drawings, programs, orders, and other lost data, plus a disk for regular cold backup, or a few hundred RMB a year for cloud backup, than spending six thousand RMB to restore afterwards is much cheaper.

If there is a problem with array, stop the machine first and then call: 020-39029800, someone will answer it 24 hours a day. Report the information in item 4 above on the phone, and we can help you judge how confident you are.