At 1:40 a.m. on August 27, a cold-chain logistics company in Panyu called our 24-hour duty line. The only server in its machine room, a Dell PowerEdge R740, had shut down without warning. The TMS dispatch system was down, temperature readings from more than a dozen refrigerated trucks had stopped coming back, and everything had to be online before 6 a.m. or a truckload of seafood would be a write-off.
Our engineer walked in at 2:20 a.m. The machine was a 2021 R740: dual Xeon Silver 4210R, 96 GB of memory, two 750 W platinum power supplies (model D750E-S1). The power button did nothing and the front-panel diagnostic LED was dark. This was not a hang. The server had no power at all.
Step 1: Read the iDRAC log before you reach for a screwdriver
The iDRAC9 in the R740 has its own power feed, so it stays reachable whenever the chassis is plugged in, even with the host off. Our engineer ran a laptop straight into the dedicated iDRAC network port, logged in as root, and went to System, then Overview, then Logs. The result was almost funny:
| Time | Log entry | Severity |
|---|---|---|
| Jul 11, 03:22 | PSU 2 input lost | Critical |
| Jul 11, 03:22 | PSU redundancy lost | Warning |
| Aug 26, 23:47 | PSU 1 failure detected | Critical |
| Aug 27, 01:38 | System power lost | Critical |
In plain English: 47 days ago, PSU 2 broke. The server ran on PSU 1 alone for close to seven weeks, with that single supply carrying the full load around the clock, until it finally gave out at the end of August and took the system down with it. Had anyone replaced the supply when the July 11 alarm fired, this outage would never have happened.
Step 2: Verify on site what the two supplies are actually doing
What iDRAC reports is not always the whole story, so we confirmed it physically:
- Power indicator LEDs. We pulled both supply modules: LED 1 and LED 2 were both amber, though green is normal. Amber means a fault or no input.
- Socket swap test. We plugged supply 2 into the socket that supply 1 had been running on, a known-good circuit, and it still would not light. That rules out the PDU outlet and confirms the supply itself is dead.
- Multimeter on supply 1's output. On the bench, under load, the 12 V main rail jumped between 9.8 V and 13.5 V. instead of sitting stable at 12 V plus or minus 5%. The fan bearing also had visible play. Classic long-term full-load aging, and a precursor to bulging capacitors.
Conclusion: supply 2 had been dead for 47 days, and supply 1 was severely aged and ready to quit at any moment. The fact that the box had booted at all was luck.
Step 3: Replacing a supply is not the whole job
A new original 750 W platinum supply (D750E-S1) runs RMB 1,300–1,800 each on the open market; refurbished units are RMB 600–800. Because this machine carries the whole company's dispatch operation, we recommended a matched pair of new original parts and quoted RMB 3,200, including on-site labor, installation and a full-load burn-in test.
Two things trip people up during the swap:
- Match the firmware. If the new supply's firmware differs from the old one, iDRAC throws a "PSU mismatched firmware" error. We ran the power firmware update from inside iDRAC and had them aligned in 10 minutes.
- Run the redundant pull test afterwards. With the server running, pull supply 1 and confirm the machine does not lose power and iDRAC raises a redundancy-lost alarm. Plug it back in, wait for the status to return to green, then pull supply 2 and test the other direction. Plenty of shops skip this step, which is why their redundancy is still fake the next time something real happens.
At 4:50 a.m. the TMS system was back online, one hour and ten minutes ahead of the customer's deadline. We reviewed the iDRAC logs remotely the next day and they were clean.
The three problems this outage exposed, all of them common
Problem 1: nobody was watching the alarms. iDRAC can send SMTP email alerts and SNMP traps. This customer had never configured either. We set up email alerts for them at no charge, going to both the IT manager and the owner. If your servers are Dell, HPE or Lenovo, this takes about 10 minutes and it is worth doing today.
Problem 2: redundant supplies are never tested. The point of a second supply is that one can fail and the server keeps running, but that only holds if the other supply is actually good. Run a live pull test every quarter, or at minimum open iDRAC twice a year and check that input voltage and fan speed look normal on both units.
Problem 3: a single power feed. Both supplies in this R740 were plugged into the same mains PDU, so a utility outage would still have taken the server down. The standard practice is to split the two supplies across A and B feeds, and at minimum to put an online UPS in front of the server. A 3 kVA online UPS with batteries costs RMB 4,000–6,000, which is considerably cheaper than a truckload of seafood.
Appendix: rack server power self-check list
| Check | Method | Frequency |
|---|---|---|
| Power LED status | Visual: green = normal, amber = fault or no input | Monthly |
| iDRAC / iLO / XCC logs | Log in to the management port and search System Logs for PSU entries | Monthly |
| Redundant pull test | With the server running, pull each supply in turn and confirm no power loss | Quarterly |
| Alert channel | Confirm email and SNMP alerts actually arrive; pull a supply on purpose to test | Quarterly |
| Fan and intake cleaning | Compressed air, and replace the supply fan at the first unusual noise | Twice a year |
If your servers are in the "installed and forgotten" state, ask us for a free health inspection. We come on site anywhere in Guangzhou and hand you a full report covering power supplies, drive SMART data, RAID status and firmware versions. No charge, and the report is yours to keep.