Firmware Issue Causes Unexpected Reboots and Loud Noisy Fans on SteelFusion Edge (SFED) Appliances (2100, 2200, 3100, 3200 and 5100)
Solution Number: S27535
Riverbed have recently discovered a P1 Hardware defect in SteelFusion Edge Hardware Platform 1U and 2U: models 2100, 2200, 3100, 3200, and 5100. and they recommend customers with these hardware models review Knowledge base article S27535 (below) for solution.
Riverbed thanks you for your patience and apologises for any inconvenience this may have caused.
The Firmware upgrade process described in this KB may take up to 90 minutes. Please plan accordingly.
Riverbed rigorously tested this Firmware upgrade process, however failure of this firmware upgrade
will make the device unusable requiring RMA.
Customers are advised:
- where these devices are deployed in HA upgrade the "Standby Box " first followed by "primary box".
- where the device is standalone off-line all remote LUNs, backup Edge Local LUNs to ensure a recovery path.
A firmware issue can cause unexpected restarts of SteelFusion Edge appliances (models 2100, 2200, 3100, 3200 and 5100) and cause the fans on the Edge to run at full RPM.
You might see the following symptoms:
- Excessively loud fan noise on SteelFusion Edge (SFED) models.
- Device is unresponsive with no web or CLI access. Power cycling is the only way to recover.
- The BMC software that monitors the health of the system triggers the appliance restarts due to a race condition that occurred when handling the keep-alive messages from the CPU.
- The following messages appear in the system logs:
Hypervisor hardware management controller has issue: Get Auth Capabilities error Error issuing Get Channel Authentication Capabilies request Error: Unable to establish IPMI v2 /RMCP+ session
kernel:IPMI message handler: BMC returned incorrect response, expected netfn 7 cmd 35, got netfn 7 cmd 24
Above issues are tracked through Bug 241350. Please follow the instructions below to upgrade the BMC (Baseboard management controller):
Note: During Firmware upgrade device should not be power cycled or rebooted.
Any interruption to the upgrade process may put the device in unknown state requiring RMA.
1. Download image 4.1.1-fwup1 from support.riverbed.com
2. If the Device is deployed in non-HA environment, please off-line all remote LUNs and backup the edge local LUNs.
3. Upgrade the device using downloaded image.
4. After successful update, revert back to the previous image of RiOS(i.e the image of the RiOS the box is running before BMC update)
Q: How can I determine my SFED unit requires this Firmware update?
A: If the new device(not RMA device) is delivered after 11/23/2015, it can be safely assumed that the device is already running latest Firmware.
All other devices's Firmware version must be checked by Riverbed TAC and upgraded if required.
Q: I don't see any of the mentioned symptoms, so how can I verify BMC version is outdated?
A: Commands to verify the BMC version require device Shell access available to Riverbed TAC personal only, so please contact TAC.
Q: Can we push firmware update form CMC/SCC as any other RiOS upgrade?
A: Firmware update requires the device to be reverted back to the previous version of RiOS. As, this step doesn’t fit into standard RiOS upgrade workflow, Riverbed suggests to upgrade each SFED device independently outside the SCC RiOS update process.
Q: Why am I advised to off-line all remote LUNs and backup the Edge local LUNs?
A: This is a precaution to make sure all data on the SFED box is safely committed to core and edge local LUN can be restored.
Q: Does the BMC update survive power cycle and future RiOS upgrades?
Q: I am not comfortable in upgrading the BMC using the special build. Can I wait for future GA version of RiOS that would include the BMC upgrade?
A: At this point only way to upgrade the BMC FW is through the above special build. Currently there are no plans to include this in future RiOS releases.
Q: I am not comfortable upgrading BMC or I can't allow up to 90 minutes of downtime. What are my options?
A: Please discuss the options with Riverbed TAC.
Q: In future if we RMA the box do we have to upgrade BMC again?
A: All future devices shipped from factory will have the latest BMC, so this is one time update in field.
Q: How do I know that system firmware upgrade has completed successfully?
A: When the user logs in through the cli, the banner will display the success or failure message. (Also can be seen from webgui, announcement page)
Firmware Version Check: Upgrade Completed.
Firmware Version Check: Please switch back to the original software image.
Firmware Version Check: Upgrade Failed. Please reboot the system to continue with the upgrade (automatic).
Environment: SFDC Boxes