PCI option ROM space errors in UEFI servers - IBM BladeCenter, Flex System and System x



Source

RETAIN tip: H21709

Symptom

The IBM BladeCenter Advanced Management Module (AMM) Event log and the Integrated Management Module (IMM) chassis log on rack mount servers show the following error:

  FW/BIOS, firmware progress (Firmware Error) hang

See the following picture for the IMM System Event Log (SEL) log entry for more details:

Current - IMM system error log entry

Note: With Unified Extensible Firmware Interface (UEFI) code release 1.73, or higher for IBM EX5 systems and 2012 products, the message improved to state following:

  Error: Error: FW/BIOS, connector (No Op ROM Space) PCI express Slot X fault

Also, for more details, see the following picture for the IMM SEL log entry:

Improved - IMM system error log entry

The slot number will be identified for the Peripheral Component Interconnect (PCI) device in which the Read Only Memory (ROM) could not run.

Note: The IMM code released in Third Quarter 2010 turns on both the FAULT Light Emitting Diode (LED) and the BRD LED on some systems (IBM System x3850 X5 and System x3690 X5).

Turning on the BRD LED is not an expected behavior, and this behavior has been corrected in IMM version 1.28, or later. The unexpected LED behavior does not exist on systems using Integrated Management Module II (IMM2).

Affected configurations

The system can be any of the following IBM servers:

This tip is not software specific.

This tip is not option specific.

The system has the symptom described above.

Workaround

Note that steps 1 through 3 should solve most option ROM issues, only use number 4 and later for as-needed basis.

Best practices for resolution on UEFI machines:

  1. Firmware updates may reduce ROM size requirements, so this may be a solution. Ensure all system and adapter firmware is at latest levels.
  2. Disable the Preboot eXecution Environment (PXE) ROM of on-board Network Interface Controllers (NICs).

    The simplest solution to some PCI Option ROM space errors is to reduce the base system ROM requirements to the minimum necessary. Typically, this can be achieved by disabling the PXE (Network start) ROM capability of the on-board Ethernet.

    Note: This does not disable the device in the Operating System, it only disables its capability to perform a network start.

    1. Select F1 setup System Settings -> Network -> PXE Configuration.
    2. Select the Media Access Control (MAC) address of the on-board NIC-1.
    3. Change the Enable PXE / PXE Mode setting to Disabled.
    4. Select Save Changes.
    5. Select the MAC address of the on-board NIC-2.
    6. Change the Enable PXE / PXE Mode setting to Disabled.
    7. Select Save Changes.
  3. Disable all Legacy Option ROMs for all devices that are not starting.
    1. Start the system and press F1 to enter the UEFI menu.
    2. Select System Settings -> Devices and I/O Ports -> Enable/Disable legacy ROM execution.

    If the machine is PXE or Storage Area Network (SAN) starting, do not disable the Legacy ROMs for adapters that actually are starting a Legacy (non-Extensible Firmware Interface (EFI)) OS on the machine.

  4. Change the ROM order.
    1. Select F1 setup.
    2. Select System Settings -> Devices and I/O -> Set option ROM execution order.

    Ensure that the on-board LSI storage controller is first in the list before the on-board Ethernet devices.

  5. Move Fibre Channel and Fibre Channel over Ethernet (FCoE) cards to a lower slot number as they tend to require more ROM space and should initialize first to reduce the chance of the issue occurrin
  6. Peripheral Component Interconnect Express (PCIe) adapters with PCI firmware spec 2.1 should be in lower slot numbers. PCIe adapter with PCI firmware spec 3.x does not need as much Option ROM space due to design changes.
  7. If the issue still is occurring and there are any Intel NICs in the machine, use the following steps:
    1. Download the Intel v15.7 firmware and driver CD (or higher/newer version): MIGR-5087548.

      See also RETAIN tip H202974 (MIGR-5087756) "UEFI SUPPORT FOR BOOT ROM FOR Intel BASED ETHERNET ADAPTERS".

    2. Create a CD from the ISO image.
    3. Find the DOS StartUtil utility in this folder:
      \APPS\StartUtil\DOS\StartUtil.exe
    4. Find the UEFI StartUtil utility in this folder:
      \APPS\StartUtil\EFI64\BOOTUTIL64.EFI
    5. Use the StartUtil with the following parameters:

      For DOS systems:

      1. Copy the StartUtil.exe onto a startable DOS floppy.
      2. Start the floppy to DOS and run this command:

        c:\> StartUtil.exe -FLASHDISABLE -ALL

      For EFI systems (if offered by the server):
      1. Copy the BOOTUTIL64.EFI utility to a Universal Serial Bus (USB) Key.
      2. Start a Full EFI Shell environment, and then run the utility from the USB key with this command:

        > BOOTUTIL64.EFI -FLASHDISABLE -ALL

      Note: Not all servers offer an EFI shell option (for example, IBM System x3690 X5, x3850 X5)

      For IBM BladeCenter HX5, type 7873, only:

      The IBM BladeCenter HX5, type 7873 has a new setup option called Non-system board PXE under Legacy Support menu ('EnableNonPlanarPxe' in Advanced Settings Utility (ASU)).

      If users do not need to use PXE start on NICs other than system board/on-board NICs, then this setting could be set to Disable to free up more space for option ROM and reduce the chances of this error. IBM BladeCenter HX5 7872 UEFI release 1.60, or later releases support this setup option.

Additional information

This issue was discovered originally on Legacy (non-UEFI) machines and extensive documentation was provided to explain how to resolve the symptoms.

For more details, see the following .pdf file "IBM Support Info - Resolving 1801, 1802 Errors", which is referenced in RETAIN Tip H194252 (MIGR-5078445):

These errors are due to a permanent restriction of legacy PCI ROM space architecture.

Applicable countries and regions

 


Document id:  MIGR-5084743
Last modified:  2014-03-26
Copyright © 2014 IBM Corporation