Print | Rate this content

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable with iLO Firmware Earlier Than 2.42 Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: c04996097

Version: 6

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors
NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

Release Date: 2016-12-07

Last Updated: 2017-08-22


DESCRIPTION

Document Version
Release Date
Details
6
08/22/2017
Updated the Resolution with specific steps to be completed.
5
05/24/2017
Updated the Resolution with additional details on running the script.
4
12/05/2016
Updated the Resolution with additional details on running the script.
3
08/08/2016y
Updated the Resolution with details on running the script.
2
07/21/2016
Updated the Description with additional errors that are observed and updated Resolution to include Step 2 through Step 4.
1
03/31/2016
Original Document Release.

On an HPE Gen8-series or HPE ProLiant Gen9-series servers with HPE Integrated Lights-Out 4 (iLO 4), the NAND flash device may not initialize or mount properly, which may cause a variety of symptoms, which are listed below:

AHS Errors

  • AHS Logs display a blank date when performing the following:
    "Select a range of the Active Health log in days From: _______ To:_________"
    AHS file system mount may fail with (I/O Error) or (No Such Device).
  • The HP Active Health System (AHS) Logs are unable to be downloaded. AHS Data is not available due to a filesystem error.
  • The iLO Diagnostic tab will display the following error message: "Embedded media manager failed initialization" or "The AHS file system mount failed with (No such device) or "The AHS file system mount failed with (I/O error)" or "Controller firmware revision 2.09.00 Could not partition embedded media device."
  • Unable to download AHS log and "bb_dl_disabled" is displayed.

Embedded Media Errors

  • Controller Firmware Version 2.09.00 may fail to restart.
  • Unable to partition Embedded media device.
  • Embedded media manager may fail initialization.

Intelligent Provisioning Errors

  • Intelligent Provisioning will not execute when selecting F10.

OneView Errors

  • Remote Insight/Integrated Lights-Out Self-Test Error 8192.
  • Unable to register this HP OneView instance with iLO: There was a problem with posting a command to the iLO.
  • Unable to register this HP OneView instance with iLO: The iLO initialization was unable to complete.
  • Unable to determine if this server hardware is being managed by another management system. Received an error from iLO <ip address of iLO> with Error: Blob Store is not yet initialized. and Status: 126

SCOPE

Any HPE ProLiant Gen8-series or HPE ProLiant Gen9-series server running iLO 4.

RESOLUTION

The resolution for this issue may take several steps that need to be completed in the order specified below.

Note: The process below assumes that the iLO 4 firmware upgrade and NAND format are performed with the server online.

OVERVIEW

Step 1) Upgrade the iLO 4 firmware to version 2.54
Step 2) Perform a NAND format Step
Step 3) Check the iLO status If the iLO status is normal, then skip to Step 6 If the iLO status is still degraded, continue to Step 4
Step 4) Schedule downtime; AC power-cycle and repeat the NAND format
Step 5) Check the iLO status If the iLO status is normal, continue to
Step 6 If the iLO status is still degraded, then skip to Step 7
Step 6) Perform these final steps if the system board does not need to be replaced: Reboot the server; reinstall IP; and refresh the server in OneView (if server is managed by OneView)
Step 7) If steps 1-4 did not resolve the degraded iLO, replace the system board.

Note: The iLO 4 firmware upgrade and NAND format can be done online without any production interruption. The reason to attempt the NAND format online is that the success of the format will then drive the rest of the action plan. This will allow for an action plan that will minimize downtime. The Intelligent Provisioning restore can be done online on ProLiant Gen8 and ProLiant Gen9 servers running Linux and ProLiant Gen9 servers running Windows. VMware will require that Intelligent Provisioning be installed offline.

Detailed steps

Step 1. Upgrade the iLO 4 firmware to version 2.54 (or later). To download the firmware. The latest version of the iLO 4 firmware is available as follows:

Note: For ESXi servers booting from the Embedded SD Card: It is strongly recommended to reboot the server immediately after updating the iLO 4 firmware or resetting the iLO.

  1. Click the following link:

    http://h20566.www2.hpe.com/portal/site/hpsc

  2. Enter a product name (e.g., "iLO 4") in the text field under Enter a Product Name or Number.
  3. Click Go.
  4. Select the appropriate product model from the Results list (if prompted).
  5. Click the "drivers, software & firmware" hyperlink under the Download Options tab.
  6. Select the system's specific operating system from the Operating Systems dropdown menu.
  7. Click the category Firmware - Lights-Out Management.
  8. Select the latest release of the HPE Integrated Lights-Out 4 Firmware version 2.54 (or later).
  9. Click Download.

Step 2. Perform NAND format (iLO 4 firmware version 2.44 or later is required to format NAND).

A. Execute the Force_Format RIBCL XML script (Force_Format.xml) to format the NAND device. The Force_Format RIBCL script is as follows:

<!-- RIBCL Sample Script for HP Lights-Out Products -->
<!--Copyright (c) 2016 Hewlett-Packard Enterprise Development Company,L.P. -->

<!-- Description: This is a sample XML script to force format ll -->
<!-- the iLO partitions. -->
<!-- iLO resets automatically for this operation to take effect -->

<!-- Warning: This command erases all data on the partition(s) -->
<!-- External providers will need to be re-configured if -->
<!-- partition is formatted -->

<!-- Input: VALUE tag: all - format all available partitions -->

<!-- NOTE:You will need to replace the USER_LOGIN and PASSWORD values -->
<!-- with values that are appropriate for your environment -->

<!-- See "HP Integrated Lights-Out Management Processor Scripting -->
<!-- and Command Line Resource Guide" for more information on -->
<!-- scripting and the syntax of the RIBCL XML -->

<!-- Firmware support information for this script: -->
<!-- iLO 4 - Version 2.42 or later. -->
<!-- iLO 3 - None. -->
<!-- iLO 2 - None. -->

<RIBCL VERSION="2.0">
<LOGIN USER_LOGIN="Administrator" PASSWORD="">
<RIB_INFO MODE="write">
<FORCE_FORMAT VALUE="all" />
</RIB_INFO>
</LOGIN>
</RIBCL>

The RIBCL script can be run in a variety of ways. The HPE iLO 4 Scripting and Command Line Guide provides details on the various methods to run the RIBCL script. The guide is located at the following URL:

h20565.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=5219994&docId=emr_na-c03334058&docLocale=en_US

For example, the script can be run using the HP Lights-Out Configuration Utility, available at the following URL:

http://h20566.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_c2f8737292c74b3b852b063ff7

To format the NAND flash, open a command prompt, change to the folder where the HPQLOCFG tool is installed and issue the command below, modifying the values between brackets as needed.

C:\Program Files (x86)\Hewlett-Packard\HP Lights-Out Configuration Utility>hpqlocfg -s -l c:\hpqcfg.log -f c:\Force_Format.xml -v -t user=Administrator,password=

When the operation is successful, the iLO-4 session (GUI or CLI) will be displayed.

The hpqcfg.log file will contain the following entry:

Forcing a format of the partition after the iLO reset

Note: For HPE server blades, the script can also be run from the Onboard Administrator CLI.

B. After the Force Format RIBCL script is run, the script response should be: RESPONSE STATUS="0x0000" MESSAGE='Forcing a format of the partition after the iLO reset.'

Note: After running the Force_Format RIBCL XML script, wait 2-3 minutes for the iLO to reboot and the NAND partitions to be recreated.


Step 3. Check the iLO status using these criteria:

A. Check the iLO event log

If the NAND Force Format was successful, there will be an entry in the iLO Event Log displaying the following message:

Embedded Flash/SD-CARD: One or more storage devices have been formatted

B. Check the NAND status on the diagnostic page in the iLO GUI:


What to do next:

If the iLO status is normal based on the above criteria, then skip to Step 6

If the iLO status is still degraded, continue to Step 4


Step 4. If the iLO status is still degraded, perform the following steps:.


a) Schedule a maintenance window
b) Shut down the server
c) Perform an E-fuse (server blade) or AC Power Pull (DL / ML series servers)
d) Perform the NAND format again (refer to the instructions in Step 2 above)

Step 5. Check the iLO status (refer to the instructions in Step 3 above)


If the iLO status is normal, continue to Step 6
If the iLO status is still degraded, then skip to Step 7

Step 6. Perform these final steps if system board does not need to be replaced:

a) For iLO 4 firmware 2.53 (or later), reboot the server
b) For iLO 4 firmware 2.50 (or earlier), log into the iLO 4 CLI and type: "oemhp_clearRESTAPIstate"; then reboot the server
c) Check iLO status (iLO GUI > Information > Diagnostics)
d) Reinstall Intelligent Provisioning (see additional details below)
e) If the server is managed by HPE OneView, perform a server refresh to bring the server back under management.

Note: Any existing errors in OneView will need to be marked as cleared after the refresh.

Step 7. If steps 1-4 did not resolve the degraded iLO, contact HPE support to arrange a system board replacement. Follow these steps to complete the remediation:

a) If not already initiated in Step 5, open an HPE support case to arrange for a replacement system board / arrange a maintenance window
b) Shutdown the server
c) Unassign the OneView profile (if the server is under OneView management) d) Replace the system board
e) Enter Server Model and Serial Number via RBSU
f) Update to iLO 4 firmware 2.54 (or later)
g) Check iLO status (refer to Step 3 above)
h) Reassign the OneView profile (if the server is under OneView management)

Note: Any existing errors in OneView will need to be marked as cleared after the OneView profile is applied.

If additional assistance is needed, contact HPE support and reference Advisory a00019495en_us as follows:

  • In North America, contact HPE Customer Support at 1-844-806-3425.

OR



OneView Considerations

OneView relies on the NAND to be accessible to perform many operations such as adding a new server or applying a profile. Because of this there are several things that need to be understood when dealing with this issue in an OneView environment.

  1. OneView interaction with the NAND – OneView uses a portion of the NAND called the iLO blob store. If the blob store is not accessible, adding a server to OneView or assigning a profile to the server cannot be completed.
  2. Impact of an inaccessible NAND when managed by OneView – If the NAND becomes inaccessible after the server was added in OneView, there are several things that can cause an outage:
    1. E-fuse – If an E-fuse reset is performed, the server will not be able to be brought back under management until the NAND issue is remediated and this will cause an unexpected outage.
    2. Server is removed and reinserted – This would essentially be the same as performing an E-fuse, so the same information in the E-fuse section applies here.
    3. Un-assign a server profile – If a server profile is unassigned, the server profile will not be able to be reassigned until the NAND issue is remediated.
  3. Migration from Virtual Connect to OneView – If an enclosure is migrated from Virtual Connect and a server has an inaccessible NAND, that server will not be able to be added properly until the NAND issue is remediated. If an in-service migration is attempted on a server with an inaccessible NAND, an unexpected outage will occur. Reference advisory

    c05384185

    for more detail.
  4. Formatting of the NAND – The NAND format wipes the iLO blob store that is used by OneView. It is important to issue a server refresh after a successful NAND format to recover the blob that OneView uses.

Reinstalling Intelligent Provisioning Considerations

There are online and offline options for reinstalling Intelligent Provisioning

  • Intelligent Provisioning can be reinstalled online if the server is running Windows (HPE ProLiant Gen9-series servers only) or Linux (HPE ProLiant Gen8-series servers/HPE ProLiant Gen9-series servers).
  • Intelligent Provisioning must be reinstalled offline if the server is running Windows (HPE ProLiant Gen8-series servers only) or VMware (HPE ProLiant Gen8-series servers/HPE ProLiant Gen9-series servers).
  • The Intelligent Provisioning software download links are provided below.

Intelligent Provisioning versions

  • HPE ProLiant Gen8-series servers are only supported with Intelligent Provisioning 1.x
  • HPE ProLiant Gen9-series servers are only supported with Intelligent Provisioning 2.x

Note: Intelligent Provisioning version 3.x is for HPE ProLiant Gen10-series servers only.

Offline method considerations:

Run the Intelligent Provisioning Restore Media to restore the Intelligent Provisioning data. Instructions to restore Intelligent Provisioning are as follows:

Instructions to create a bootable DVD with the IP image are provided on the HPE Intelligent Provisioning recovery media download site under the Installation Instructions tab.

Reference:
http://h20566.www2.hpe.com/hpsc/swd/public/detail?sp4ts.oid=1008862660&swItemId=MTX_77b94511492441a08eb272ed27&swEnvOid=4166#tab3

Note: The DVD can be used multiple times.

Please note the Intelligent Provisioning Recovery Media DVD may be remotely mounted using HPE Integrated Lights-Out 4 (iLO 4) Virtual Media functionality, in order to reinstall Intelligent Provisioning.

Additional information is available in the HPE iLO 4 User Guide at the following URL regarding how to mount an ISO image (federated or un-federated) and perform basic virtual media operations. Reference Pages 189 and 223-237:

HPE iLO 4 User Guide:
http://h20566.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c03334051-16

In addition, it is possible to write a script that utilizes HPE Integrated Lights-Out 4 (iLO 4) to reinstall Intelligent Provisioning on multiple servers. HPE Lights-Out management processors support an advanced scripting interface for group configuration and server actions. Scripts would need to be customized for the specific environment and task. Sample scripts are available for customers to reference at the following URL:

HPE Lights-Out XML Scripting Sample for Windows: https://www.hp.com/swpublishing/MTX-459b8adc29c04317ad1d6a6752

Intelligent Provisioning software download links

HPE ProLiant Gen8-series servers are only supported with Intelligent Provisioning 1.x HPE ProLiant Gen9-series servers are only supported with Intelligent Provisioning 2.x

a.) Linux RPM located at the following URL: https://downloads.linux.hpe.com/SDR/repo/ip

For HPE Gen8-series servers: hp-firmware-intelligentprovisioning-ip-1.64-1.1.x86_64.rpm
For HPE Gen9-series servers: hp-firmware-intelligentprovisioning-ip-2.61-1.1.x86_64.rpm

b.) Windows (HPE ProLiant Gen9-series servers only) located at the following URL: https://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_b031497eafd94b9ea2156cdef6&swEnvOid=4231

For HPE ProLiant Gen9-series servers: Intelligent Provisioning for Windows x64 2.50.0.0 (6 Jan 2017) cp031091.exe

c) ISO images (e.g. for VMware) on Hewlett Packard Enterprise Support Center located at the following: http://h20566.www2.hpe.com/hpsc/swd/public/readIndex?sp4ts.oid=1008862660&lang=en&cc=us

For HPE ProLiant Gen8-series servers: Intelligent Provisioning Recovery Media 1.64(B) (8 Feb 2017) HPIP164B.2017_0201.3.iso (886 MB)
For HPE ProLiant Gen9-series servers: Intelligent Provisioning Recovery Media 2.61 (19 May 2017) HPIP261.2017_0518.11.iso (1.0 GB)

RECEIVE PROACTIVE UPDATES : Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively via e-mail through HPE Subscriber's Choice. Sign up for Subscriber's Choice at the following URL: Proactive Updates Subscription Form.

NAVIGATION TIP : For hints on navigating HPE.com to locate the latest drivers, patches, and other support software downloads for ProLiant servers and Options, refer to the Navigation Tips document .

SEARCH TIP : For hints on locating similar documents on HPE.com, refer to the Search Tips document .

To search for additional advisories related to iLO, use the following search string:

+Advisory +ProLiant -"Software and Drivers" +iLO


Hardware Platforms Affected: HPE ProLiant ML30 Gen9 Server, HPE ProLiant DL20 Gen9 Server, HPE ProLiant SL230s Gen8 Server, HP ProLiant SL250s Gen8 Server, HP ProLiant SL270s Gen8 Server, HPE ProLiant BL460c Gen8 Server Blade, HPE ProLiant DL360p Gen8 Server, HPE ProLiant DL380p Gen8 Server, HP ProLiant DL380p Gen8 Server, HPE ProLiant ML350p Gen8 Server, HPE ProLiant BL465c Gen8 Server Blade, HP ProLiant DL160 Gen8 Server, HPE ProLiant BL420c Gen8 Server Blade, HPE ProLiant DL360e Gen8 Server, HPE ProLiant DL385p Gen8 Server, HPE ProLiant ML350e Gen8 Server, HPE ProLiant DL380e Gen8 Server, HPE ProLiant BL660c Gen8 Server Blade, HPE WS460c Gen8 Graphics Expansion Blade, HPE ProLiant DL320e Gen8 v2 Server, HPE ProLiant ML310e Gen8 v2 Server, HPE ProLiant SL210t Gen8 Server, HPE ProLiant ML350e Gen8 v2 Server, HPE ProLiant DL580 Gen8 Server, HPE ConvergedSystem 700x (CS700x), HP ProLiant XL220a Gen8 v2 Server, HPE ProLiant XL730f Gen9 Server, HPE ProLiant DL160 Gen9 Server, HPE ProLiant DL180 Gen9 Server, HPE ProLiant DL360 Gen9 Server, HPE ProLiant BL460c Gen9 Server Blade, HPE ProLiant DL380 Gen9 Server, HPE ProLiant XL230a Gen9 Server, HP ConvergedSystem 700x v1.1 VMware Kit, HP ConvergedSystem 700x v1.1 Microsoft Kit, HPE ProLiant XL740f Gen9 Server, HPE ProLiant XL750f Gen9 Server, HPE ProLiant ML150 Gen9 Server, HPE ProLiant DL60 Gen9 Server, HPE ProLiant DL80 Gen9 Server, HPE ProLiant SL4540 Gen8 1 Node Server, HPE ConvergedSystem 700 (CS700), HPE ConvergedSystem 700, HP ProLiant ML110 Gen9 Server, HPE ProLiant XL170r Gen9 Server, HPE ProLiant XL190r Gen9 Server, HPE ProLiant WS460c Gen9 Graphics Server Blade, HP ProLiant DL580 Gen9 Server, HPE ProLiant BL660c Gen9 Server Blade, HPE ProLiant DL560 Gen9 Server, HPE ProLiant XL450 Gen9 Server
Operating Systems Affected: Not Applicable
Software Affected: Not Applicable
Support Communication Cross Reference ID: IA04996097
©Copyright 2017 Hewlett Packard Enterprise Company, L.P.
Hewlett Packard Enterprise Company shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided "as is" without warranty of any kind. To the extent permitted by law, neither HPE nor its affiliates, subcontractors or suppliers will be liable for incidental, special or consequential damages including downtime cost; lost profits; damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett Packard Enterprise Company and the names of Hewlett Packard Enterprise Company products referenced herein are trademarks of Hewlett Packard Enterprise Company in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.

Provide feedback

Please rate the information on this page to help us improve our content. Thank you!