Researchers recently discovered for the first time that a rootkit virus (also known as iLOBleed) is launching an attack on HP enterprise servers, capable of infecting facilities remotely and erasing data. The attack was discovered by Amnpardaz, an Iranian cybersecurity company, that iLOBleed was the first malware in history that targeted iLO firmware.
Experts explained that malware targeting iLO is very difficult to prevent and control because it runs with high privileges (higher than any access level in the operating system) and can be undetected by administrators and detection software. By tampering with this module, the malware is allowed to continue to exist after the operating system is reinstalled.
In this article, we will analyze the rootkit’s attack process and how it is hidden in iLO, and cannot be deleted through firmware upgrades. It is hidden and continues to attack. The malware has been used in the wild for some time, and we have been monitoring its performance.
Since analyzing this malware requires a clear understanding of the HP iLO firmware architecture, we will first introduce the HP iLO architecture. Then, the discovered malware and its various modules will be analyzed. Finally, we will discuss strategies and solutions for protecting iLO.
HP iLO architecture
The iLO management panel of the HP server is a haven for malware, which cannot be detected or erased by conventional methods after infection.
Not only can iLO be accessed and infected through the iLO network port but also iLO can be accessed and infected through the system administrator or root access to the main operating system. This means that if an attacker has access to a user with administrator/root privileges on the main operating system installed on the server, it can communicate directly with iLO (without any further authentication) and infect it if it is vulnerable.
Years of research have revealed multiple vulnerabilities in HP iLO that have led manufacturers to make changes to patches and architecture.
In iLO4 and its earlier versions for servers of G9 and below, there is no secure boot mechanism with an embedded trusted root key in the hardware. Therefore, these versions of firmware are more susceptible to modification and infection by malware.
Even if iLO has been updated to the latest version without any known vulnerabilities, it can still be downgraded to a lower version, which makes it possible to infect the fully patched firmware. If non-default settings are enabled, you can only prevent this in the G10 series. On older servers, the downgrading mechanism cannot be prevented.
Because of the above, simple solutions such as completely disconnecting the iLO network cable or upgrading the firmware to the latest version are not enough to prevent malware infection.
Since 2020, the malware analysis team of Amnpardaz Software has discovered a rootkit that adds a malicious module named Implant.ARM.iLOBleed.a to the iLO firmware and modifies multiple original firmware modules. The rootkit will silently block the firmware update while completing falsely. It also provides access to the server hardware; one of the results is to completely wipe the server disk.
Tools for verifying the integrity of HP iLO firmware will be released to the public soon.
Indicators of Attack (IOC)
Although it is customary to provide the hash value as an IOC, we believe that this is not effective for this kind of malware. Mainly because if you don’t have an iLO dump tool at hand, you won’t be able to read the firmware and check its hash value. Moreover, the real iLO firmware set is very small, so the whitelisting method is possible and more suitable. That is, compare the hash value of the firmware with a list of known good hash values.
However, if you are worried about whether your server is infected with this malware, you can use the following simple methods:
As mentioned earlier, to maintain persistence and prevent erasure, malware will silently block the firmware upgrade process. The malware strives to simulate the upgrade process, and it is difficult to display a false “upgrade” version in the iLO Web UI and elsewhere, but there is a problem: HP has made considerable changes to the iLO UI. Therefore, you can easily detect the presence of malware.
In the image below, you can see two screenshots of the iLO-4 firmware, both claiming to be version 2.55. But as you can see, one of them uses the old version 2.30 UI, which uses a completely different theme.
Comparison of fake (infected) and real iLO login pages
Of course, like other IOCs, we hope that attackers can find techniques to bypass this detection method. But at the same time, you can use it to easily “discover” malware.
HP iLO technology
HP provides system administrators with iLO technology as a means of managing servers. This technology allows system administrators to use special network interfaces to remotely access various functions of their servers, including:
Open and close the server;
Configure various hardware and firmware settings;
Remote access to the system console;
Remote installation of CD/DVD images;
Multiple hardware and software indicators of the remote monitoring and control system;
Over the years, HP has released various versions of iLO firmware for its servers of different generations. Table 1 shows these versions.
ILO firmware versions of HP servers in different periods
Due to the key authority and performance of this firmware interface in all server families, various scenarios for attacking this interface can be imagined. These attacks include obtaining the password of the management interface, exploiting security vulnerabilities, and flashing the infected firmware on the server instead of the main firmware.
In recent years, researchers have conducted a series of studies to identify security vulnerabilities in the HP iLO firmware management interface. These studies eventually led to the discovery of many vulnerabilities with severe to high and moderate risks. Unfortunately, in recent years, the widespread release of these vulnerabilities and proof-of-concept code snippets has allowed individuals and hacker organizations to use these vulnerabilities to attack the network infrastructure of organizations that use HP servers in their corporate networks.
iLO firmware architecture
From a hardware point of view, iLO is integrated with the system’s motherboard, including the following:
ARM processor with GLP/Sabine architecture;
Flash memory for firmware storage;
Dedicated network interface;
A set of hardware ports used to communicate with other control units;
iLO firmware solution
The figure above shows a schematic diagram of the iLO hardware. It can be seen from the figure that the ARM processor is connected to the South Bridge through the PCI-Express interface and connected to the main processor of the server through it. iLO can also communicate directly with CMOS. This type of connection is used to set variables, such as the boot sequence that the iLO management interface will provide to the user.
The processors used in the iLO hardware are from the 7th and 8th generation ARM processors. These processors provide good processing power while being considered very low consumption. This helps to provide a management interface for network administrators when the server is in standby (power off) mode without consuming too much power and current.
The iLO processor can communicate with different memory chips. Two of these chips will be discussed in this section. The first is the main system chip that stores the iLO firmware, which is used to load the firmware during the iLO boot sequence.
The second is on-chip memory called iLO NAND flash memory, which can be used by iLO firmware as external system storage. After loading, the iLO firmware uses this storage to save files such as system event logs and history records. It is also the storage space for applications run by the iLO operating system.
Connection to the main server
As a server management and control unit, iLO technology can directly access all server hardware components, such as memory, processors, input and output ports, and hard drives. In addition, the main processor of the server detects iLO as a PCI module and can communicate with it.
iLO firmware structure
The iLO firmware is stored as a binary file in an SPI flash drive (usually 16 MB in size). As shown in the figure below, the firmware consists of 3 main parts, including the boot loader, operating system kernel, and user mode module. Of these 3 parts, only the Boot Loader part is not encrypted, and the other two parts are compressed (using the LZMA algorithm) and contain signatures. All iLO firmware executable content is C code compiled with ARM architecture.
Internal structure of iLO firmware
From a software perspective, iLO provides server administrators with multiple services such as Web Server and SSH Server. In fact, iLO is a complete operating system. Once the system is plugged into the grid, it will start and provide its services even if the hosting server is shut down.
When iLO starts, each part will check the integrity of the next part before running any of its parts. Therefore, the Boot Loader part will be responsible for verifying the signature, extracting and compressing and loading the kernel part. The kernel part will also be responsible for verifying signatures, extracting compression, and loading user mode modules.
The operating system used in the iLO firmware is a real-time operating system called Integrity developed by Green Hills Software, which is responsible for executing tasks in the user area. The user area is actually an ELF binary file of ARM architecture developed and packaged by HP. This file has various modules, and each module has a specific task. Each task is a process with a dedicated virtual memory space and a set of threads running in the user area. In the following sections of this document, some of the most important iLO modules will be introduced.
The figure below shows multiple UserLand modules in the iLO 4 firmware. Some of the most important modules will be introduced in the following sections.
Multiple UserLand modules in iLO 4 firmware
Web server module: iLO Web management interface
The Web Server module is responsible for providing the iLO management interface in the form of Web services. This module contains parts such as Web interface, XML programming interface, Redfish programming interface and remote console.
The connection to the Web server module can be HTTP or HTTPS. The module has four processing threads, each of which is responsible for managing and responding to one of the connections established with the module. Each connection request is processed line by line, its content is analyzed, and if the authentication and access level are correct, a corresponding response is provided. Although access to almost all web pages requires user authentication, some data is provided in XML format and does not require an authentication process.
CHIF module: connection with the host operating system
The CHIF module is a module in iLO that communicates with the CPU and server memory components and transmits messages between iLO and the host operating system. In simple terms, the tasks of this module can be listed as follows:
Waiting to receive messages from the main operating system of the server;
Send the received message to the message processing unit (Command Handler) according to the message type;
Redirect specific messages to relevant modules for processing;
By default, the messages and commands sent to the module do not undergo any authentication process;
FUM module: firmware update
The main task of the FUM module is to update the iLO firmware. This task can be accomplished in three ways:
Through the HP Intelligent Provisioning management interface;
Use the host operating system installed on the server and access the PCI-E interface;
Remotely through the iLO Web interface management interface;
The FUM module performs the firmware update operation in 5 steps:
Receive new firmware files through the host server or Web server module;
The new firmware file is sent to the FUM module;
The FUM module checks and verifies the digital signature of the new firmware file;
The FUM module also requires the kernel to verify the integrity of the new firmware file;
Finally, the FUM module sends the new firmware to the SPI module to program it on the SPI flash memory.
ILO firmware update process for FUM module
In this process, the integrity of the firmware is verified by checking the digital signature of the firmware, so no modules will be added or changed on the basis of the original firmware provided by HP. The important point is that under normal circumstances, you can downgrade to a lower version of the firmware.
SPI module: access to flash memory
The main function of this module is to communicate with the SPI flash memory containing iLO firmware and server BIOS firmware. This module provides a low-level interface for reading, erasing and writing the firmware on the relevant flash memory chip. As mentioned earlier, some iLO firmware update operations are performed through this module.
ConAppCli module: console (command line) service
This module is responsible for providing command-line user interface services for receiving commands from server administrators. These commands can be user management, power management, viewing system events, etc.
SSH module: remote command line
In addition to the web interface, the iLO firmware also uses the SSH protocol to provide users with encrypted shell services. Through this module, the user can communicate with the firmware through port 22 and execute a set of commands.
Health module: monitoring system components
The task of this module is to periodically check the system status and record server events. System status includes operating temperature, fan speed, power status, system memory status, network status, processor status, etc.
Black box module: black box system
This module acts as a server “black box”. Many sensitive and important system information and events recorded by the Health module every day are compressed and stored by the module.
In addition to the modules mentioned in the previous section, the iLO firmware has various other modules, each of which has a specific task in the UserLand section. Modules such as SNMP, SNTP, and SVCSiLO are responsible for system and network management tasks, while modules such as USB, GPIO, and I2C provide iLO control access to server hardware components.