“Securing your IPMI: just because it’s lights-out, doesn’t mean you shouldn’t leave your porch lights on”
There’s always a fine line between security and accessibility. In the case of infrastructure management, downtime is the enemy, and remote management access is a powerful tool to maintaining your compute resources. Consoles and “crash carts” are a must for deploying physical servers, but managing systems at scale means implementing effective out-of-band management networks. Driver changes, BIOS & firmware updates, low-level sensor monitoring, control media access – these are all things that need to be administered at scale. This is known as Out-of-Band (OOB) or Lights-Out Management (LOM), and means that no humans are local and the actions are happening below the OS management layer. For you, it’s the only way to manage your data center without constantly engaging “smart hands” or truck rolls — but your convenience is a potential attacker’s beachhead into your infrastructure. It’s hard enough to lock down the IPMI network in your data center, but remote location pose an even larger threat.
A prominent tool for this is the Intelligent Platform Management Interface (IPMI), which is a set of computer interface tools for management and monitoring capabilities from a remote location. It provides access to controls underneath the host system’s CPU, firmware (BIOS or UEFI) and operating system. It is a well intentioned way to have remote server management be easier to scale, however, vendor implementations vary widely and it brings a range of security issues that often exist without being acknowledged. This makes it a prime security hole that can be exploited.
Many security experts have been very vocal about IPMI and the dangers of using it, including Dan Farmer, who was the initial ‘whistleblower’ on the topic. He has since posted stats and other pieces, and security writers have joined in speaking out against IPMI due to security concerns. The oft-used method of compromising something within an enterprise network and using it as a jumping point to other assets within a network is aided greatly by a loosely-protected remote management system, so awareness is critical to stamping out some of the most widespread vulnerabilities.
If you’re worried about attackers laterally spreading through your network, IPMI is a dangerously effective way to cross many types of security boundaries, compromise the underlying infrastructure and exfiltrate data in an area that’s typically not well audited and monitored, making it a perfect storm of vulnerability if left insecure.
So if IPMI is potentially insecure, but also a necessity for remote management, what do you do? Like any threat surface, out-of-band management must be deployed for operational reasons, but also must be just as carefully managed as any other production resource. Like any other threat surface area in your environment, there are steps to control and audit access, and limit the ability to use it as a vector for laterally attacking other systems. While important in data center installations, hostile environments require more vigilant control. Below are a few simple recommendations to better secure your IPMI infrastructure:
Separate the management network from the data network
While it is always a good design principle to separate management and normal data access to systems, it is especially important for remotely managed servers given the vulnerabilities in IPMI. The ports with IPMI should wire into separate networks that have their own sets of VPN connections that authenticate the administrators, firewalls, and event logging and auditing links.
Scan your non-management network regularly to ensure IPMI isn’t exposed
Since some vendor implementations of IPMI wind up turning it on by default across all interfaces on the server, it is wise to regularly scan your environment to ensure nothing is exposed where it shouldn’t be. If your network is already adhering to our step 1 above, than this step is really just a compliance test. Regardless, it should be done regularly since in large organizations things can change in ever more locations and the likelihood of error increases along with the potential for harm from any error. Nmap is a free tool that can help.
Use SSH when possible
Whenever possible, Secure Shell (SSH) should be used as an access medium for remote server locations. Since SSH allows a secure encrypted channel to be formed over an untrusted network, it can be an optimal method to establish a remote connection to a server. This method is more secure than having reusable passwords as certain access credentials since the passwords could be compromised and still be useable.
Have a method to audit the remote management sessions
Having an effective method to audit management sessions is critical to decipher what happened in the event of any problems. A proper audit system should include a capture of who accessed what, during what time period, and where the management session was initiated from. Much of this information will come from the tools controlling access to the management network (the VPN & firewall) and SSH events. Logs should be stored in secure vaults that are tough to reach and tamper. Without this type of system in place, you will be flying blind.
Make sure Cypher-Zero is disabled
There have been some widespread security vulnerabilities discovered in the IPMI protocol that pertain to the use of cypher type zero, which is used as an indicator that the client wants to use clear-text authentication, and actually allows access with any password. This means that often there are large vulnerabilities that are totally unbeknownst to the IT organization due to this flaw. It can be fixed by disabling cypher-zero upon setup. More detail can be obtained from the following post.
These five points should cover the basics of remote management security. It’s important to note, however, that the security vulnerabilities exposed by remote management are even more complicated if the servers are deployed in hostile environments. If the servers are running in untrusted physical environments, network environments, or countries, then there is even more work to be done for remote management to be effective.
If you’re interested in some of the deeper issues of remote management and what we have done to address them, take a look at our white paper on the topic.