In our previous post on Remote Server Management, we dive into the topic of managing servers without the administrator needing to be physically standing near the server and without local interfaces. Generally speaking, this is what IT practitioners mean by ‘remote’. The underlying assumption within our previous remote management post is that all of the contact established between the administrator and the servers being administered is done within a trusted IT environment. This presumes that the network being used to transport the remote management data is (relatively) trusted. Typically this means that there would exist Firewalls protecting between the internal networks and the outside internet, along with standard network visibility and segmentation within the network itself.
But many scenarios require a server to be remotely managed in a hostile environment. This could take the form of the server being outside of your physical control, being in a separate jurisdiction, or some combination of these two. if the nature of your business is particularly interesting from a control and/or monitoring perspective, the risks could be compounded since any attempts to penetrate your organization would likely be made where vulnerability is perceived to be the highest. Sometimes servers will need to be remotely managed in an environment that is not known or trusted, which can make it a part of your IT environment where attackers may focus their efforts. There are generally fewer instances of remote server management in hostile locations for this exact reason – it only happens if it absolutely must.
“Hostile”, unfortunately, is a vague term, so let’s define it more generally as ‘untrusted’, i.e. not entirely within your organization’s scope of control. How do you know if any of your systems are in a hostile environment?
If they are not on your premises,
if they are operating in a shared space,
if they are in another jurisdiction, (RELATED – Watch our nation-state cyber-security conversation with Greg Kesner HERE)
if there exists particular incentive for your business activity to be monitored,
if they are manufactured in a shared facility,
if they are transported and/or installed via third parties;
we will consider the systems to be in a hostile environment.
For example, let’s say your organization has its headquarters and datacenters in Atlanta, Georgia, and needs to start building out geographical redundancy elsewhere, and has tasked you with building out capacity in Europe. These systems will meet many of the points outlined above, and qualify as hostile deployments. What should you be concerned about? Our previous post on remote server management is a good start; below we will re-emphasize some points and introduce some new points to address this type of deployment.
Challenges to Securing Servers in Hostile Locations
As a recap of the previous post, the standard approaches to protecting remote servers include locally assembling and hardening the server platform, encrypting sensitive data stored locally, securing the local network environment with firewalls, using a VPN to protect communications to and from the site, and potentially implementing a method for secure secure lights-out remote management such as hardened IPMI.
These approaches are effective, however some require re-emphasis and some new points arise in a hostile environment scenario. Below, we will go over the most prominent challenges faced in this situation:
1: Unauthorized Physical Access
Servers and other hardware equipment that is set up in an uncontrolled environment can be subject to physical tampering and access while in use. This tampering can be made easier if USB or other periphery access is available.
2: Unsanctioned Hardware Seizure (Theft!)
If your business activities are known to be high-profile or particularly prone to attempts to compromise, there is a high chance that attempts will be made to compromise your equipment while it is in transit or while it is operating at a distant location. This could take the form of true theft by deception or force, or seizure by a governmental intelligence agency. This risk can be minimized through physical security and compliance with local law, however it can never be entirely eliminated. Potential consequences can be minimized by ensuring that what is on your equipment is very difficult to extract value from, through encryption, compartmentalization, and other hardening methods.
3: In-Transit Tampering
This issue arises most often when equipment is sent over national borders into countries which have tight controls over the technology they allow in and out of their country. This can take the form of changing installed software (BIOS, etc) or installation of passive tap of some kind which would relay information back to some centralized hub for processing and monitoring.
4: Deployment Risk
In a classic remote deployment scenario, software and configurations would be loaded on a server before it is shipped to its final production location. This opens your organization up to the possibility that components, with your intellectual property, could be either seized or compromised in transit. A more security-conscious approach would be to ship the components via a trusted channel, and remotely integrate and setup the equipment in the location where the server will operate.
5: Lights-Out Management Risk (server lifecycle tampering)
As we discussed in our previous remote server management blog post, IPMI can be useful for remote server management if used with extreme caution. In hostile locations it is all the more critical to be careful since the physical and network security are even less under your supervision and control. In these scenarios, IPMI should be avoided and disabled wherever possible, and extra care should be spent hardening this if it is to be used.
6: High Cost and Complexity of monitoring and orchestration VM activity
In order to monitor what each VM in any given environment is doing, many complex and different products and methods need to be configured and maintained in harmony. This is expensive in terms of time and money and the very complexity can add many more areas where human error could decrease your level of security. In hostile locations, each additional layer of complexity is another attack surface which could be exploited by a malicious actor.
7: Remote Attestation Risk
Our previous post outlines how attestation is a way to detect the integrity of hardware and software in a server system by measuring unique fingerprints of each component therein. There is a protocol to support this process, however, if the protocol is compromised, the validity of the system is impossible to verify. In essence, a compromised server can ‘lie’ to you and appear to be uncompromised, when in fact the compromise was just a layer below this detection gate.
8: Audit Provenance Risk
The role played by audit is much more critical in hostile environments since the overall sanctity of information being relayed is potentially lower due to the environmental uncertainty of your systems’ surroundings. Additional steps should be implemented to compartmentalize and harden the audit function to ensure the visibility it provides is sound. Precautions, such as a hardware root of trust or additional encryption, must be taken to ensure that the system providing the audit is trusted.
This post should give you a better idea of the primary vulnerabilities that may be exploited if you attempt to set up servers in a hostile, or untrusted, environment. Many of these points have solutions that can be implemented to eliminate, or reduce, these vulnerabilities, so proceed with caution! At Skyport we have spent considerable time mapping out these issues and many of the core features of our product have been purpose built to eliminate and reduce them in a turn-key manner. If this type of server administration risk is present for you, or is of particular interest to you, we’d love to hear if we have missed anything and get a conversation started.