I was having a fun discussion with a co-worker this week about how to secure network infrastructure. The challenge being that if someone were to gain access to a switch, router, firewall, or other important transit device they could pretty easily execute a man-in-the-middle attack, or wreck complete havoc on an organization. We saw a great example of this with the City of San Francisco back when Gavin Newsom was the mayor and Terry Childs, a former Cisco SE, decided to show, with great fanfare and aplomb, what power he wielded.
In 2008 he changed the passwords on the network equipment and then deleted the configuration stored on non-volatile flash. The password recovery procedure requires a reboot of the device – thus the recovery of the password deletes the run-time configuration effectively rendering the device inoperative. We can assume that the City of San Francisco did not have a configuration repository separate from what the network admins had access to, or any keystroke logging via a jump server (a few simple things that would have prevented this situation).
Seven years later, today – the password recovery procedure on the switches and routers has not changed. Any networking admin or person with reasonable access and skills can hijack and destroy a network. Vendors have done little to nothing to improve the security and operational manageability of the systems we all depend on. A case in point: Ars Technica reported this week that “Attackers are hijacking critical networking gear from Cisco.” I read through this article a few times and it basically occurred to me to say, “duh… of course they are – vendors have been incredibly lazy regarding security.”
Cisco’s official statement is that “No product vulnerability is leveraged in this attack, and the attacker requires valid administrative credentials or physical access to the system to be successful. The ability to install an upgraded ROMMON image on IOS devices is a standard, documented feature that administrators use to manage their networks. No CVE ID will be assigned.”
Let me get this straight – because the vendor is not signing images or protecting the ROMMON in any way its not a vulnerability? Any system vulnerability that gets to root/enable and can then install permanently invisible malware is analogous to a BIOS-level root kit. But because this is a standard documented “feature” this is, according to the vendor, not a problem. So what can you do, and what should we reasonably expect IT vendors to do?
I assume if you are reading this you are a network or security admin who is understandably a bit scared about the exposure you have to insider malicious threats against your infrastructure. I’d do a few things…
- I would lock down all management access to the network infrastructure removing any/all in-band management access.
- Connect all OOB management/consoles to a dedicated infrastructure that is physically separate.
- Integrate the two with a jump server/bastion host that is the only device to exist on both networks. All communications from the user networks must go through the jump server to get to the management network.
- Geo/IP-fence the jump server – only allow access into it via SSH/RDP/VNC from a very limited number of workstations. Preferably have those workstations in a place that has physically controlled access.
- Force two-factor authentication for all inbound connections into the jump server – this increases the validity that the admin is who they say they are.
- Flag all administrative sessions consuming more than 1Mb/s or other arbitrary bandwidth that would be indicative of a data export and not an administrative session.
- Enable VT100/Terminal recording on the jump server for all sessions. The administrators of the jump servers should have no access to the switches/routers. The administrators of the switches/routers should have no access to the jump server.
- Use synthetic passwords within the jump server connecting to the network devices – this way any theft of the password would be useless as the password known by the network admin for access would be only applicable for that user on that jump server.
- Set up your RADIUS/TACACS user permissions such that the network admins do not have the ability to change the passwords. Additionally have a separate repository of all configurations that is, again, decoupled from the network admins.
I can go on, but with these 9 steps you have eliminated many of the most common insider-threat attack vectors and provided yourself with an environment where if someone does something malicious or just erroneous you have the forensics and data available to know who did what, when, and have the ability to recover.
As far as vendors go the idea that providers of switches and routers are not providing signature signed images and firmware is pretty unconscionable given the threats we are all facing and the types of attacks that have been successfully executed against IT systems. All images from vendors should be signature signed. Coupled then with signed images vendors should provide some method of verifying the integrity of the systems they provide – TPM modules are a method of this provided the PCRs are meaningful and the attestation system has some record of what each device should be measuring.
Noodle through this, read up on the vulnerability that a vendor says is not a vulnerability. Think about how amazingly evolved some of the security threats we see are nowadays. Then ask yourself what could happen to your network and thus, your business, if a third party could load their own code into the routers and switches and firewalls on your network and you couldn’t detect it. Makes for a fun weekend…