Hostile Private Networks in the Cloud
A while ago I deployed a blog using DigitalOcean's infrastructure. I used separate droplets for the database and web tier, and used the "private IP" feature to only allow MySQL traffic over that interface. I noticed the private network seemed larger than necessary for just hosting my own stuff, so I did a little digging.
DigitalOcean's "private networking" is quite literally the opposite of private. Your droplet's private IP will be in a shared subnet with every other customer's if they end up on the same network as you. Do not think for a minute you can "trust the LAN" or anything like that; I'm sure if I scanned deeper I could find dozens of unauthenticated Redis and database servers people inadvertently have opened to this not-so-private network.
Unfortunately, this isn't just DigitalOcean's problem. I ran a couple tests from one of my instances on Linode and was shocked to find their Private IP feature both assigns the address to the same interface as the public IP, and shares the subnet with many other customers. These are just the two providers I have used; I would hazard to guess this is a common practice among the smaller non-Openstack cloud providers.
To demonstrate this issue, I spun up a droplet in DigitalOcean's TOR-1
region. The private interface has the entire
network routed, and I decided to try doing a basic scan to enumerate the
Starting Nmap 6.40 ( http://nmap.org ) at 2018-03-03 17:06 UTC Nmap scan report for 10.137.0.4 Host is up (0.0037s latency). MAC Address: F6:BF:AB:FF:ED:AA (Unknown) Nmap scan report for 10.137.0.5 Host is up (0.0037s latency). MAC Address: AE:FC:E8:AD:78:7F (Unknown) [...] Nmap scan report for 10.137.255.252 Host is up (0.0082s latency). MAC Address: 1A:73:75:CA:44:1A (Unknown) Nmap done: 65536 IP addresses (5588 hosts up) scanned in 229.23 seconds
I have exactly one other droplet in that region, and yet my brand new droplet can access over five thousand other customers' servers on a supposedly "private" network. This is not only misleading, but the messaging makes it sound like you can trust it; I don't even want to try and knock on ports to see how many open-to-the-world services are running.
I won't pretend to be all-knowing and condescending: nothing about running infrastructure is easy. However, having separate networks per customer would make dramatic improvements to the service's scalability and security, and at the very least having one VXLAN per customer seems like a painfully obvious good practice.
To verify this, I spun up a droplet in two completely separate DigitalOcean accounts (I added my personal CC to my work account to test this).
Right off-the-bat I was able to confirm that both droplets had private IP
addresses in the same subnet range. Then, on one droplet I started
-l 64321, and on the second I tried to
nmap the private
IP of my second droplet. I was successful.
Starting Nmap 6.40 ( http://nmap.org ) at 2018-03-03 21:57 UTC Nmap scan report for 10.132.146.7 Host is up (0.0056s latency). PORT STATE SERVICE 64321/tcp open unknown MAC Address: 6A:1E:6E:F2:92:59 (Unknown) Nmap done: 1 IP address (1 host up) scanned in 0.19 seconds
And here's an obligatory screenshot. The left-side droplet is running under my personal account, the right-side droplet is under a completely different account (but the droplets are in the same region).
Luckily my further attempts to
tcpdump the traffic between the
two hosts were unsuccessful. But one could still do a fair amount of damage
if someone entrusted a service like Redis or NFS to allow connections from
the entire network of the private interface.
I take great issue with these features being labelled "Private Networking" as there is nothing private about them whatsoever. There's nothing stopping me from spinning up a new server in the same region as you and suddenly I am on the same "private" network as you.
By calling this feature "private", it implies privacy. This may mislead people into setting up unsecured services because they assume the network they are exposing them on is only for their consumption. As long as "hostile LAN" practices are followed, having a public private network should be fine, but I would strongly urge these cloud providers to implement at the very least customer isolation, if not full OpenStack-style SDN.
I hope this post serves as a warning to anyone running services on the "underdog" cloud providers: your private network is absolutely not private, and you may have an unforseen attack vector left wide open.