One of the greatest, yet seemingly unknown, dangers that face any cloud-based application is the deadly combination of an SSRF vulnerability and the AWS Metadata endpoint. As this write up from Brian Krebbs explains, the breach at Capital One was caused by an SSRF vulnerability that was able to reach the AWS Metadata endpoint and extract the temporary security credentials associated with the EC2 instance's IAM Role. These credentials enabled the attacker to access other Capital One assets in the cloud and the result was that over 100 million credit card applications were compromised.
The purpose of this blog post is to explain the technical details of such a vulnerability and give some helpful suggestions for avoiding a similar situation in any organization.
In order to fully understand the impact of this cloud one-two punch it is necessary to break down the attack chain into its various components: SSRF and the AWS Metadata Endpoint. First, Server Side Request Forgery (SSRF) is a vulnerability that allows an attacker to control the destination address of an HTTP request sent from the vulnerable server. While this is not always the case (see Blind SSRF), the attacker can often see the response from the request as well. This allows the attacker to use the vulnerable server as a proxy for HTTP requests which can result in the exposure sensitive subnets and services.
Consider the following PHP code:
<?php echo file_get_contents("http://".$_GET['hostname']."/configureIntegration.php"); ?>
The code above sends an HTTP request to the hostname specified by the attacker in the "hostname" GET parameter. Logic like this is commonly found in the "Integrations" section of applications. This code is vulnerable to SSRF. Consider the following scenario: There is a sensitive service running on the loopback interface of the vulnerable server. This is emulated by the following configuration:
The PHP code above is hosted on the web server that faces the internet. When an attacker discovers this endpoint, he/she might use the following to grab the data from the internal application:
Which would result in a hit on the internal HTTP server:
┬─[[email protected]:/t/secret]─[02:29:48 PM] ╰─>$ python3 -m http.server 8081 --bind 127.0.0.1 Serving HTTP on 127.0.0.1 port 8081 (http://127.0.0.1:8081/) ... 127.0.0.1 - - [15/Aug/2019 14:30:56] "GET /secret.html?/configureIntegration.php HTTP/1.0" 200 -
and return the following to the attacker:
This is only available on a loopback interface
Now that the danger of SSRF is clear, let's look at how this vulnerability may be exploited in the context of the cloud (AWS in particular).
Due to the dynamic nature of the cloud, it became necessary that server instances (EC2 for example) have some way to get some basic information about their configuration for the purpose of orienting themselves to the environment in which they were spun up. Out of this need the AWS Metadata endpoint was born. This endpoint (169.254.169.254), when hit from any EC2 instance, will reveal information about the configuration of the particular EC2 instance. There is quite a lot of information available via this endpoint including: hostname, external ip address, metrics, lan information, security groups, and last but not least, the IAM (Identity and Access Management) credentials associated with this EC2 instance. It is possible to retrieve these security credentials by hitting the following url where [ROLE] is the IAM role name:
The response contains several things: the AccessKeyId, SecretAccessKey, and the Token for this account. Using these credentials, an attacker can login to AWS and compromise the server and potentially many other assets. In the case of the Capital One breach, these credentials were used to access an S3 bucket which contained millions of records of user information.
In summary, the poor implementation of the metadata service in AWS allows for an attacker to easily escalate an SSRF vulnerability to control many different cloud assets. Other cloud providers like Google Cloud and Microsoft Azure also provide access to a metadata service endpoint but requests to these endpoints require a special header. This prevents most SSRF vulnerabilities from accessing the sensitive data there.
In order to prevent this type of vulnerability from occurring firewall rules will need to be put in place to block off the metadata endpoint. This can be done using the following iptables rule:
sudo iptables -A OUTPUT -d 169.254.169.254 -j DROP
This will prevent any access to this ip address. However, if access to the metadata endpoint is required, it is also possible to exclude certain users from this rule. For example, the iptables rule below would allow only the root user to access the metadata endpoint:
sudo iptables -A OUTPUT -m owner ! --uid-owner root -d 169.254.169.254 -j DROP
These blocks MUST be done at the network level - not the application level. There are too many ways to access this IP address. For example, all of these addresses below refer to the metadata service:
http://[::ffff:169.254.169.254] http://[0:0:0:0:0:ffff:169.254.169.254] http://425.510.425.510/ Dotted decimal with overflow http://2852039166/ Dotless decimal http://7147006462/ Dotless decimal with overflow http://0xA9.0xFE.0xA9.0xFE/ Dotted hexadecimal http://0xA9FEA9FE/ Dotless hexadecimal http://0x41414141A9FEA9FE/ Dotless hexadecimal with overflow http://0251.0376.0251.0376/ Dotted octal http://0251.00376.000251.0000376/ Dotted octal with padding http://169.254.169.254.xip.io/ DNS Name http://A.18.104.22.168.1time.169.254.169.254.1time.repeat.rebind.network/ DNS Rebinding (22.214.171.124 -> AWS Metadata)
And there are many more. The only reliable way to address this issue is through a network level block of this IP.
The easiest way to check the IAM roles associated with each EC2 instance is to navigate to the EC2 Dashboard in AWS and add the column "IAM Instance Profile Name" by clicking the gear in the top right hand corner. Once the IAM role for each EC2 instance is easily visible, it is possible to check these roles to see if they are overly permissive for the what is required of that EC2 instance.
It is also imperative to understand the pivoting potential of these IAM Roles. If it is possible that an SSRF, XXE, or RCE vulnerability was exploited on any cloud system, the logs for the IAM Role associated with this instance must be thoroughly audited for malicious intent.
Coming from someone who can officially say that information security has given me a few gray hairs, I'm writing this article from the perspective of someone who's been around the block. With over 15 years in information security, I feel like I've seen it all. And while I can't claim to be a great penetration tester myself, I can say that I work with (and have worked with) some truly talented pentesters. I can also feel confident stating that I've read more pentest reports than most.
So, having this background… I get asked by businesses and defenders all the time, "What advice would you give?" and, "What lessons can be learned?"
Well, thanks for asking…. (insert deep breath here)
In fact, we've known that passwords are a weak form of authentication since the moment the first password-based authentication system was created. Passwords can be weak for several compounding reasons. Whether it be due to their limited length and complexity (keyspace) or the fact that they can be shared, guessed, written down, or reused, let's face it, they provide almost no security. Until we stop using passwords or ensure that every last account has a strong and unique password that can't be guessed or cracked, we accept significant risk.
(MFA) is not enabled or required for all remote access. While it is almost common place now to find MFA on VPNs, we still find roles, groups, and even URLs allowing MFA to be bypassed. Further, other types of remote access like Citrix and Remote Desktop, Outlook Web Access, and SSH are more overlooked. Remember that when passwords are weak (and they probably are), attackers will be quick to take advantage when MFA is not enforced.
Your mom said it, and now I will too. In SynerComm's reporting, we consider both #1 and #2 to be high-severity findings in our pentest reports. When combined, these result in a critical weakness. Password spraying allows an attacker to easily guess common passwords (think Summer19) and gain immediate access to email and internal networks.
Don't get me wrong, get your EternalBlue and Heartbleed patched, but don't think just because you're well patched that you are secure. Vulnerability scanning is important, but at its best, it discovers live systems, missing patches, default credentials, weak services, and other well-known vulnerabilities. What it doesn't tell you is that your systems may already include a roadmap to access anything and everything on your network.
Pentesters, just like modern attackers, typically don't rely on missing patches to traverse networks, gather privileges, and access protected data. No vulnerability scanner will warn you that all laptops share the same local administrator password or that a domain admin RDP'd into one of them to troubleshoot an issue (and left their cleartext password cached in memory).
Again, don't get me wrong, I am a big fan of solutions like Palo Alto and CrowdStrike. BUT, simply purchasing and deploying these solutions doesn't make your networks and systems more secure. Like any control, all security solutions must be configured, tuned, and VALIDATED.
Lesson #5: It isn't uncommon to find best of breed security controls running in "monitor only" or "log only" state. After all, the easiest way to start is to convert that old layer 3 ASA config and turn on the security features later. And let's not forget that ALL IT EMPLOYEES should always be whitelisted in these controls because we don't need that stuff in our way.
Contractual, industry, and especially regulatory compliance are all important, but don't let compliance get in the way of being secure. Information security programs should be designed to protect the confidentiality, integrity, availability, and usefulness of information; compliance should just be a benefit of good security.
Secure coding isn't a new concept, but the concept is (unfortunately) new still to many developers. Widely-used and commercial off the shelf (COTS) applications are heavily scrutinized, but your applications may be waiting for the right attacker to come along. A lesson worth sharing is that a breach can be far more costly than validating and potentially fixing issues before the attack.
If you've made it to this point, thank you for reading through. This often isn't what people expect to hear or even want to hear, but sometimes honesty can be blunt and surprising. My advice is always start with a solid foundation and then build on it. Use frameworks like the CIS Top 20 to provide a prioritized roadmap and don't get caught skipping ahead. Good security can be as simple as keeping to the basics.