At Greystone we run multiple AWS EC2 instances both for ourselves and to host websites and applications for some of our customers. We recently had a securityn initiative to upgrade our instances to use the newest version (v2) of the Instance Meta Data Service (IMDS). We thought that documenting the process we used might help other people to upgrade their instances.
The AWS definition of the Instance metadata service is :
“Instance metadata is data about your instance that you can use to configure or manage the running instance.”
The Instance Meta Data Service (IMDS) allows you to access that metadata from within the instance. The IMDS service currently has two versions, version 1 (v1) and version 2 (v2). With v2 being released in November 2019 to address some serious security flaws in the service.
Both versions are still supported and there are no plans for v1 to be decommissioned, but it is highly recommended to migrate your current and future instances to v2. Why? You may ask, because IMDSv2 adds protections for misconfigured website application firewalls, reverse proxies, layer-3 firewalls, network address translations, and unpatched SSRF vulnerabilities.
Interestingly, these additional protections would have mitigated the exploits used in the Capitol One breach in 2019.
That all sounds excellent, “we should upgrade right away,” we thought to ourselves. But how did we go about updating all our instances, or in fact even find if we were even using the IMDS service?
Upgrade Process
Firstly, we broke down the upgrade process into three steps, discovery, preparation, and remediation.
Discovery
To upgrade our instances to IMDSv2, firstly we needed to know which ones are using IMDSv1.
There are multiple ways of doing this. I have listed the three below, and each is equally as valid as the other for discovery.
CloudWatch
The first is to use AWS CloudWatch and use the metric MetadataNoToken to view which instances are making calls to the IMDS without using a token to authenticate to it first.
To do this
- Navigate to the CloudWatch console page.
- Click Explorer.
- Type MetadataNoToken in the Metris field.
- Select State: Running from the From field.
This generated a new graph to show us the MetadataNoToken calls made by our instances. Some instances did not make any calls and can be ignored but if like in the example above the number of calls is above 0 then the instance is still using the IMDSv1 service.
We then made a note of the instance ID from the legend below the graph as we will need this going forward.
Metabadger
If you do not want to use CloudWatch for whatever reason, Salesforce publish an excellent tool called Metabadger, which can be found here.
After installing it, Metabadger will allow you to find the same information using the command
metabadger cloudwatch-metrics.
We ran the Metabadger app with the command line switch cloudwatch-metrics which displayed the same information as the CloudWatch method, but it is displayed in the Command Line Interface (CLI) with a nice table view.
Prowler
A third option is to use the excellent Prowler which can be found here: https://github.com/prowler-cloud/prowler
Once prowler is installed, we ran it using the following command:
This then ran checks on all instances and reported back on the state of the IMDS.
?Prowler is so much more than just a tool to check IMDS status, but its capabilities are beyond the scope of this article.
Preparation
Once we knew which instances were using IMDSv1 we needed to prepare them to use IMDSv2.
AWS says, “To use IMDSv2, make sure that your EC2 instances have the latest versions of the AWS SDKs and CLIs“. So, we needed to find and update all the AWS SDKs and CLIs on our instances.
Windows
For our Windows based servers, we followed the following procedure (For this example, I will use one of our database servers that was still making IMDSv1 calls).
Firstly, we used AWS Systems Manager to search the software inventory on the server using the Fleet manger tooling.
We could have also done this by RDPing onto the server and looking in the Programs and features item (Start > Control Panel > Programs and Features) or using our RMM tooling.
Once we had a list of AWS published software, we checked that it was all up to date (Spoiler, it was not as it was still making IMDSv1 calls).
In our example server the EC2ConfigService install was out of date and needed to be updated.
Finding Windows processes making IMDSv1 calls.
For some of ours instances it was not immediately clear what software was out of date and we needed to find the specific app that was making the IMDSv1 calls. In these cases, we used the inbuilt Windows Resource Monitor Network monitor to find the Image and PID of the application making calls.
To do this open Resource Monitor (Start->Search ->Resource Monitor) and click on the Network tab. Then look for calls in the Network Activity section made to either the IP or DNS entries listed in the table below:
? If your instances are not in the eu-west-1 region the DNS entry will display a different region.
In the image below you can see the application Ec2Config.exe making a call to the address instance-data.eu-west-1.compute.internal. Using this technique, we found all the applications that were making calls the IMDS and consequently needed to be updated.
?Note. Instance-data.eu-west-1.compute.internal is an internal DNS zone for EC2 instances that resolves to the instance metadata ip of 169.254.169.254.
Linux
For our Linux instances we used the netstat command to find what applications are making calls to the instance-data.eu-west-1.compute.internal.http address by running the command:
Below is an example of curl making the connection, but this would be different depending on what software is making the call.
Once we had found all the software on the instance that were making calls to the IMDS and had updated them to the latest versions, we moved onto the next stage of the process.
Remediation
Once we were confident that we had found the instances making IMDSv1 calls, updated the appropriate software on them and then gone back and re-run our discovery tools again to check it had worked we moved onto the next stage which was making sure all instances present and future are forced to use IMDSv2.
There are many ways of enforcing IMDSv2 on our instances, but we used the following two.
AWS CLI
To force our instances to use IMDSv2 we need to set the following EC2 instance switches:
We did this using the AWS CLI and using the following command:
Once the HTTPTokens switch is set to required we were not able to access the IMDS service without authenticating using a token, and thus were forced to use the IMDSv2 for all calls.
Metabadger
We could also have used our trusty friend Metabadger to set all the instances to use IMDSv2. Once we are confident that all our instances are ready to be updated, we ran the command:
which set all the instances to require HTTPTokens:
Going forward
To make sure all instances use IMDSv2 in the future we used the following techniques to set the IMDSv2 on all new EC2 instances.
EC2 launch templates
We use EC2 launch templates to deploy autoscaling groups in our environment and these were updated to set all future instances to require tokens for accessing the IMDS.
To do this, we edited our launch templates EC2 Console -> Launch templates -> select template -> Modify template (Create a new version).
Then set the following fields:
?Once you have saved the template do not forget to change the default version of the template to the latest version you have just created.
Manually deploying servers
If you are manually configuring servers through the portal, you can set the Metadata accessible and Metadata version switches in the Advanced Details section of the Configure instance page.
They should be set to:
Terraform
We also use the Terraform metadata options resource block in the aws_instance resource, we just needed to the set the following fields
Service Control Policies (SCP)
We can also use SCP policies to enforce the use of IMDSv2 on all new instances.
We can use the example policy from the official AWS documentation found here
The following code will create a policy to force a condition that the ec2:MetadataHttpTokens field is required.
Checking it has worked
To check all our hard work, we simply reran one of the checks listed in the discovery section to see if the instance is still making IMDSv1 call.
Additional Links
AWS has an article with guidance on how to transition to using IMDSv2 for instances which can be found here.
