Call us: 0161 726 5020

Careers | Blog | Contact Us

GreystoneGreystone
  • IT Services
    • Managed Support Services
    • Cyber Security Services
    • Modern Workplace and Intune
    • Microsoft 365
    • IT Director as a Service (ITDaaS)
    • Data Backup and Disaster Recovery
    • Network connectivity and Managed Wi-Fi
    • Virtualisation and Servers
  • Software
    • Consultancy
    • Custom Software Development
    • System Integrations
    • Web Development
  • Cloud
    • What is the Cloud
    • Cloud Consultancy
    • Migration Services
    • Security Services
    • Cost Management
    • Automation
    • Monitoring and Reporting
    • Optimisation
    • Auditing
  • Case Studies
  • About
  • Contact

Careers | Blog | Contact Us

GreystoneGreystone
Greystone
  • IT Services
    • Managed Support Services
    • Cyber Security Services
    • Modern Workplace and Intune
    • Microsoft 365
    • IT Director as a Service (ITDaaS)
    • Data Backup and Disaster Recovery
    • Network connectivity and Managed Wi-Fi
    • Virtualisation and Servers
  • Software
    • Consultancy
    • Custom Software Development
    • System Integrations
    • Web Development
  • Cloud
    • What is the Cloud
    • Cloud Consultancy
    • Migration Services
    • Security Services
    • Cost Management
    • Automation
    • Monitoring and Reporting
    • Optimisation
    • Auditing
  • Case Studies
  • About
  • Contact

The mysterious case of the failing vss backups

10 August 2023 /Posted bygR3yS103

Internally we use AWS backups to backup our EC2 instances. The backup plan is set to use windows vss aware snapshots for these backups. 

During monitoring we noticed that this worked for some servers and not for others. For the servers that it was failing on, it would fail with the error:  

Completed with issues. 

Windows VSS Backup attempt failed because of insufficient privileges to perform this operation. 

This seemed odd. All our EC2 instances have the same IAM role attached to them, and we know that it worked for some of the instances.  

So, we started to dig, the first thing we needed to figure out is what was causing the insufficient privileges error. A good place to start was to see if the Systems manager Run command output was generating any error messages. We navigated to AWS System manager -> Run Command and went to command history added a filter for Status. 

Set filter to Failed. 

We found a failed AWSEC2-CreateVSSSnapshot command for one of the impacted servers and clicked on the Command ID to open the details page up. 

Within the run command you can see the output of the impacted Instance ID in Targets and outputs then clicking the View output button. 

We can see that we’re getting the error message.  

Call to Get-EC2Instance threw and Exception, Verify that your instance role has the Describe-Instances permission 

This seemed a little off to us as when we looked at the IAM role attached to the EC2 instance we can see the following policy associated with it that has the “ec2:DescribeInstances” IAM action for all resource (line 15). We also know that this role/policy combination works for other servers. 

So, to test that the role was working properly on the EC2 instances impacted by the problem we remoted on using Fleet manger, opened a PowerShell window, and ran the command get-ec2instances. 

This command should have picked up the IAM role associated with the EC2 instance and returned some results, but the command is saying that we don’t have any permissions.  

Making a call to the IMDSv2 service we can see that the instance has got the correct IAM role associated with it. So why is it not working? 

A clue to why this was not working is back in the output of the run command we looked at earlier, I’ve highlighted two sections, one with the error but the first highlighted section shows the PowerShell module version. 

Check your trusted entities look correct.

Check you have assigned the correct permission policies and add tags to the role. Then click Create role.

Add Permissions - AWS

Having done some prior checking for this problem we looked at the prerequisites listed here https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/application-consistent-snapshots-prereqs.html to do VSS based snapshots. There is a requirement that AWS PowerShell is above version 3.3.48.0. So, we initially ignored this as 3.3.542.0 is higher than 3.3.48.0, so it should work right? 

Previously we had enabled IMDSv2 on all our EC2 instances, which means that you can’t just call the metadata service on 169.254.169.254 and get a set of temporary access keys, you have to request a token first and use that (see this article for more information https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-metadata-v2-how-it-works.html ) 

It turns out AWS PowerShell versions below 4.0.1.0 do not support this and so weren’t able to retrieve a set of temporary access keys using IMDSv2, which in turn meant that the calls it was making for access  via it’s IAM role failed and stopped the backups from performing vss snapshots. 

To test this, we installed the newest version of the AWS PowerShell modules on the instance and reran our test command get-ec2instances and as you can see it now returns a list of instances, which means it was able to use the EC2 instance IAM role attached to it.  

Then to see if this had fixed our original problem, we ran a manual vss backup and looked at the run command results. You can see below the backup was Successfully without any warnings. 

Latest Blog Posts

Man leaping from one rock to another with sunset

How to Successfully Transition from In House ...

22 April 2025 Comments Off on How to Successfully Transition from In House IT to Managed IT Support
Larger wooden rocket with smaller rockets aside it

The Long Term Value of Outsourcing IT. ...

25 March 2025 Comments Off on The Long Term Value of Outsourcing IT. Why SMEs Thrive with Managed Support.
Man sat at laptop with small green office block

Driving Sustainability Across Business and Technology Operations 

26 February 2025 Comments Off on Driving Sustainability Across Business and Technology Operations 
Multiple office windows

The IT Implications of Employers Moving Employees ...

19 February 2025 Comments Off on The IT Implications of Employers Moving Employees Back to the Office Full Time

What UK SMEs Need to Know to ...

30 January 2025 Comments Off on What UK SMEs Need to Know to Stay Competitive In 2025

Categories

  • Cloud
  • Greystone News
  • IT Security
  • IT Support
  • Management Consulting
  • Networking
  • Servers
  • Software Development
  • Uncategorised

Our Partners

aws partner network

microsoft partner

databarracks

internet central

highlander

mitigate

cyfor

huntress

pax8

Twitter Linkedin

0161 726 5020 | info@greystone.co.uk

Copyright | Cookie Policy | Terms & Conditions | Privacy Policy | Sitemap

Play It Green

Cyber Essentials

Website design by Creative Essence.

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.Ok