Quick Tip: Enabling S3 Bucket Versioning in CloudFormation

A cloth and broom handle in a bucket

Yesterday I needed to find a way to automate enabling versioning on an S3 bucket that’s created as part of a CloudFormation stack. While it would be nice to be able to do this within CloudFormation templates, and you’d think it would be an obvious feature to have, unfortunately it’s not possible at the moment.

So to work around this, I used the aws-cli tool from within the instances which are also created by the template. The aws-cli tool is available here;

There’s a slightly out-of-date version included in the Amazon Linux 2013.03 AMI, however I found that this version has a bug that causes it to pass an invalid signature to AWS. I did however find that this has been fixed in the latest version, 0.11.0 as I write.

So, here’s the steps I went through to upgrade aws-cli on Amazon Linux 2013.03;

yum remove aws-cli
wget https://github.com/aws/aws-cli/archive/master.zip -O aws-cli-latest.zip
unzip aws-cli-latest.zip
cd aws-cli-master/
python setup.py install

Then you need to setup your AWS credentials. There’s a couple of methods described on the GitHub page, however if you’re using CloudFormation you probably want to consider using an IAM role to avoid/minimise the need to inject credentials into your instances. Following the principal of least privilege, the only permission you need to grant the role is the “s3:PutBucketVersioning” action with a policy similar to this;

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:PutBucketVersioning"
      ],
      "Sid": "Stmt1370075170000",
      "Resource": [
        "*"
      ],
      "Effect": "Allow"
    }
  ]
}

And of course filter the “Resource” list appropriately for your setup.

After you’ve got that setup you can test it by trying to read the current versioning setting from your bucket;

aws s3 get-bucket-versioning --bucket <bucket-name> 

And then set the versioning as required;

aws s3 put-bucket-versioning --bucket <bucket-name> --versioning-configuration status=Enabled

or;

aws s3 put-bucket-versioning --bucket <bucket-name> --versioning-configuration status=Suspended

So stick the appropriate command into the instance metadata, so that it’s run by cfn-init as the instance boots, and Bob’s your uncle! For example;

"AWS::CloudFormation::Init" : {
  "config" : {
    "sources" : {
      "/opt/aws-cli" : "https://github.com/aws/aws-cli/archive/master.zip"
    },
    "commands" : {
      "1-remove_old_aws-cli" : {
        "command" : "yum remove aws-cli",
        "ignoreErrors" : "true"
      },
      "2-install_latest_aws-cli" : {
        "command" : "/usr/bin/python setup.py install",
        "cwd" : "/opt/aws-cli/aws-cli-master"
      },
      "3-enable_instancebucket_versioning" : {
        "command" : { "Fn::Join" : ["", [ "/usr/bin/aws s3 put-bucket-versioning --bucket ", { "Ref" : "YourBucket"}, " --versioning-configuration status=Enabled\n" ]]},
      },
    }
  }
}

Thanks for reading :)

Update 04-Jun-2013:

I found out late last night that there’s a small issue with this method of enabling bucket versioning. It turns out that installing newer versions of aws-cli breaks cfn-init!

It’s all down to the “requests” Python library. The current version of aws-cli requires requests 1.2, and unfortunately cfn-init isn’t compatible with this version and instead requires requests 1.1. When requests 1.2 is present, cfn-init fails to run with an error ending in;

 KeyError: 'pre_request'

So, the CloudFormation snippet I posted earlier works fine so long as you only need to run cfn-init once, during the initial boot of the instance. If you need to run cfn-init again after boot, for example if you’re using cfn-hup, please don’t use it as it will cause cfn-init to fail!

Apologies for that! If I find a workaround or alternate method of doing this I’ll update this post with that information as well.

Advertisements

2 responses to “Quick Tip: Enabling S3 Bucket Versioning in CloudFormation

  1. the aws-cli cfn-init problem has been fixed on the curent amazon linux version. so now you just need to run yum update and the aws-cli and cfn-init are both good to go.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s