Automated infrastructure deployments with CloudFormation

If you are like me (and many other people in the industry), I am sure you can relate to this: You get your hand on a new toy (like CloudFormation) and just want to get going. Therefore you ignore all the on-line help material that is kindly written and published by a myriad of technical writers. Instead you just ‘borrow’ a number of existing scripts (or templates in this case) from the web and tweak them for your purpose. Until you hit a dead end. Where something doesn’t work as expected and you need to start to debug the work you have done borrowed.

This is likely the time where you wish you had a better understanding of the inner workings of a tool. I have used CloudFormation for quite some time now. Looking back at my path of enlightenment I do remember a number of items I wish I had understood better or paid some more attention at the time. For that reason I would like to share a few nuggets that will provide new starters on the topic a somewhat flattened learning curve and provide an outlook on the opportunities and challenges that follow on your first discoveries with CloudFormation templates.

Open your mind to automation

From the standpoint of new adopter of AWS cloud services you may be tempted to disregard CloudFormation templates as a time waster: Automating the deployment of a vanilla EC2 instance with CloudFormation doesn’t save you any time over the manual provisioning of the resources through the AWS Console.
However, we need to be careful to not cheat ourselves here. In the ‘old’ days, when did you ever have to provision a bare metal server containing the core operating system only? Never! One always had to touch the machine to install or at least configure the application stack, database environment, etc. You name it. At this time it was also commonly accepted that the process of ‘crafting’ a new server instance was taking days if not weeks. And this doesn’t even include the time for ordering and delivery.

Also, when was the last time that you only ever needed one of each type? That must have been at one of our side projects where development, test and production was all based on a single code base running on a server hidden somewhere in the attic. Today you tend to require at least four environments to facilitate the software development life cycle. For good measure you probably also want to add one more for a BlueGreen Deployment.
This is the time where you (should) start to manage your infrastructure as code. And at the same time automation becomes the king of the kingdom.

AWS provides a good variety of helper scripts that come pre-installed with all Amazon provided machine images or as executables for the installation on your own images.
In combination with the instructions you provide within the stack templates those automation scripts enable you to deploy an entire infrastructure stack with the click of a few buttons – unless of course you even automate this bit through the use of the CloudFormation API.
Understanding the interdependencies and differences between the various sections of the template and automation scripts helps you with the successful development your stack.

CloudFormation init (cfn-init)

The most commonly used script, aside from cloud-init (but more on this in a later post) would unarguable be cfn-init (CloudFormation init).
CloudFormation init (cfn-init) reads and processes the instructions provided insight the instance metadata as part of the CloudFormation template.
To run cfn-init you need to call it from insight the user data instructions or as part of any of the start-up processes of your own image.

You might like to know that the user data instructions are ‘magically’ executed by cloud-init. Cloud-init is an open source package that is widely used for the bootstrapping of cloud instances and we can dive deeper into this tool set in another post.

Cfn-init accepts a number of command line options. As a minimum you need to provide the name of the CloudFormation stack and the name of the element that contains the instance metadata instructions.

/opt/aws/bin/cfn-init -v --stack YourStackName --resource YourResourceName

This either is the launch configuration or an EC2 instance definition inside the CloudFormation template.
It is important for you to understand that the instance itself does not get ‘seeded’ with the template instructions as part of the launch. In fact: the instance itself has no appreciation of the fact that its launch was initiated by CloudFormation. Instead, the cfn-init script reaches out to the public CloudFormation api endpoint to retrieve the template instructions. This is important to realize if you are launching your instance inside a VPC that has no connectivity to the internet or provides the connectivity via a proxy server that requires explicit configuration.

Configuration sets

CloudFormation init instructions can be grouped into multiple configuration sets.
I strongly suggest you take advantage of this functionality to support the separation of concerns and enable reuse of template fragments.
With its procedural template instructions, CloudFormation doesn’t necessarily support DRY coding practices – and neither is it supposed to do so.
However, if your set-up requires a common set of applications or configurations installed and configured on each instance (think anti-virus, compliance or log forwarding agents, etc.), you are well placed to keep those parts separated in their own configuration set. In combination with a centralised source control management system or an advanced text editor like sublime or notepad++, you can then fairly easy maintain and quickly re-use those common parts of your stack.
Bear in mind though that this isn’t the only solution to ensure common components are always rolled into the stack. In a previous post I have written about the advantages and trade-offs for scripted launches of instances vs the use of pre-baked, customised machine images.
Above solution doesn’t scale well for larger environments. If you want to automate your infrastructure across tens or hundreds of templates, you will soon hit the limits. As your environment requires patching, and you start re-factoring your code fragments, you need to ensure that every stack in your environment is kept up-to-date.
Once you have reached that point, you will start to investigate the use of continuous integration solutions that hook into AWS for a more automated management of stacks across multiple environments.

Be mindful of alternatives

Which leads me nicely to my closing words. I am sure everyone is aware of the popular saying that goes along the line of:

‘if all you have has a tool is a hammer,
everything looks like a nail’.

Rest assured that your infrastructure and deployment solution is subject to the same paradigm. When I started with scripted deployments in AWS I made good use of the user data script. I partioned the various steps within my script and made it re-usable. I split it up into individual bash or powershell scripts that I deployed them to the instances and called them from within the user data or cascaded them amongst each other. And felt very clever! Until my fleet of instances started increasing. Therefore I discovered that a lot of the effort managing the fleet could be saved in using CloudFormation. As part of this the instance definitions moved to CloudFormation Init metadata and provided me with additional flexibility. CloudFormation Init then allowed me to define in a declarative way what actions I wanted to perform on an instance and in which order – much alike the YAM based cloud-init configuration, but at the scale of a whole stack, not just a single instance. No longer did I have to navigate to a specific directory, download a RPM package using wget or curl, install it using the package manager, ensure the application is started at boot time, and so on. Instead I can just provided declarative instructions inside one or more of the 7 supported configuration keys.
As discussed earlier, I yet again felt very smart, I started to organise my individual declarative instructions in configuration sets, managed them in a central repository for re-use, etc. Until, well, you can probably already guess it by now: until I discovered that it is worth considering the use of AWS Opsworks and Elastic Beanstalk resources inside CloudFormation stack. AWS Opsworks abstracts your configuration instructions further away from the declarative configuration in the init metadata. Using a managed Chef service you have access to a large variety of pre-defined recipes for the installation and configuration of additional components of your system.
Since those recipes are continuously maintained and updated by the wider community you don’t need to re-invent the wheel over and over again.
While I do have to admit that the re-invention of the wheel has served humankind quite well to date (imagine our cars would still use stone wheels), it is quite obvious that the wisdom and throughput of a whole community can be much higher then the capability of an individual.
The same can be said about Elastic Beanstalk. Where OpsWorks helped you to accelerate the deployment of common components, Elastic Beanstalk allows you to automate the resilient and scalable deployment of your application into the stack without you even having to describe or configure the details for load balancing and scaling.

In summary

The point I would like to make is that in a world where “the slow eats the fast”, we can never settle at a given solution at any given time. Our whole community, including AWS, is constantly evolving to allow organisations to innovate, develop and ship features at an ever increasing rate. This is achieved in the continuous abstraction away from the core underlying infrastructure and services and the combination of traditional features with new functionality and innovation.
To stay on top of the game as an IT professional there is the need to constantly challenge the status quo and, where applicable, make the leap of faith to investigate and learn new ways of doing our business.

Amazon S3 vs. Amazon Glacier: a simple backup strategy in the cloud

When you start out to design your first application for the hosting on AWS (Amazon Web Services) you will eventually end up considering your options for the protection of your and your customers’ data against accidental losses.
While you may have designed a highly resilient and durable solution, this does not necessarily protected you from administrative mishaps, data corruption or malicious attacks against your system. This can only be mitigated with an effective backup strategy.
Thanks to Amazon’s Simple Storage Service (S3) and its younger sibling Amazon Glacier you have the right services at hand to establish a cost effective, yet practical backup solution.

Within Amazon S3 data is managed as individual objects. This is contrary to Amazon’s Elastic Block Store (EBS) or the local file system of your traditional PC, where data is managed in a directory hierarchy.
The abstraction, away from the lower layers of storage, and the separation of data from its metadata come with a number of benefits. For one, Amazon can provide a highly durable storage service for the fraction of the cost of block storage. You also only pay for the amount of storage you actually use. Therefore you don’t need to second-guess and pre-allocate disk space upfront.

Hierarchical storage with AWS Glacier

Lifecycle rules within S3 allow you to manage the life cycle of the objects stored on S3. After a set period of time you can either have your objects automatically delete or archived off to Amazon Glacier.

AWS S3 LifeCycle

Amazon Glacier is marketed by AWS as “extremely low cost storage”. The cost per Terrabyte of storage and month is again only a fraction of the cost of S3. Amazon Glacier is pretty much designed as a write once and retrieve never (or rather rarely) service. This is reflected in the pricing, where extensive restores come at a additional cost and the restore of objects require lead times of up to 5 hours.

Amazon S3 with Glacier vs. Amazon Glacier

At this stage we need to highlight the difference between the ‘pure’ Amazon Glacier service and the Glacier storage class within Amazon S3. S3 objects that have been moved to Glacier storage using S3 Lifecycle policies can only be accessed (or shall I say restored) using the S3 API endpoints. As such they are still managed as objects within S3buckets, instead of Archives within Vaults, which is the Glacier terminology.

This differentiation is important when you look at the costs of the services. While Amazon Glacier is much cheaper than S3 on storage, charges are approximatey ten times higher for archive and restore requests. This is re-iterating the store once, retrieve seldom pattern. Amazon also reserves 32KB for metadata per Archive within Glazier, instead of 8 KB per Object in S3, both of which are charged back to the user. This is important to keep in mind for your backup strategy, particularly if you are storing a large number of small files. If those files are unlikely to require restoring in the short term it may be more cost effective to combine them into an archive and store them directly within Amazon Glazier.

Tooling

Fortunately enough, there is a large variety of tools available on the web that allow you to consume AWS S3 and Glacier services to create backups of your data. They reach from stand-alone, local PC to enterprise storage solutions.

Just bear in mind that whatever third party tool you are using, you will need to enable with access to your AWS account. You need to ensure that the backup tool only gets the minimum amount of access to perform its duties. For this reason it is best to issue a separate set of access keys for this purpose. You may also want to consider the backup of your data to an entirely independent AWS account. Depending on your individual risk profile and considering that your backups tend to provide the last resort recovery option after a major disaster it may be wise to keep those concerns separated. Particularly to protect yourself against cases like Code Spaces where all services and data within the account got wiped out entirely.
For reference we have included instructions below for the configuration of dedicated backup credentials on the example for my backup tool of choice CloudBerry.

AWS Identity and Access Management

Identity and Access Management (IAM) allows you to manage users and groups for your AWS account and define fine grained policies for the access management of the various services and resources. To get started log-in to the AWS Management Console and open the link for IAM

This opens the IAM Dashboard. Once in the Dashboard you can navigate to Users and select the Create New Users option. Selecting the “Generate an access key for each User” option ensures that an access key is issued for each user at creation time. An access key can be issued at a later time as well though in case you miss that step.

After confirming the dialogue you will be given the opportunity to download the Security Credentials, consisting of an unique Access Key identifier and the Secret Access Key itself. Naturally the Access Key should be stored in a secure place.

As a default, new users will not have any access to any of the resources within the account. Access is granted in attaching an IAM policy directly to a user account or in adding the user to a group with an IAM policy attached. To attach a user policy to an account, select the user and open the Permissions tab.

IAM policies allow for very granular access to AWS resources; hence I am not going into too much detail here. Policies can be defined using pre-defined templates or the policy generation tool. For the purpose of allowing your backup tool write access to your AWS S3 bucket just select the Custom Policy Option.

Below policy grants three different sets of rights:

  • Access to AWS S3 to list all buckets for the account,
  • Access to the bucket MyBucketName and
  • The ability to read, write and delete objects within the MyBucketName bucket.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetBucketLocation","s3:ListAllMyBuckets"],
      "Resource": "arn:aws:s3:::*"
    },
    {
      "Effect": "Allow",
      "Action": [ "s3:ListBucket" ],
      "Resource": [ "arn:aws:s3:::MyBucketName" ]
    },
    {
      "Effect": "Allow",
      "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
      "Resource": [ "arn:aws:s3:::MyBucketName/*"]
        }
  ]
}

If you don’t want to give access to list all available buckets within your account, just omit the first object within the JSON statement. In this case though the bucket name cannot be selected within the application.

    {
      "Effect": "Allow",
      "Action": ["s3:GetBucketLocation","s3:ListAllMyBuckets"],
      "Resource": "arn:aws:s3:::*"
    },

Finally

While this post primarily focussed on backup options for your hosted environments, it is not limited to this. Amazon S3 and Glacier are available world wide through public API endpoints.
Additionally, enterprises can make use of the AWS storage gateway to backup your on-premises data in AWS. Commonly known enterprise backup software from vendors like Commvault, EMC or Symantec also provide you with options to utilise Amazon’s cloud storage as an additional storage tier within your backup strategy.

Misleading error message at ST Meeting Server installation

This week I was challenged with a completely deceptive error message during the installation of the Sametime Meeting Server. The error message claimed that

System Clocks are not synchronized within 5 minutes of one another, Please synchronize for federation.

IBM Installation manager error message

Since I was working in a somewhat unreliable test environment I obviously believed in the message and dutifully compared the system times between the Sametime Systems Console server and the Sametime Meeting server. Without a lot of surprise the times were synchronised and the time zone settings didn’t cause any trouble either. This obviously didn’t help me in any shape or form hence I started to investigate the various log files created by the system.

Looking at the Deployment Manager’s System log I discovered an error, which was logged every time I tried to confirm the deployment plan for the Sametime Meeting Server:

[20/04/12 16:21:13:150 NZST] 00000065 exception     W com.ibm.ws.wim.adapter.file.was.FileAdapter login CWWIM4512E The password match failed.
[20/04/12 16:21:13:151 NZST] 00000065 exception     W com.ibm.ws.wim.adapter.file.was.FileAdapter login
                                com.ibm.websphere.wim.exception.PasswordCheckFailedException: CWWIM4512E The password match failed.
at com.ibm.ws.wim.adapter.file.was.FileAdapter.login(FileAdapter.java:2025)
at com.ibm.ws.wim.ProfileManager.loginImpl(ProfileManager.java:3519)

[20/04/12 16:21:13:153 NZST] 00000065 LTPAServerObj E   SECJ0369E: Authentication failed when using LTPA. The exception is <null>.

This triggered my memory. We recently had to change the password for the local Websphere administrative user. Researching above error message I learned from Dave (Thanks!) that the local file registry could still hold the wrong password. The link in Dave’s post pointed me to the instructions on how to How to reset the administrator’s password in the file registry. While those instructions obviously aim for Websphere Portal they still helped me solve my issues with the Sametime Systems Console deployment manager.

After synchronising all the nodes and restarting the application servers, node agents and the deployment manager I managed to successfully deploy the Sametime Meeting Server.

 

How to reset the administrator’s password in the file registry

IBM Connections Documentation HTML wrapper

If you are like me and constantly being frustrated about the performance of the IBM Connections wiki you can try those two HTML wrapper documents for a more instant experience:

HTML wrapper

HTML wrapper accessible

Just note the comment on the top of each document “This draft is refreshed quarterly, as necessary, but does not contain all of the latest updates and fixes made to the community-owned version of the wiki articles.”

AusLUG12 presentation

Thanks to everyone who attended my session on Lotus Notes package customisation and the positive feedback I have received so far.

Don’t hesitate to drop me a line if you need to know more details on any of the topics covered.

Feel free to download the presentation from here. Most presentations should also be made available in the AusLUG community after the event.