Automated infrastructure deployments with CloudFormation

If you are like me (and many other people in the industry), I am sure you can relate to this: You get your hand on a new toy (like CloudFormation) and just want to get going. Therefore you ignore all the on-line help material that is kindly written and published by a myriad of technical writers. Instead you just ‘borrow’ a number of existing scripts (or templates in this case) from the web and tweak them for your purpose. Until you hit a dead end. Where something doesn’t work as expected and you need to start to debug the work you have done borrowed.

This is likely the time where you wish you had a better understanding of the inner workings of a tool. I have used CloudFormation for quite some time now. Looking back at my path of enlightenment I do remember a number of items I wish I had understood better or paid some more attention at the time. For that reason I would like to share a few nuggets that will provide new starters on the topic a somewhat flattened learning curve and provide an outlook on the opportunities and challenges that follow on your first discoveries with CloudFormation templates.

Open your mind to automation

From the standpoint of new adopter of AWS cloud services you may be tempted to disregard CloudFormation templates as a time waster: Automating the deployment of a vanilla EC2 instance with CloudFormation doesn’t save you any time over the manual provisioning of the resources through the AWS Console.
However, we need to be careful to not cheat ourselves here. In the ‘old’ days, when did you ever have to provision a bare metal server containing the core operating system only? Never! One always had to touch the machine to install or at least configure the application stack, database environment, etc. You name it. At this time it was also commonly accepted that the process of ‘crafting’ a new server instance was taking days if not weeks. And this doesn’t even include the time for ordering and delivery.

Also, when was the last time that you only ever needed one of each type? That must have been at one of our side projects where development, test and production was all based on a single code base running on a server hidden somewhere in the attic. Today you tend to require at least four environments to facilitate the software development life cycle. For good measure you probably also want to add one more for a BlueGreen Deployment.
This is the time where you (should) start to manage your infrastructure as code. And at the same time automation becomes the king of the kingdom.

AWS provides a good variety of helper scripts that come pre-installed with all Amazon provided machine images or as executables for the installation on your own images.
In combination with the instructions you provide within the stack templates those automation scripts enable you to deploy an entire infrastructure stack with the click of a few buttons – unless of course you even automate this bit through the use of the CloudFormation API.
Understanding the interdependencies and differences between the various sections of the template and automation scripts helps you with the successful development your stack.

CloudFormation init (cfn-init)

The most commonly used script, aside from cloud-init (but more on this in a later post) would unarguable be cfn-init (CloudFormation init).
CloudFormation init (cfn-init) reads and processes the instructions provided insight the instance metadata as part of the CloudFormation template.
To run cfn-init you need to call it from insight the user data instructions or as part of any of the start-up processes of your own image.

You might like to know that the user data instructions are ‘magically’ executed by cloud-init. Cloud-init is an open source package that is widely used for the bootstrapping of cloud instances and we can dive deeper into this tool set in another post.

Cfn-init accepts a number of command line options. As a minimum you need to provide the name of the CloudFormation stack and the name of the element that contains the instance metadata instructions.

/opt/aws/bin/cfn-init -v --stack YourStackName --resource YourResourceName

This either is the launch configuration or an EC2 instance definition inside the CloudFormation template.
It is important for you to understand that the instance itself does not get ‘seeded’ with the template instructions as part of the launch. In fact: the instance itself has no appreciation of the fact that its launch was initiated by CloudFormation. Instead, the cfn-init script reaches out to the public CloudFormation api endpoint to retrieve the template instructions. This is important to realize if you are launching your instance inside a VPC that has no connectivity to the internet or provides the connectivity via a proxy server that requires explicit configuration.

Configuration sets

CloudFormation init instructions can be grouped into multiple configuration sets.
I strongly suggest you take advantage of this functionality to support the separation of concerns and enable reuse of template fragments.
With its procedural template instructions, CloudFormation doesn’t necessarily support DRY coding practices – and neither is it supposed to do so.
However, if your set-up requires a common set of applications or configurations installed and configured on each instance (think anti-virus, compliance or log forwarding agents, etc.), you are well placed to keep those parts separated in their own configuration set. In combination with a centralised source control management system or an advanced text editor like sublime or notepad++, you can then fairly easy maintain and quickly re-use those common parts of your stack.
Bear in mind though that this isn’t the only solution to ensure common components are always rolled into the stack. In a previous post I have written about the advantages and trade-offs for scripted launches of instances vs the use of pre-baked, customised machine images.
Above solution doesn’t scale well for larger environments. If you want to automate your infrastructure across tens or hundreds of templates, you will soon hit the limits. As your environment requires patching, and you start re-factoring your code fragments, you need to ensure that every stack in your environment is kept up-to-date.
Once you have reached that point, you will start to investigate the use of continuous integration solutions that hook into AWS for a more automated management of stacks across multiple environments.

Be mindful of alternatives

Which leads me nicely to my closing words. I am sure everyone is aware of the popular saying that goes along the line of:

‘if all you have has a tool is a hammer,
everything looks like a nail’.

Rest assured that your infrastructure and deployment solution is subject to the same paradigm. When I started with scripted deployments in AWS I made good use of the user data script. I partioned the various steps within my script and made it re-usable. I split it up into individual bash or powershell scripts that I deployed them to the instances and called them from within the user data or cascaded them amongst each other. And felt very clever! Until my fleet of instances started increasing. Therefore I discovered that a lot of the effort managing the fleet could be saved in using CloudFormation. As part of this the instance definitions moved to CloudFormation Init metadata and provided me with additional flexibility. CloudFormation Init then allowed me to define in a declarative way what actions I wanted to perform on an instance and in which order – much alike the YAM based cloud-init configuration, but at the scale of a whole stack, not just a single instance. No longer did I have to navigate to a specific directory, download a RPM package using wget or curl, install it using the package manager, ensure the application is started at boot time, and so on. Instead I can just provided declarative instructions inside one or more of the 7 supported configuration keys.
As discussed earlier, I yet again felt very smart, I started to organise my individual declarative instructions in configuration sets, managed them in a central repository for re-use, etc. Until, well, you can probably already guess it by now: until I discovered that it is worth considering the use of AWS Opsworks and Elastic Beanstalk resources inside CloudFormation stack. AWS Opsworks abstracts your configuration instructions further away from the declarative configuration in the init metadata. Using a managed Chef service you have access to a large variety of pre-defined recipes for the installation and configuration of additional components of your system.
Since those recipes are continuously maintained and updated by the wider community you don’t need to re-invent the wheel over and over again.
While I do have to admit that the re-invention of the wheel has served humankind quite well to date (imagine our cars would still use stone wheels), it is quite obvious that the wisdom and throughput of a whole community can be much higher then the capability of an individual.
The same can be said about Elastic Beanstalk. Where OpsWorks helped you to accelerate the deployment of common components, Elastic Beanstalk allows you to automate the resilient and scalable deployment of your application into the stack without you even having to describe or configure the details for load balancing and scaling.

In summary

The point I would like to make is that in a world where “the slow eats the fast”, we can never settle at a given solution at any given time. Our whole community, including AWS, is constantly evolving to allow organisations to innovate, develop and ship features at an ever increasing rate. This is achieved in the continuous abstraction away from the core underlying infrastructure and services and the combination of traditional features with new functionality and innovation.
To stay on top of the game as an IT professional there is the need to constantly challenge the status quo and, where applicable, make the leap of faith to investigate and learn new ways of doing our business.

Architecting on AWS: Optimising the application design

In our practice we hear a variety of misconceptions and misinterpretations in relation to the benefits of moving workloads ‘into the cloud’. You should be very vary if someone wants to make you believe that the pure migration of a traditional application to a cloud services vendor will make it any more scalable or reliable. Of course, you can scale vertically in increasing the size of your compute nodes. However, this still restricts you to the maximum size of instances available. Scaling horizontally on the other hand, in distributing your workload over multiple instances, requires special considerationswith regard to optimizing the application design.

As part of your migration strategy you should also critically review your existing application components and consider if any of the high level capabilities provided by your cloud vendor can deliver the same functionality at reduced cost, increased reliability, higher flexibility, et cetera. This story from marketing software provider Moz also serves as a timely reminder that your mileage may vary depending on the type of your workload and non-functional requirements. Therefore we can only provide a general guide as each option needs to be considered against its advantages and disadvantages against the overall solution.

Please note that we tend to angle our posts along the service catalogue of Amazon Web Services (AWS). However, most of the strategies and patterns we describe will also apply to other cloud vendors that provide capabilities with similar characteristics. With that in mind, let’s review our two tier example application from the first post in this series. To support the elasticity and reliability of our solution, we should consider a few concerns from the application perspective.

Optimizing the application design: session management

In the design we are using the Elastic Load Balancer (ELB) for the distribution of load between instances. By default, the ELB routes each request to the instance with the smallest load. This is also referred to stateless load balancing and needs to be considered in the design of your application. Unless you are having strong reasons to store your session state in memory on the web server, you should migrate to an external session provider. A common approach for this is the use of a session table within a relational database system. However, this doesn’t come without risks as the database system may be unnecessary flooded with session requests. This is potentially impacting on the overall performance of the application and causing scalability issues in the database tier.

On AWS you are better off to use DynamoDB. Amazon DynamoDB is a key-value data store (with recently added support for document data model) which delivers configurable, predictable performance. For that reason DynamoDB is an ideal candidate to consider for the management of session state. As a fully managed service you won’t even have to worry about any operational or administrative cost.

Optimizing the application design

A word on RDS and scaling

If you are an observant reader you may have already spotted a snag. If you really want your overall solution to be truly scalable, you need to ensure that this is applied to each tier of the application. This is posing a couple of issues in the traditional relational database space. While you have the ability to ‘crank-up’ the amount of provisioned IOPS for your RDS instances, there is no ability to autoscale database instances in the same fashion as EC2.

A common design pattern is the separation of write operations from read operations. By the general nature of storage systems read operations tend to be proportionally higher than database write operations. For that reason you can offload read requests to a dedicated (set of) read replicas to provide some level of scalability. Based on the underlying design limitations within the storage engines though, this cannot be provided fully automated. Amongst other limitations we also need to highlight that the support of read replicas in RDS is limited to MySQL, PostgreSQL and the new Aurora storage engines.

Alternative database technologies

All this is obviously not going to be an issue for you if you are dealing with very steady and predictable workloads. If you do have the need to deliver a scalable solution though, you will eventually get to the stage where you need to consider alternative mechanisms to reduce the load on your relational database environment. An obvious choice would be the consideration of alternative database technologies like Amazon DynamoDB or Amazon SimpleDB for the data that doesn’t require a relational structure.

Content distribution

You can also reduce the strain on your application and database services in employing caching services like Amazon CloudFront. As described earlier, CloudFront provides a large number of edge locations across the globe that act like a massive cache for web and streaming content. The cache behaviour settings allow you to optimise the cache behaviour for the unique needs of your application. As an added bonus this will also improve the overall user experience for your customers.

Optimizing the application design

Object storage

Finally we briefly want to touch on the Amazon S3 storage service. Again, many traditional application designs make us of relational or file system resources for the storage of BLOB objects. While you can certainly continue on that path, we recommend to rethink that approach for a number of reasons. For one, you obviously need to continue to provide and manage your own file system or relational database environment. You also have to ensure that the systems are always up-to-date, ensure that you have got enough available disk space available and the systems are actually available to meet your service level agreements.

If those operational reasons haven’t put you off yet, you may want to consider the actual costs for storage. Based on the figures for my ‘home’ region Sydney, the cost of storing 100 GByte of data on S3 is approximately 30% of the cost of Elastic Block Storage or RDS. And the pure cost of storage doesn’t even include the cost for utility compute to power the relational database or file system environments. So unless there is a specific need for a direct attached, high performing local disk e.g. for the hosting of a COTS solution like SAP, we strongly recommend to consider the use of the S3 object store where applicable.

With those initial teasers in mind you should start exploring the AWS service catalogue and our rich training content on CloudAcademy to consider what other services you could utilise to address the unique concerns of your solution.

DISCLOSURE: This post has originally been created for and sponsored by CloudAcademy.com.

Using IAM credentials to grant access to AWS services

Having used Amazon Web Services (AWS) for quite some time now, I realised that it should be time to start sharing some of my experiences on this blog, particularly considering that I haven’t contributed to the World Wide Web community for quite some time.
Today’s post is about Amazon’s Identity and Access Management (IAM) service and why it is a good idea to use it.
I am using the great backup solution from CloudBerry to backup important files from my laptop on Amazon S3. While I am absolutely excited about the capabilities of the application, I still did not feel comfortable to provide my AWS root account access keys (as described in CloudBerry’s help file) to the application.

Why is that?
To fully understand the issue we need to be aware about the difference between AWS root credential and an IAM identity. The AWS root account is provisioned for all AWS users and has full access to all resources and services in the account. Sharing the secret access key for this identity with a 3rd party potentially gets you into big trouble; a malicious piece of software may use the credentials to wipe out all your data, terminate instances or, potentially worse, subscribe to a new raft of additional services that you will have to pay for at the end of the month. Scary stuff right? So please read on. Continue reading

Doclink does not work with LS created Doclink

Today I had an interesting issue I definitely need to put out here to be able to remember it in the future.

I used a piece of code (see below) to append a link to a document, sent out via an email message to another user. When the recipient opened that message and clicked the doclink a dialogue box came up “The Doclink database cannot be located.” asking the user to select the server where the linked database is located. This issue however did only happen in some circumstances. Continue reading