Architecting on AWS – design for graceful service degradation

Throughout our series of posts we have already seen a variety of architectural patterns that allow us to design scalable and resilient solutions in using the capabilities provided to us by Amazon Web Services (AWS). However, even the best design can have flaws and may show signs of bottlenecks over time or as the demand on your application increases. This could be caused by additional load created by an influx of additional customers using your application or an increasing amount of data that needs indexing in your relational storage tier, to only name a couple.

As the saying goes; the devil is in the detail and your service quality can degrade for a large variety of reasons. While you may not be able to predict and detect each and every potential issue through load testing, you can use a number of architectural patterns to ensure that you continue to interact with your customers or users and therefore have a higher chance of keeping them satisfied.

No matter how well  you plan your solution, it’s unavoidable that some dependencies or processes will live beyond the control of the calling process.. A typical response to this has originally been described by Michael Nygard with the Circuit Breaker Pattern. Many sources already talk about the use of the pattern in the application development space.
The same concept can also be implemented in the AWS infrastructure layer.

Your key ingredients for this are the Route 53 managed DNS service in combination with Route 53 health checks. Route 53 allows you to create primary and secondary DNS record sets for a given record. This is best explained with an example.

Primary DNS record set

Imagine your web site is hosted on a number of web servers that are load balanced using an AWS Elastic Load Balancer (ELB). So in Route 53 you would create an alias record set that points to the ELB endpoint.

AWS Route 53 primary

We then set the routing policy to failover with a record type to Primary. This advises Route 53 to only respond with the IP address of the configured endpoint if the associated resource status is healthy.
For this to work you also need to create a Route 53 health check and associate it within the current record set.
In its most basic configuration you would point the health check to the same target as the DNS entry. Most of today’s modern web applications though are dependent on a variety of service tiers. Therefore you may want to consider the deployment of a custom health service as mentioned in my earlier post on AutoScaling. This way, the status of all sub-services contributing to the overall user experience can be included in your web site’s overall calculation.

Secondary DNS record set

Next we need to configure the secondary record set with the IP of your failover solution. Route 53 will respond with this target when the primary is considered unhealthy. Again, in its most basic form this could just be a public S3 bucket with a static web page that is enabled for website hosting.

AWS S3 Website

When setting up the static site, you need to ensure that the bucket has the same name as your domain as described above. When you finished configuring the static web site, you can jump back to Route 53 to associate the secondary DNS alias record for your domain. This time we are selecting the S3 bucket as the target.

AWS Route 53 secondary

In summary

We tend to stumble across a large variety of different needs in our daily work, each of which demands its own unique solution. For this reason this post can yet again only serve an appetiser.
AWS and Route 53 allows for far more complex scenarios and cascading DNS configurations that allows you to combine regional, weighted and failover records to cater for a wide variety of use cases.

Your solution can also be more sophisticated than a basic static web page that is hosted on S3. Instead you could also fail over to a secondary data centre in a different region or a secondary environment that may provide a limited set of features to the users or your site.
This again may be controlled by the logic in your health reporting service in combination with your application logic. You may, for example, still be able to take orders when the warehouse service is unavailable, though you may not be able to display real time availability information.
However, you would want to switch over to an alternative web site that is informing your customers when your web site is overloaded. The combination of an intelligent application and infrastructure design ensures that existing customers with an active transaction (e.g. a full shopping basket) can continue to check-out, while new visitor to the site are asked for a bit of patience.

As mentioned before, every solution is different. Therefore it is important for you to understand the capabilities that are provided as part of today’s Cloud offerings. This way you can start considering solutions that are beyond the limitations of your traditional infrastructure services.

DISCLOSURE: This post has originally been created for and sponsored by