Posts Tagged ‘web applications’

Having worked on a lot of different projects at a lot of different companies over the years, there are some issues that recur time after time. These are problems that maybe we just don’t have satisfactory patterns for, or developers have a blind spot for. But whatever the reason, they certainly merit more attention in order to reduce how frequently they arise.

Below, I’ve listed three with some suggestions as to how to help mitigate them.

Caching

Caching is essential to good performance in most applications. Remember your ABC – Always Be Caching. Indeed, if you ever want to sound clever in a discussion about system architecture, say “have you thought about caching that?”. Caching is the solution to almost everything. But causes some problems too.

For the non-technical, caching is storing information in short term memory so it can be delivered faster. It’s like remembering things. You probably have a big pile of books at home, and you can’t remember most of them, so for most facts you have to look them up in a book. But you can remember a few things, so for the questions you get asked most, you memorise the facts so you can say them quicker. That’s basically what caching is.

The problem is that when you update something, the cache can end up serving the old content, rather than the new. This results in data being out of date and often results in web applications not appearing correctly because the HTML has updated, but the CSS and JavaScript happens.

Here are a few tips to help mitigate such problems:

• Have scripts to clear out your various caches. Make sure you know what/where the cache layers are.
• Understand what cache headers you are serving via you sending via your web server.
• Have a mechanism for changing files such as stylesheets and JavaScript when you release a new version.
• Turning caching off on dev environments is fine, but make sure you have it turned on on your staging/pre-production environment so you can spot potential problems.

Timezones

While Britain insisted that time starts and stops in Greenwich, we did allow the rest of the world to make adjustments at hourly (or even half-hourly) intervals based on where they are in the world. This has caused no end of headaches for software developers. Even within one country, one problem that comes up time after time is that systems go wrong when the clocks are changed for daylight savings.

Every six months it all goes wrong every six months and we promise to fix it by the next time the clocks change, but usually never do. Even Apple got it wrong on their iPhone alarm – twice!

Here are a few tips to help mitigate such problems:

• Set your servers to GMT/UTC and leave them there, regardless of where you are in the world and where DST is in effect or not.
• Use prper DateTime objects, and specify the timezone.
• When sending a DateTime string, use an ISO standard and be explicit as to what TimeZone it is in.
• Unit test DST changes, using a mock system time if required.
• Perform manual testing on the DST switch.

Load balancing

As systems scale, they inevitably need to spread across multiple processes, applications and bits of hardware. This brings a whole new layer of complexity, though, having to keep everything in sync. It’s very easy to lose consistency and that almost always ends in one thing – say it with me everyone, “race conditions”!

There are two common problems we find in this situation. The first is that one node stops working but the other one is working fine, or maybe something as simple as one node having old code on it. You get intermittent problems but when you try it it works fine or at least works fine 50% of the time. It could be that it is hitting the working node but at other times is hitting the other.

Here are a few tips to help mitigate such problems:

• Ensure you can (and ideally do automatically) test individual nodes.
• Use a tool like Chef or Puppet to make sure all your nodes are consistent.
• Have the ability to easily take out nodes that are not working.
• Aggregate your logging where possible so you don’t have to tail them on each individual node.

The second is race conditions. This is particularly prevalent when using a master and slave database setup. You will write the update to the master than query for it the results – but because your second query is a read, it will be passed to the slave, which may nor may not have updated yet with the new information.

Here are a few tips to help mitigate such problems:

• Have the ability to manually specify master or slave so time critical reads can be pointed at the master.
• Ensuring your staging/pre-production environment load balances everywhere your production environment does so you can spot potential issues.
• Where possible, rely on one server for the time as different machines can become slightly out of sync.

The problem with unit testing is that developers don’t want to write them. No surprise, there, they aren’t the most exciting thing in the world, unless you’re actually a tester and they are your pride and joy.

So, the challenge for creating a good development and testing cycle is how to ensure your developers are motivated to actually write the unit tests. Test driven development is a good way to go about this, and helps stop developers over-engineering too, but it requires a radical shift in your workflow.

Perhaps a less extreme modification, and one developers are more likely to buy into is to replace functional testing with unit testing.

Back in the old days, when you were working on a web application for example, you would write some code, you would then load it up in your browser and see if it worked. If it did you would move onto the next thing and if it didn’t you would go back and fix it.

Then the rise of unit testing for web applications came along and developers were asked to write the code, functionality test it and then write a unit test for it. But none of them did because they didn’t want to write a boring unit test.

However, recently I was writing a model for a web application. Having made my changes, I then wrote a new test for it in our set of unit tests to ensure that it worked as it should. I ran the tests, and I did. But, being pressed for time, I never actually ran a functional test on it. But it worked. Which is no huge surprise, given it passed the unit test.

But what if that was policy? All you had to do was to write the unit test and for it to pass that, you didn’t have to do a functional test. Developers might buy into that, because it takes roughly the same time (probably a little more, but not too much to worry about) and you don’t lose too much. Here is why:

You can’t get away from functional testing. At some point, someone is going to have to do a functional test and at that point they will see if it works or not, and knock anything that doesn’t work back to the developers. This isn’t the case for unit tests – you can go without them, it just means you end up producing pretty rubbish unreliable code and end up spending ages tracing issues down later. But it isn’t an in your face blocker, so developers often ignore it.

Of course, I wouldn’t recommend this for organisations who already have a testing culture embedded. But for organisations that don’t currently unit test, relaxing your rules on functional testing could help you take that first step down the road.

We recently launched Village Chief, a Facebook game in which you take on the role of a local village chief with the aim of growing your community into a thriving settlement.

The project was designed from the ground up to be highly scalable – Facebook cames can go viral very quickly so you need a system which can scale up very quickly if your game suddenly becomes very popular, or you risk losing out on huge amounts of market capitalisation.

First off, it’s written in Ruby. Ruby is a cool scripting language that is similar to Python and it’s popular web framework Ruby on Rails has received a lot of praise being the framework that Twitter and 37signal’s suite (Basecamp, Highrise, etc) are written in. It’s a very forgiving language, even compared to PHP, so if you like your static typing consider Scala instead, but for rapid web development, Ruby is a gem.

Despite the popularity of the Rails framework, I was looking for something a little more lightweight, so Village Chief is built using Sinatra. The documentation is a little underdeveloped but is never the less sufficient to get buy and lets you get a web application up and running nice and fast.

Ruby also includes a great module system called Ruby Gems (spot the earlier pun now 😉 ), a system similar to PHP’s PECL. The game makes heavy use of these – for example, using Koala to interact with Facebook and Dalli to handle our caching.

For a data source, it made sense to use a NoSQL solution. I originally settled on Apache Cassandra. It doesn’t have the flexibility or functionally of Mongo but it terms of raw speed it’s hard to beat and scales onto multiple servers with eventual consistency beautifully.

When it came to deployment, I decided to go with Amazon AWS’s DynamoDB instead. I’ll discuss more about why this was later but this meant rewriting the database interaction. This was easy to do thanks to our database abstraction layer and allowed us to continue using Cassandra for our testing systems while rolling out the production environment on DynamoDB.

Some data was also rolled out using SQLite. This was used for data such as the product catalogue which doesn’t contain user data – just things you can buy. Keeping it in an SQLite database allows us get quick access while still being able to query against the records – and importantly, allows us to version control our product catalogue.

To keep the database load down, some of the sorting was moved into the web server layer instead. On top of this we built a caching system using Memcache to save a lot of the data in memory, thus making things much faster.

When it came to deployment, I decided that Amazon’s EC2 cloud. You can start up a new server in under a minute, which is fantastic for scalability. Thanks to Amazon’s Elastic IPs, you can maintain a set of public IPs and map them to any one of your EC2 servers.

The servers are based on the latest Ubuntu release and to make sure that they are delivering content at top speed, I choose to go with the nginx web server rather than Apache, which works in conjunction with Passenger to provide the platform.

One limitation of EC2 is that you either have to allocate EBS (Elastic Block Storage) which adds additional cost, or use the ephemeral storage which you lose every time you turn the box off. The solution? Put everything into an external database so that we can switch boxes on and off without a problem – hence the decision to use DynamoDB rather than running Cassandra on the servers.

Code is then deployed using Capistrano which goes onto each of the boxes and performs a structed deployment that you can easily roll back, pulling the code direct from our Git repository.

That is a whistle-stop tour of the technologies used in Village Chief. If you have any questions about any of them, feel free to post a comment.

You know, except for the fact that I don’t have a dubiously serviceable spaceship named Serenity (I don’t have a perfectly serviceable one for that matter either). Nor do I have a crew, rugged good looks or such a free-spirited approach to whether to follow the law or not.

Look at him there, what a man.

Pioneers, out there, working for themselves. Well, that is what I will be doing. I recently handed in my notice at Buzz Sports. It has been an amazing three years, but the time has come to move on and establish my own technical consultancy, which will be opening its doors in April.

So if you need help with your software project, get in touch. With expert knowledge of scalable web application architecture and a reassuringly expensive price tag, you’re in safe hands.