Multivariate testing: how to prove your website changes work
For many big internet companies – Amazon, say, or Google – the idea that there’s a single version of their website is laughable. That’s because these firms are constantly experimenting with different designs, layouts, colours, processes or text. By using website analysis and optimisation tools, these companies can create multiple variants of a page and then measure which version of the page delivers the best results. In practice, this means that you might see a different version of Amazon’s checkout screen to everyone else. Or you might get a sneak preview of a new Google feature. These sites are being constantly fine-tuned, as testing tools provide an insight into what page design sparks the best results. A/B and multivariate testing The simplest form of this sort of testing is called A/B testing or split testing. It’s where you compare your current version of a page with an alternative version which you think might product different results. For instance, you might change the layout of your order form to see if doing so results in a greater proportion of people completing their orders. A more complex option is multivariate testing. When you perform multivariate testing, you can experiment with many different website elements at once in order to determine which combination of elements performs best. The beauty of A/B and multivariate testing lies in its accuracy. Everything you try is tested in a live environment, on real website visitors. You can measure the results exactly and be sure they’re representative. It’s not like doing a survey or asking for people’s opinions, where people can lie or their answers are open to misinterpretation. In short, you can prove which version of a page produces the best results, with statistical accuracy. Getting started with multivariate testing It could be easier than you think to implement some A/B or multivariate testing on your own website. If you’re willing to pay someone to do it for you, there are lots of companies who can advise you on what to test and set tests up for you. However, if you’d rather give it a go yourself, there are some excellent free tools built into Google Analytics (Google’s free web analytics service) that can help you. If you’re interested in trying out some A/B or multivariate testing then this is an excellent place to start – but it does mean you’ll need to be using Google Analytics to measure your website’s traffic too. Assuming you are, you’ll find all the testing tools under the ‘Content Experiments’ option. To find this, log in to Google Analytics, click ‘Standard Reporting’, and then click the ‘Content’ menu. There’s not enough space here to explain how to use Content Experiments, but there’s heaps of helpful stuff on Google’s pages, plus a good tutorial here. (Please note that Content Experiments is a new service from Google which is replacing the company’s existing Website Optimizer. If you can’t see it in your Analytics settings straight away then you it should appear soon.) Deciding what to test What you decide to test on your website will depend on its aims. For instance, you might experiment with: The size, color or position of your ‘Buy now’ buttons (e.g. are blue buttons better than green ones?) The fields displayed on your registration form (e.g. with fewer fields, do more people complete the form?) How you communicate special offers (e.g. is ‘save 20%’ or ‘save £5’ more effective?) The length and format of product descriptions (e.g. does a list of bullets perform better than a paragraph of text?) You can test virtually any aspect of your site, so to narrow down the possibilities, think about your website’s aims, then consider which elements might have the greatest impact. Getting accurate results One thing to remember when it comes to testing on your site is that it takes time to achieve meaningful results. You can’t count on the first ten, 50 or even 100 visitors to be representative of how you users behave overall. Look for the ‘statistical confidence’ figure in your testing tool. As more people visit your website this should creep closer to the testing standard of 95% confidence, and it’ll give you a really good idea of how reliable the results are. You also need to think about how many visitors your website receives overall. High-traffic websites can test many different variations because they have enough visitors to show them all to. But if your website only receives a few thousand visitors each month, it’ll take you much, much longer to see any reliable results. A/B tests generally achieve high confidence scores much quicker than multivariate tests, so they’re the best way to start if your website doesn’t receive masses of traffic. However, A/B tests usually only tells you about a specific change to a certain page. Multivariate testing can give you insight that can be used across a site. Dangers of multivariate testing Website testing – and multivariate testing in particular – can be strangely addictive. It’s amazing that you can come with an idea, test it, and see scientifically what’s best for your website. But beware: a devotion to website testing could have you focusing on tiny incremental changes while your competitors make a giant leap ahead. A/B and multivariate testing can be a really valuable way to improve your website and boost your bottom line. But it’s only one weapon in your website-building armoury. Learn more about testing on your website: Smashing Magazine have an excellent and comprehensive guide Here’s a good introduction to A/B testing from A List Apart Read five good A/B testing case studies from Econsultancy
Is the EU’s cookie law confusing you too?
You might remember that in May last year some new EU rules came into force. Dubbed the ‘EU cookie law’, the legislation says websites have to get permission from users before storing pieces of information called cookies on their computers. What are cookies? Cookies are small text files which websites place on visitors’ computers. They’re typically used to identify that particular visitor and provide them with a better experience. Cookies are a crucial part of many website functions which we take for granted. For instance, it’s usually cookies which allow online shops to remember what items you have in your shopping trolley. They’re also used to keep you logged in to a website, or to provide valuable usage statistics and information to website owners. If you run a website, it almost certainly uses cookies. Their most likely function is to monitor visitor numbers and behaviours through tools like Google Analytics. They may also be used to display relevant adverts to visitors, or – if you sell online – to power key parts of your online shopping system. What the cookie law says Although the cookie law came in last year, the Information Commissioner’s Office (ICO) decided to give websites a year to implement the rules. The deadline is 26 May 2012 – after that date, websites which don’t comply with the law could be fined up to £500,000. The implications of the new cookie law could be far-reaching, yet there’s still a lot of confusion about how the rules should be interpreted and what websites need to do to comply. As a result, many websites are taking a ‘wait and see’ approach. In short, the new rules require websites to get permission from visitors before placing any cookies on their computer. Permission must be informed and overt, which means you have to ask visitors outright if you can put cookies on their computer, and explain clearly what the cookies are used for. You can’t bury the information in your website’s terms and conditions and leave it at that. The only exceptions to this are cookies deemed essential to providing functions visitors have asked for. But you can’t rely on this to cover many of your cookies. The ICO guidance on the new law (PDF link) says that “this exemption is a narrow one”. It’s certainly not likely to cover cookies for analytics purposes, which is one of the most common uses of cookies – particularly for smaller websites. Implementing the cookie law Even with the May deadline fast approaching, hardly any websites have done anything obvious to comply with the cookie law. The few which have begun seeking permission tend to use messages displayed at the top of the screen or overlaid on the website’s pages. As you’d expect, the ICO website has an opt-in message at the top of every page. And BT has begun displaying a message (right) to explain a bit more about its cookies, although it’s debatable whether this is a proper opt-in. But by and large, most websites are keeping their cards close to their chest. That’s understandable, when you look at the commercial implications: when the ICO added an opt-in message to its website, measured visitors dropped by about 90%. That didn’t mean the site was receiving fewer visitors – it just meant people weren’t opting in to cookies, so the ICO couldn’t track what those people were doing. For sites which rely on accurate visitor data to make money, the implications are significant. Adding an opt-in could leave them at a disadvantage to competitors, which is why there’s a real reluctance on the part of website owners to be the first to move. The bare minimum It’s looking very unlikely that many websites will have implemented a cookie opt-in by 26 May. But that doesn’t mean they won’t be doing anything at all. In fact, the ICO – which is responsible for enforcing the rules – has suggested that the most important thing is to take steps in the right direction. If you can show you’re moving towards full compliance then you’re unlikely to be targeted. As a bare minimum, it’s important to have an understanding of what cookies your website uses and plan how you might implement an opt-in. It’s useful to know exactly what cookies your website uses anyway, and you can do this relatively easily using tools like Bitstorm View Cookies or Attacat Cookie Audit. These will show you what cookies your website creates as you move through the pages. The harder bit is working out how you would create an opt-in function on your site. As we’ve seen, most websites display a message to users – but the tricky bit is creating the back-end logic to ensure cookies are only set once people have given their permission. If you’re not adept with computer code, you might need some help from a web developer to accomplish this. There are some ‘plug and play’ tools available, like Optanon and this tool from Wolf Software (designed for sites using Google Analytics), which could make things easier. Don’t do nothing Although there is still a fair amount of uncertainty around the new law, it’s fair to say that doing nothing is not a good option. In the long-term, people’s attitudes and understanding of cookies may change – and web browsers may include settings to help provide a consistent opt-in. But for now the onus of complying with the law falls very much on individual website operators. And while the chance of being prosecuted may be small (at least to begin with), doing nothing at all is a dangerous move. To start with, review guidance from the ICO (PDF link) and try and establish what cookies your website currently uses. Even Europe doesn’t understand the cookie law That’s the way the cookie crumbles