Testing site speed
We all know there’s an important correlation between site speed and important KPIs like conversion and customer satisfaction. We’re told this repeatedly – speed has been a ranking factor in Google search for over ten years precisely because it’s so important to users.
The problem is, within businesses, it can be very hard to get traction for improving site speed:
- Inertia is too great (improving site speed can be an involved, dev-heavy and expensive)
- Driving factors are too abstract (there are studies, plus of course Google tells you to do it, but it’s hard to quantify what site speed really means to your customer base and your bottom line)
- Results are hard to prove (improving site speed globally – yes, you can compare before and after the changes, but invariably other factors cloud the results and other initiatives claim a share of the benefit, as is the case with any before-after measure)
AB test site speed
The obvious choice is to AB test your site speed. This article on the Web Dev website gives interesting options (and you will need to adopt some of the advice in this article in order to effectively measure page speed in the first place). However, the choices given here don’t really solve the problem of getting traction for me. They are:
- Create a faster variant of a page to AB test (unappealing because it is more involved, dev-heavy and expensive then we’d like an AB test to be)
- Improve page load speed by pre-fetching etc (same issue as above, and also can slow down the initial landing page, which you’d then also need to measure)
- Artificially slow a page down (easier to achieve, but unappealing because you have to choose an important page on the site and, laudable though the quest for knowledge is, intentionally creating a test variant that you know is going to depress conversions is unlikely to gain many fans)
An alternative – AB test tags
An alternative solution, that Loop Horizon recently deployed with one of our clients, was to AB test tag configurations instead.
Tag speed analysis
The first thing we did was analyse the performance of tags on the site, to understand which tags to go after.
Firstly, not all tags would be appropriate to remove for an arbitrary section of users (e.g. those needed for audience creation or handling parts of the user experience), so we could count those out up front.
Secondly, to have a chance of having the biggest impact, we needed to target tags that not only load slowly, but that load for a large number of users. So; slow and loads loads.
To do this analysis, we wrote some script that pushes tag names and load speeds into a string to be collected. If you’re an Adobe customer, this is available as an extension in Adobe Launch (TAGS by Loop Horizon) – though the code will work in any tag manager – and can be pushed into a list variable in Adobe Analytics.
The AB tag test
Having identified a tag to go after – in this instance; for a tool that was slow, loaded on every page, was only used by one team within the business and could handle sampled data with little impact – we used the tag manager to turn this tag off for a percentage of users and pass the variant data to analytics.
When we analysed that data, we saw a significant uplift in conversions for the test group.
This approach has a few key advantages:
- While it is somewhat technical, it can be delivered entirely via the tag manager (Tealium, Adobe Launch, Google Tag Manager etc), meaning whichever team owns and manages the tag manager can handle the almost the entire process
- It does not require involved, dev-heavy and expensive re-working of pages
- It only intends to speed the site up, so you don’t have to explain why you’re intentionally slowing important parts of the site down in a data gathering exercise
- It can be deployed across the entire site, so even small improvements in conversion can be quite impactful
- Even if it fails to achieve significance, the data still has value. It answers questions about the impact of tags and where site speed improvement efforts should be focused
- It points to a more sensible approach to adding new tags to the site – one in which you can quantify the impact of adding a tag against the benefits of the tool
- It can provide a compelling case for migrating tags to a server-side tag management system