We receive a lot of questions about features, UI elements (user interface elements), or wording that seemingly appear, change, or disappear “overnight” on our websites.
While it may seem random, we use the scientific method as a basis for building and determining changes, new enhancements, and features. We use A/B testing as a key experimental process to validate our hypotheses.
There are a few reasons why we’re not able to communicate every A/B test.
- Testing is heavily reliant on achieving a successful result
- Sometimes it requires several iterations to get right
- Many tests will not be successful and would not make it as a permanent addition to the website.
Since we’re not going to be able to provide you with details and the how/why for the hundreds of test that we’re doing, we can give you the background and underlying insights into how we approach changes with our test-and-learn culture.
We want you to be able to better navigate the changes you see and try to understand why you might be seeing what you do.
Ultimately, you could apply these concepts to improving your own property listing to get more bookings and provide feedback to us about test or testing ideas you think we should try.
View this short video where a HomeAway Product Manager provides a high-level overview of our test-and-learn culture.
Let's go through an overview of the process discussed in the video above and touch on some key points. Afterward, we'll walk through some examples of test we've done in the past.
We don't come up with ideas out of the blue. We base them on what we observe through analyzing data from across our websites, past test, talking with partners, and through various types of research studies, we conduct with users.
short version: identify the problem
Usability could be rolled up into the observation section above but we wanted to break this out to spend a little time on it. We have dedicated usability labs where we watch and learn from real users.
This could be watching the steps a partner goes through when responding to a booking request through Reservation Manager or to see how they would react to different designs and feature concepts we want to validate before dedicating time to building it.
HomeAway is also learning a great deal from Expedia on taking our research to the next level. Here is a great article by Bloomberg on Expedia's usability labs, which will give you insight into how it works at HomeAway too.
short version: get real feedback from users (partners and travelers)
Stating the definition: A supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation.
A simpler way of saying this: You have a possible explanation of why something you observed is happening. You can then propose a change (i.e. explanation) which you can measurably test to see if it is true.
Here's a silly example using the concepts above.
You're sitting across from the condiments station at a coffee shop drinking your routine morning coffee. You notice that there are two boxes next to each other. The left box has small plastic straws and the right box has biodegradable wood stirring sticks. As an eco-conscious person, you want more people to use the wood sticks since it would be better for the environment.
- You noticed 25% of the time (1 out of 4) customers picked the wood sticks to stir their drinks.
- You also noticed the box with plastic straws is closest to the coffee pickup station and the sugar box is next to the plastic straw box
- Switching the placement of the boxes on the condiment station so that the wood stirring sticks are closer to the coffee pickup station will increase the % of customers using the wood stirring sticks (i.e. a small win for Earth)
An A/B test is a controlled experiment that splits users between a control and a variation (or multiple variations). The key goal is to validate whether the hypothesis is right or not.
There are several components to think about when putting together an A/B test.
- The “control”: this is the current state of an object. Example, the request to book button on the property pages are orange.
- The “variation”: this represents the one change to make. Example, make the request to book button green. Also, you can have several different variations.
- The “goal” or “metric”: this is important as it determines if the test is a success or failure. Example, will more travelers click the green button and does it lead to more booking requests.
We would then send a random set of travelers to each variation and measure what happens. Example: there are 100 travelers. We show the “control” version to 50% of the travelers so we have a baseline number. We show the “variation” version to the other 50% of travelers. The key point is that this is done randomly so any bias is removed. Another common example you might be familiar with is the cola “blind taste test” – which taste better.
NOTE: this is a grossly simplified explanation. We’re testing against millions of travelers on a daily basis. There is quite a bit of statistical analysis done on the data we get in order to develop results for our test. It’s out of scope for this blog post but if you’d like to read more on general statistics here is another Wikipedia article with the basics.
Going back to our example, let’s say the data showed the following (assume, statistical significance).:
- 50 travelers who saw the control version, 10% of them requested a booking.
- 50 travelers who saw the variation version, 15% of them requested a booking.
So in this example, the variation was a positive outcome. Travelers who see a green request to book button, submit more booking request.
In this example, we would choose to launch the change live to 100% of the site and it becomes a permanent change.
Let’s suppose the result came back negatively. For example, the variation version showed that only 5% of travelers requested a booking. In this instance, we would have stopped the test and evaluated why the variation might have failed.
The cycle would repeat.
- Is there any new observation we could see? Or is there any other feedback we can gather?
- Develop a new hypothesis
- Set up a new test to validate our new hypothesis
- See the results
We hope this blog post provides you a bit more insight into HomeAway’s test-and-learn culture.
Take a couple minutes and leave us feedback and suggestions here.
Here are some real test we’ve done recently.
TEST 1 - Increase the number of listings displayed on the search results page
- We see that travelers view many properties when searching.
- We currently display 30 properties per page
Hypothesis: Changing the number of results will make it easier to browse and compare listings, improve property page views, increase booking request.
- Control: 30 results
- Variation 1: 18 results
- Variation 2: 50 results
Results: V2 won. Launched across all websites.
TEST 2 - Adding an image carousel to the search results page
- Many sites allow image scrolling and previews on the search results page
- Our current implementation buries other images under the magnifying glass, not very intuitive or easy to use
Hypothesis: Adding this feature to the search page will encourage more image browsing from search results page and allow travelers to quickly find properties that interest them and improve booking request.
- Control: No image carousel
- Variation 1: Image carousel
Results: Control won. Gather new observations. Reevaluate hypothesis. Develop new test.