Conducting A/B/C/D tests to uncover 120%+ profit increases
Role: UX / UI Designer Skills: User Experience, Wireframing, Competitive Analysis Time: 2021-2022
What’s the opportunity?
After a major redesign of the Walmart design system, the customer lost access to the add-ons which added protections if items broke. I re-designed the Item Page, but there was still opportunity to consider the customer who didn’t interact with the Protections module.
With the Analytics team, I created tests of the A/B test as well as an A/B/C/D test to understand which placement would produce the highest financial yield.
What’s the challenge?
Walmart’s “Buy Now” or “Add to Cart” buttons sit quite high up in the buy-box traditionally, and if the customer forgets to interact with the add-on module, he/she/they will not have another chance to add insurance to their products until the Cart screen. I worked closely with our business team to create ideas for how to engage the customer and offer them a way to protect their purchases.
What was tested?
The test was between the two screens featured below, A and B.
Screen A was the modal, strictly focusing on on one Walmart Protection Plan offering. Per business rules, we chose to display the lowest price plan, which also happens to bring 60% of the revenue.
Screen B was what we called “An Enhanced PAC” or an enhanced post-add-to-cart screen. In this section, I chiseled out substantial size of the screen, dedicated to the Walmart Protection Plan as well as various services provided by Angi. Similarly, to avoid analysis paralysis, I chose to display only the lowest tier option, and make the decision quick with a large, prominent button for easy adding.
Analytics show that screen A outperformed screen B significantly in mWeb, mobile, and Desktop alike.
What did the iterations look like?
The designs themselves were very dependent on whether the customer interacted with the module on the Item Page. We had to be thoughtful and delicate about giving the customer autonomy and not adding extra frustration if the customer already denied adding protections in any previous steps.
I worked through a lot of decision matrices to capture all of the considerations.
Below are shown Desktop and Mobile considerations-
Impact
The designs are now in production for more than 50% of the population and will allow customers to get add-ons for a variety of items ranging from technology to home-remodeling. There is a significant improvement in Web, iOS, and Android ranging from 5% to 36% conversion rate on various platforms. The analytics of which A/B/C/D option has outperformed its competition is still being tested.
Next steps
The analytics team is looking at the patterns we are observing from both the A/B test and the A/B/C/D test.
Lastly, iterations also included a focus on a different A/B/C/D test, to analyze whether the placement of our modules would have any effect on the bottom line. This test was created in a partnership with the Item Page to understand the conversion rate and the financial gains of not only the Protections module, but also the entire page as a whole.
We considered questions like :
If the Add-Ons module sits prominently, will it then affect fulfillment? Walmart is working to introduce customers to the Shipping feature to compete with Amazon and BestBuy.
Will the placement of Add-Ons affect the sale of the “Ads” section featured right below the Buy Box module?
Will the Add-Ons module affect customers choosing variants of items? Do customers focus less or more on customizations of items? Will customers scroll down further to find the Add-Ons module if there are a lot of variants?
