A/B Testing Can Validate Your Price Increases Safely

By A/B testing a price change, you can ensure that users don't drop you or your product.

Jan 17, 2024

Test Don’t Guess

Often when working with advising clients, I ask questions that the company doesn’t know how to answer or doesn’t have data to be able to answer.

Today’s newsletter is focused on illustrating the important role that user data and testing should play in a growth strategy.

If your brand is trying to grow, scale or make more with less, this simple example is for you.

The Pricing Test

I met with a client recently to talk about growing their revenue when a concern about pricing came up.

The client said that they needed to raise prices to match business costs due to inflation but were afraid that the price change would result in fewer sales and potentially upset users.

So together we set out to identify how we would test a potential price increase and determine the effect it would have on users.

In order to setup our test we needed the following:

Baseline data
A hypothesis
Test setup
A way to measure results

Baseline Data

The first step to solving the pricing problem is collecting baseline data.

The reason we collect baseline data is to understand what the current situation is. In this case we were looking for what the demand is for the various product offerings so we can determine which product should be tested.

Once we had identified the most popular product (which also happened to be the product that offered the best experience) we started constructing our pricing test.

Disclaimer - when testing pricing, be careful. Pricing is an easy place to make a stupid testing mistake and make some users very upset.

Once we knew which product sold the most, we had as much baseline data as was available.

I would have liked to have had more, but my client isn’t a data scientist and doesn’t have incredible data, yet.

The reality is in business experimentation you’ll likely never have enough data to feel 100% comfortable. I try to collect as much data as is available and work with what I have.

Hypothesis

For our pricing test, constructing a hypothesis was simple, here it is:

If we change the price of the product, then sales will remain the same.

When you’re raising your prices on a product, the best case scenario is that sales stay the same. It’s unlikely that sales would jump, but if sales decline, raising the prices might not be the right move. So our goal was to keep sales flat.

Test Setup

With our baseline data and a hypothesis to work with, it was time to setup the test.

My client’s budget for this project was basically non-existent, so I opted to use a testing tool called VWO. My client falls within their free tier, so we now had a free testing tool to do most of the heavy lifting for us.

To set up the test I created two variations, one control cell with no changes to the price, and one variant cell, where the price change has been applied.

The testing platform collects unique visitors and click data, but as a backup, and as a way to have clearer attribution for other marketing efforts, I made one small change to the product page URL.

I added a parameter to the url with this info “?price=new-price”. With that URL parameter in place I also added a parameter to the control cell that said “?price=old-price”. Once these parameters were in place I added a hidden field to the checkout page that automatically collected the product page URL and therefore the parameters I added.

This gives me secondary data to use to correlate with VWO and their test results, but it also gives me information about which variation users saw and which variation led them to checkout.

Now to run the test.

Running The Test and Reaching Statistical Significance

In order to make a data-based decision, I knew I needed to collect statistically significant results. There are plenty of tools out there to help you identify how you can reach statistical significance.

In this case here are the numbers I used to make my determination:

15% Unique Visitor to Purchase CVR

20% Minimum Detectible Effect

95% Statistical threshold for concluding the test

With these as my inputs to achieve statistical significance, I was able to determine that we would need around 1,600 unique visitors to feel confident that the data we are seeing is reflective of most users.

In most tests I run, I’m hoping to get about 1,000 users in each cell. There are some exceptions to that rule, but that’s a good place to start, especially if you have low traffic.

Results

Once the test concluded and reached statistical significance, it was time to analyze the results.

We were able to identify (statistically significantly) that those who saw the higher price converted at around 16% (basically a flat result), while those who saw the lower price were converting at 13%.

This data made it very clear that not only was the price change driving more revenue for my client, but for some reason when users saw the higher price, they converted slightly more.

There are plenty of reasons that could contribute to this, but my guess is that by raising the price just a little, we are sending a message to users about the quality of the product. There is a psychological connection between paying more for something and getting more perceived value from it, and I think that’s what’s happening here.

Conclusion: Testing A Price Change Can Lead To More Revenue

Not all pricing tests go this way, there are several variables to consider. In this case, I think my client is undercharging compared to competitors but still offers the best product in the market. Raising prices doesn’t come as an offense to these users, but it comes as a vote of confidence.

We’re saying “By raising our prices, we can continue to offer the best product on the market and make a little more cash to continue to provide a better experience.”

🧑‍🔬 Do You Want To Run A Pricing Test? 🧑‍🔬

If you want to make a change to your prices, convert better or understand your users better, let’s talk about running some tests together.

Experimentation Labs

Discussion about this post