Testing the Credit Card Landing Page
(the PIZZA test)
Spring 2011
Mission
Improve how successful the credit card page is at converting visitors into Paid Members.
Strategy
The current credit card page was leaving money on the table. It was lacking in effectively communicating to visitors that:
1)It’s easy to create an account and get instant access.
2)Our credit card forms are “Secure” and their information is safe.
3)Paid members get a lot of great content & can customize how they get it.
4)We have a 30-day guarantee for all new accounts.
We designed a new version of the page to meet these needs and ran a split A/B test against the Old Page.
Half of our traffic was shown the Old Page and half of our traffic was shown the New Page. The version that produced the most sales after 6 weeks was viewed as the winner.
The Results
-Paid List, Walkup and Partners will remain steady, matching previous trends.
What we discovered
-FreeList visitors purchase 13.82% more often on the New Page.
-PaidList visitors purchase23.67% less often on the New Page.
- (at 1/3rd the volume of FreeList purchases)
-Walkup visitors and Partners purchase at the same frequency.
-Site Traffic affects Sales Conversion Rate
- During periods of heavy site traffic (Red Alerts) the Old and New pages convert visitors to Paid at the same rate. During times of normal day-to-day traffic the New Page out-performs the Old Page by 17.4%
Ending the Test
-We will begin showing the New Page to 100% of FreeList, Walkup and Partner visitors.
-PaidList visitors will continue to see the Old Page as it currently out-performs the new page.
- This is a Band-Aid and will serve as a temporary solution.
Long term plan
-In an effort to increase sales for our Paid List funnel, a version of the New page should be specked out and tested against our Old Page.
- This will allow us to have one style of landing page for all visitors.
- The split testing infrastructure is already in place so this can be a low-cost / low involvement test.
-Begin testing the New Page following the same methodology that Site Tuners used to give us an 83% lift on our Barrier Page.
- This would follow a true Multivariate (MVT) testing plan
- Low IT involvement is required when setting up the test
(estimated 15 hours)
Supporting Data:
Old Page / 4.6% / 3.0%
New Page / 5.4% / 3.1%
Conversion Rates During Fluctuating Traffic
* During times of high traffic, conversion rates drop due to a higher number of “first time visitors” who do not purchase on their first visit.
Sales & Subscriptions during the Pizza Test
Cash / HeadcountFreelist / New / $137,271 / 1115
Old / $120,589 / 971
PaidList / New / $39,280 / 230
Old / $50,140 / 300
Partner / New / $26,522 / 198
Old / $27,377 / 203
Walkup / New / $64,354 / 354
Old / $64,778 / 352
1