MEC Retailer's Research Log

December 7 Clinic Notes

NOTES:

We will be adding notes here. Please refresh this page when instructed to do so.

Simple A/B Split Test
	Order Form A	Order Form B
Order Displays	17,598	17,682
Completed Orders	1056	1151
Conversion Rate	6.00%	6.51%

Multivariate Test
	Views	Conversions	Conv. Rate
A1 - Background Color - White	72	29	40.28%
B1 - Call to Action - "Download It Now"	74	29	39.19%
C1 - Drop Shadow - Yes	72	26	36.11%
D1 - Border - No	74	26	35.14%
E1 - Headline - How to Get More Conversions	70	24	34.29%
F1 - Hero Shot - Angled	74	25	33.78%
F2 - Hero Shot - Vertical	72	22	30.56%
E2 - Headline - Hot to Get Better Results	76	23	30.26%
D2 - Border - Yes	72	21	29.17%
C2 - Drop Shadow - No	74	21	28.38%
B2 - Call to Action - Click Here	72	18	25.00%
A2 - Background Color - Opaque	74	18	24.32%

Multivariate Test - Headline Results
Headline	Views	Conversions	Conv. Rate	Benefit %
1	306	62	20.26%	20.39%
2	315	57	18.10%	7.52%
3	306	49	16.01%	- 4.85%
4	299	45	15.05%	- 10.57%
5	303	40	13.20%	- 21.56%
6	313	57	18.21%	8.21%

A/B split testing offers the following advantages:

1. A/B testing is quick and easy to set up. You can think of a new page headline over lunch, implement a test, and see results within a few hours or days.

2. A/B testing offers unambiguous results. If you are testing only a single page variable with two possibilities, the results are typically available quickly and the best course of action is usually quite clear.

3. If you are optimizing pages for a new website, or for a client, you may want to test the existing page against a page containing all of the "best practices" you have discovered for designing landing pages, order pages, etc. The MarketingExperiments.com library of past experiments should be a helpful guide for identifying best practices in each of these areas.

Multivariable testing should be used in different circumstances:

1. It takes some time to set up a multivariable test. You have to design and implement all of the individual page variations at once.

2. However, if you have a number of variations and combinations to test, multivariable testing can save you much time and aggravation in the long run.

JoAnn.com Multivariate Tests
Metric	Improvement
Average Order	137%
Revenue Per Visitor	209%
Sewing Machine Conversions	30%

Multivariate Test
	2 Values Each	3 Values Each	4 Values Each
1 Variable	2 pages	3 pages	4 pages
2 Variables	4 pages	9 pages	16 pages
3 Variables	8 pages	27 pages	64 pages
4 Variables	16 pages	81 pages	256 pages
5 Variables	32 pages	243 pages	1024 pages
6 Variables	64 pages	729 pages	4096 pages
7 Variables	128 pages	2187 pages	16,384 pages
8 Variables	256 pages	6561 pages	65,536 pages

1. A/B split testing is usually better for basic comparative testing, while multivariate testing is better for larger scale optimization. If it is cost-effective for your business, you will have the capability for both and will choose wisely between them depending on your near-term goals. Resist the urge to create large, complex tests if you are just getting started in comparative testing.

2. Although small scale micro-tests can often suggest a superior value for a given variable, the most accurate results require adequate time for a valid sample to accumulate. While the Taguchi Method can be quite effective, results containing any ambiguity should be tested for a longer period before being implemented site-wide.

3. Seasonal trends can affect the outcome of an A/B split or multivariate test if the traffic during that season is markedly different than normal traffic. Likewise, sources of traffic can affect the outcome of testing; if an influx of traffic from an unusual source comes in during an A/B micro-test, it can skew results. This further supports longer testing periods.

4. Prepare to have your intuitive expectations proven wrong. Use your intuition and experience to design good tests, but in a well-designed test with sufficient data, the numbers do not lie.

5. Don't stop refining your marketing after just one test. Continue to optimize over the lifetime of your site. A number of small improvements over time will result in a large improvement and a highly optimized site. See our recent brief on "The Compounding Effect" to read more about this. In addition, the expectations of your customers and the Internet community as a whole may change over time; ongoing testing is the best way to keep your finger on the pulse of your customers.

6. Regarding the number of page views or visitors needed to conduct a reliable test, this can vary widely, and depends primarily the conversion rate of the page itself. A free offer page may convert at 25% or higher, while a product sales page may convert lower than 1%.

   A good rule of thumb is that you must have AT LEAST 10 conversions for each composite page in a multivariate test. A safer figure is to generate 30-50 conversions for each combination.

URLs:

1. Become an MEC Research Partner

2. A/B Split Testing Brief

3. Headline Results (Vertster)

4. Monster.com Control Page

5. Monster.com Optimized Page

6. Vertster

7. Offermatica

8. Optimost

9. Inceptor

10. ATG

11. SiteSpect

12. http://en.wikipedia.org/wiki/Genichi_Taguchi

BIO:

Dr. Flint McGlaughlin

The Marketing Experiments Journal is published by Digital Trust Inc. (DTI). Dr. Flint McGlaughlin is the Director of DTI.

Dr. McGlaughlin is a pioneer in strategy and communications. His primary research is focused on discovering the most effective means to market intellectual assets across the Internet.

Dr. McGlaughlin has advised more than 1500 companies including, AT&T, The Federal Reserve, Pitney Bowes, and Merrill Lynch. He has conducted over 2000 hours of keynote speaking. He has directed the development of more than 20,000 web pages.

Dr. McGlaughlin currently serves as the Co-Executive Producer of the new FOX Family series: "Courage" - hosted by actor Danny Glover. He also serves as a senior advisor or board member to twelve growing companies, and four charitable foundations.

His writing and television production activities have, thus far, won two Tellys, an Edwin Murrough, and an Emmy.

Dr. McGlaughlin is quick to acknowledge that the research at MEC is largely dependent on the intense, focused efforts of the Lab's Leadership Team.

Posted by Brian Alt at December 7, 2005 02:08 PM