MEC Retailer's Research Log

December 21 Clinic Notes

NOTES:

We will be adding notes here. Please refresh this page when instructed to do so.

A/B/C Split Test - Survey
	Page A	Page B	Page C
Total Votes	22	51	103
Percentage of Votes	12.50%	28.98%	58.52%

A/B/C Split Test - Actual Results
	Page A	Page B	Page C
Percent of Traffic	34%	33%	33%
New Sales	244	282	114
Change	N/A	15.57%	- 53.28%

JoAnn.com Multivariate Tests
Metric	Improvement
Average Order	137%
Revenue Per Visitor	209%
Sewing Machine Conversions	30%

Keywords Tested:

B.C.
Foxtrot
Ziggy
boondocks comic
calvin & hobbes
calvin hobbes
cartoon comic
comic art
comic page
comic store
comic strip
comics
doonesbury
free comic
funny comic
garfield
newspaper comic
online comic
sunday comic strip
web comic
webcomic
yahoo comic
"Andy Capp"
"BC Comic"
"Dog eat Doug"
"For better or for worse"
"bloom county"

Google AdWords CTR Test - Survey
Headline	Total Votes	Percentage of Votes
ReadComics Comics	270	20.80%
DailyComics Comics	790	60.86%
GoComics Comics	155	11.94%
uComics Comics	83	6.39%

Google AdWords CTR Test - Actual Results
Headline	Click-Through Rate
ReadComics Comics	0.9%
DailyComics Comics	1.2%
GoComics Comics	0.9%
uComics Comics	0.8%

Email Subject Line Test - Survey
Subject Line	Total Votes	Percentage of Votes
Test Your Marketing Intuition	268	33.38%
An MEC Invitation: Test Your Marketing Skills	138	17.31%
So You Think You're A Marketing Expert? Prove it here...	393	49.31%

Email Subject Line Test - Actual Results
Subject Line	Open Rate	CTR
Test Your Marketing Intuition	11.99%	3.34%
An MEC Invitation: Test Your Marketing Skills	12.59%	2.93%
So You Think You’re A Marketing Expert? Prove it here...	11.46%	2.90%

1. Intuition can be effective in determining what needs to be tested. It can provide the insights you need to decide which general direction to explore.

2. Intuition may fail you when you may attempt to predict, which of those tests may prove successful. Depending on intuition to select one of four possible email subject lines, for instance, it not a reliable way to maximize your chances of achieving the best open rate.

3. Intuition can be used, sometimes, to dig deeper down in to your test results. Sometimes it can help you glean an insight as to why the market behaved in a particular way.

4. Intuition typically fails miserably when applied to initial test results. In the beginning stages of the tests the conclusions may be seriously flawed. You have to wait until you have a large enough sample. The danger is particularly real when provisional results happen to match your own intuitive opinion.

5. A mistake many marketers make is to take the initial findings of the test, make an intuitive determination as to which is best, and stop the experiment. More often than not, this leads to a bad decision.

6. Intuition is often limited, in that it depends on previous experiences. It depends on pattern recognition. Because of this you may make natural assumptions that are contrary to absolute changes in the context, or to changing conditions in the marketplace.

URLs:

1. Become an MEC Research Partner

2. Email Subject Line Survey

3. Split Test - Page A

4. Split Test - Page B

5. Split Test - Page C

6. Transparent Marketing

BIO:

Dr. Flint McGlaughlin

The Marketing Experiments Journal is published by Digital Trust Inc. (DTI). Dr. Flint McGlaughlin is the Director of DTI.

Dr. McGlaughlin is a pioneer in strategy and communications. His primary research is focused on discovering the most effective means to market intellectual assets across the Internet.

Dr. McGlaughlin has advised more than 1500 companies including, AT&T, The Federal Reserve, Pitney Bowes, and Merrill Lynch. He has conducted over 2000 hours of keynote speaking. He has directed the development of more than 20,000 web pages.

Dr. McGlaughlin currently serves as the Co-Executive Producer of the new FOX Family series: "Courage" - hosted by actor Danny Glover. He also serves as a senior advisor or board member to twelve growing companies, and four charitable foundations.

His writing and television production activities have, thus far, won two Tellys, an Edwin Murrough, and an Emmy.

Dr. McGlaughlin is quick to acknowledge that the research at MEC is largely dependent on the intense, focused efforts of the Lab's Leadership Team.

December 7 Clinic Notes

NOTES:

We will be adding notes here. Please refresh this page when instructed to do so.

Simple A/B Split Test
	Order Form A	Order Form B
Order Displays	17,598	17,682
Completed Orders	1056	1151
Conversion Rate	6.00%	6.51%

Multivariate Test
	Views	Conversions	Conv. Rate
A1 - Background Color - White	72	29	40.28%
B1 - Call to Action - "Download It Now"	74	29	39.19%
C1 - Drop Shadow - Yes	72	26	36.11%
D1 - Border - No	74	26	35.14%
E1 - Headline - How to Get More Conversions	70	24	34.29%
F1 - Hero Shot - Angled	74	25	33.78%
F2 - Hero Shot - Vertical	72	22	30.56%
E2 - Headline - Hot to Get Better Results	76	23	30.26%
D2 - Border - Yes	72	21	29.17%
C2 - Drop Shadow - No	74	21	28.38%
B2 - Call to Action - Click Here	72	18	25.00%
A2 - Background Color - Opaque	74	18	24.32%

Multivariate Test - Headline Results
Headline	Views	Conversions	Conv. Rate	Benefit %
1	306	62	20.26%	20.39%
2	315	57	18.10%	7.52%
3	306	49	16.01%	- 4.85%
4	299	45	15.05%	- 10.57%
5	303	40	13.20%	- 21.56%
6	313	57	18.21%	8.21%

A/B split testing offers the following advantages:

1. A/B testing is quick and easy to set up. You can think of a new page headline over lunch, implement a test, and see results within a few hours or days.

2. A/B testing offers unambiguous results. If you are testing only a single page variable with two possibilities, the results are typically available quickly and the best course of action is usually quite clear.

3. If you are optimizing pages for a new website, or for a client, you may want to test the existing page against a page containing all of the "best practices" you have discovered for designing landing pages, order pages, etc. The MarketingExperiments.com library of past experiments should be a helpful guide for identifying best practices in each of these areas.

Multivariable testing should be used in different circumstances:

1. It takes some time to set up a multivariable test. You have to design and implement all of the individual page variations at once.

2. However, if you have a number of variations and combinations to test, multivariable testing can save you much time and aggravation in the long run.

JoAnn.com Multivariate Tests
Metric	Improvement
Average Order	137%
Revenue Per Visitor	209%
Sewing Machine Conversions	30%

Multivariate Test
	2 Values Each	3 Values Each	4 Values Each
1 Variable	2 pages	3 pages	4 pages
2 Variables	4 pages	9 pages	16 pages
3 Variables	8 pages	27 pages	64 pages
4 Variables	16 pages	81 pages	256 pages
5 Variables	32 pages	243 pages	1024 pages
6 Variables	64 pages	729 pages	4096 pages
7 Variables	128 pages	2187 pages	16,384 pages
8 Variables	256 pages	6561 pages	65,536 pages

1. A/B split testing is usually better for basic comparative testing, while multivariate testing is better for larger scale optimization. If it is cost-effective for your business, you will have the capability for both and will choose wisely between them depending on your near-term goals. Resist the urge to create large, complex tests if you are just getting started in comparative testing.

2. Although small scale micro-tests can often suggest a superior value for a given variable, the most accurate results require adequate time for a valid sample to accumulate. While the Taguchi Method can be quite effective, results containing any ambiguity should be tested for a longer period before being implemented site-wide.

3. Seasonal trends can affect the outcome of an A/B split or multivariate test if the traffic during that season is markedly different than normal traffic. Likewise, sources of traffic can affect the outcome of testing; if an influx of traffic from an unusual source comes in during an A/B micro-test, it can skew results. This further supports longer testing periods.

4. Prepare to have your intuitive expectations proven wrong. Use your intuition and experience to design good tests, but in a well-designed test with sufficient data, the numbers do not lie.

5. Don't stop refining your marketing after just one test. Continue to optimize over the lifetime of your site. A number of small improvements over time will result in a large improvement and a highly optimized site. See our recent brief on "The Compounding Effect" to read more about this. In addition, the expectations of your customers and the Internet community as a whole may change over time; ongoing testing is the best way to keep your finger on the pulse of your customers.

6. Regarding the number of page views or visitors needed to conduct a reliable test, this can vary widely, and depends primarily the conversion rate of the page itself. A free offer page may convert at 25% or higher, while a product sales page may convert lower than 1%.

   A good rule of thumb is that you must have AT LEAST 10 conversions for each composite page in a multivariate test. A safer figure is to generate 30-50 conversions for each combination.

URLs:

1. Become an MEC Research Partner

2. A/B Split Testing Brief

3. Headline Results (Vertster)

4. Monster.com Control Page

5. Monster.com Optimized Page

6. Vertster

7. Offermatica

8. Optimost

9. Inceptor

10. ATG

11. SiteSpect

12. http://en.wikipedia.org/wiki/Genichi_Taguchi

BIO:

Dr. Flint McGlaughlin

The Marketing Experiments Journal is published by Digital Trust Inc. (DTI). Dr. Flint McGlaughlin is the Director of DTI.

Dr. McGlaughlin is a pioneer in strategy and communications. His primary research is focused on discovering the most effective means to market intellectual assets across the Internet.

Dr. McGlaughlin has advised more than 1500 companies including, AT&T, The Federal Reserve, Pitney Bowes, and Merrill Lynch. He has conducted over 2000 hours of keynote speaking. He has directed the development of more than 20,000 web pages.

Dr. McGlaughlin currently serves as the Co-Executive Producer of the new FOX Family series: "Courage" - hosted by actor Danny Glover. He also serves as a senior advisor or board member to twelve growing companies, and four charitable foundations.

His writing and television production activities have, thus far, won two Tellys, an Edwin Murrough, and an Emmy.

Dr. McGlaughlin is quick to acknowledge that the research at MEC is largely dependent on the intense, focused efforts of the Lab's Leadership Team.