Don’t Test Whispers
April 15, 2013 2 Comments
Key to marketing success is a disciplined approach to testing ideas and action. After all, marketing writing and consulting is filled with tremendously attractive and detailed theories about action “X” causing result “Y”. Yet all these theories were developed based on specific experiences under specific circumstances. So there’s no guarantee that taking them and applying them in your world will create the same result.
So we should test, test test. And yet…testing experience shows that far more things are tested than are found to conclusively help or hurt. Why? One quite common testing error is to “test whispers” – small changes that simply can’t have a large enough impact to drive measurable change.
I once watched Rubbermaid test whispers in focus groups where a series of 5 statements of brand differentiation were evaluated. But rather than vary the statements with ideas that were truly significant to consumers, the statements traded off tiny wording changes. (I found it ironically enjoyable to watch the focus group participants quite frankly explain that all the statements said the same thing.)
Focusing on testing important things is even more critical in retail markets. In part, a mature market is extraordinarily noisy. And changes in the retail world are never isolated enough to be perfectly separated from other actions – often actions we might not even be aware of like promotional choices made by a local retail store manager.
In part, the statistical variations in normal day-to-day activity are quite often far larger than the size of change we’re hoping to test. We hired a statistician to evaluate some results a decade ago or so for a tool client who sold through Sears. The size of change that retail merchandising efforts produced (e.g. appearing in the Sunday circulars) far outpaced any of the changes we’d see from advertising spending (which was a long-term & constant effort). The statistician identified impact from the advertising, but in order to damp out the impact of the merchandising efforts he could say no more than “there was impact” because he had lost all valid ability to quantify the size of the impact. (Reminded me of my qualitative vs. quantitative chemistry work in school…)
One might think testing becomes far different when we shift to direct response marketing. It doesn’t. Even in DR one has to be careful to test actions that are significant enough to generate a change you can measure with confidence.
So read on in my direct response television based article from this month’s Response Magazine. (Link here.)
Testing offers tremendous organizational advantage – especially responding to political attack with knowledge gained from testing. So embrace testing. And focus your testing efforts where you can successfully evaluate the test results.
Copyright 2013 – Doug Garnett – All Rights Reserved