The Research Playbook

Archive for November, 2009

Written by Carey V.  Azzara

As previously stated a poor questionnaire design is one such mistake, but not the only one. Good solid sample development is also necessary. Here is another Research Axiom worth your consideration.

You can never fully recover from a sample that lacks validity; and once again:
No amount of analysis, regardless of how brilliant
No degree of insightful interpretation, regardless of intellectual prowess
No manipulation of the variables, regardless of how cleverly done
Nothing can save you from a poorly developed sample!

The value of sample development is underappreciated, as are the skills related to creating a valid sample. Project managers, research analysts, and all those who lose sleep over the quality of the sample sources they have available and who work hard to provide the best possible sample for each research project they conduct, are worth their weight in gold.

With numerous challenges to good sample development always hovering over us, if the research team conducting the study does not pay close attention to this critically important task the chances of deriving useful results are likely to diminish rapidly.

One of the worst situations to be in, is standing in front of a room full of executives and presenting the research implications when from off in the far corner an EVP asks you, “Are you sure about that finding? Who were these respondents? They don’t appear to have any knowledge about the market or our products.”

If you can definitively reply, “We believe the respondents in this sample are qualified” and give a crisp response about the QC steps used to verify the validity of the sample, you’ve saved the day.
If on the other hand, you hesitate and cannot defend the validity of the sample, you’ve lost your audience – there is nothing more they want to hear from you because the voice of the respondents do not reflect the people they are trying to reach – the day ends badly.

If you don’t care about the quality of the research you conduct, well shame on you, but at least recognize that a sample of good quality is a necessity for self-preservation – enough said.

Written by Carey V.  Azzara

Results from a recent AtHeath study Market Research Customer and Prospect Requirements (N=253 purchasers of market research) sheds light on this question. Moreover, the answer to the question might surprise you. If you guessed free content and/or access to blogs, you would be wrong!

Access to blogs and other regularly updated content, Ask the Analyst, or free downloads are not the features valued most. What people really care about are, Accurate information, Easy navigation, and Robust search capabilities. Yes the basics. Get that right and then you can add, Details on report content, Pricing information, and Access to content purchased.

Once you have accomplished all the items on these two short lists you can play with free content, blogs, and ask the expert features.

Why am I telling you this? Simply because we all get caught up in trying to impress our web users with the newest stuff available and in the process sometimes lose our way. Consider this as a simple gut check – make sure you are covering the basics very well before you worry about the nice to have items.

This is not only true for websites it’s true for all aspects of business and certainly for conducting market research. Until you achieve basic milestones, such as “developing and refining project objectives” all the “fancy stuff” doesn’t really matter.

Written by Carey V.  Azzara

Apparently many people do and with good reason.

We recently posted a poll on LinkedIn. The question posed was:
“When you select a sample provider, how much importance do you place on their panel recruitment process?” Is it:

1. Not at all important
2. Somewhat important
3. Important
4. Very important
5. Extremely important

While the results derived are from a sample of convenience and hardly scientific, they are nevertheless instructional. So what did the 91 people (as of 11-11-09) who took time to participate tell us?

  1%  Not at all important
13%  Somewhat important
16%  Important
37%  Very important
31%  Extremely important

If you are a panel provider and you didn’t already know that recruitment practices play a deciding factor, it surely would be obvious now, with nearly seven out of ten (68%) prospective buyers voting “very and extremely” important.

Women are more likely to view recruitment as very or extremely important (72%) than men (57%) are. Differences by age were interesting. There was a very small portion of the sample (N~6)
under the age of 35 and none of these younger folks voted extremely important. The 35-54 age cohort, which was the largest age group in the sample, had a high proportion of votes in the extremely important category (27%) and another 41% who voted very important. The oldest group (55+) did not share this concern, only 17% said very important and the rest said recruiting was only important or somewhat important.

Job function seemed to be related to this issue. Academics, perhaps not surprisingly, all voted extremely important (100% N~18). Business development folks were at the opposite end of the spectrum with 100% (N~18) voting somewhat important. Technical professionals (N~18) all (100%) said that recruitment was important. Marketing professional (N~36) were split 50% very important and 50% important.

What can we learn from examining the results of this poll? I think the message is straightforward. Our teachers (academics) would want us to be the best we can be – they set the standard. Women are more likely to hear that message than are men. The people in marketing who depend on data to make decisions were also more likely to place a higher value on recruitment, which is of course a quality indicator. Those who have less immediately at stake, such as business development folks, seem to be less concerned. However, overall this simple polling question seems to have hit a nerve.

We believe that sample development is one of the cornerstones of good research. Market research is not an academic exercise – real business decisions are made, or at least influenced, by the results of the research we conduct – how can you make a good business decision if the sample is faulty? Simply put, you can’t.

If you would like to add your thoughts on this topic feel free to go to the LinkedIn URL and vote.

 http://polls.linkedin.com/poll-results/63972/zvphd

Written by Carey V.  Azzara

Research Axiom: You can never fully recover from a poorly written questionnaire.

• No manipulation of the variables, regardless of how cleverly done
• No amount of analysis, regardless of how brilliant
• No degree of insightful interpretation, regardless of intellectual prowess

Nothing can save you from a poor research foundation. The building will collapse like a house of cards!

If there is one part of the research process that I know, it is questionnaire design. It is a task repeatedly given insufficient time and attention. Clients and research professional alike often underestimate the time it will take to develop a truly well structured and concise instrument.

What amazes me most is when this task is somehow relegated to a status depicted by the attitude of:  “Once the questionnaire is done we can get on with the important stuff, like analysis and reporting.” The assumption that analysis work is the essence of the research and the expectation that interpreting the results is where the mastery of research ultimately lies is a mystery to me.

Have we not pounded the concept of garbage-in garbage-out into our heads? Can new internet tools substitute for critically thinking and the hard work of aligning the research instrument to the purpose of the study – answering the business questions that sponsors paid to learn?

If this seems like a bit of a rant, well I guess I’m guilty. My own research on research including the use of a 25 point questionnaire audit system has shown me that even well healed researchers are less diligent about quality than one would hope. Research is not only science it is a craft [perhaps an art] and if the proper fundamentals are not applied the product is less than artful.

I’ll end the ranting with an analogy [but don’t be surprised to hear more on this topic]. If you have not studied and then practiced writing poetry would you expect to publish a book of poems simply because your company’s marketing department asked you to? Designing a good quality research instrument probably takes less talent than being a good poet, but it’s close.

In the days, weeks, months, and years to come we will post content primarily on the subject of market research.  However, we define the scope of topics under the heading market research very broadly. It will include topics such as the role of social media in research and extend the scope of market research to topics such as web analytics and optimization.

This blog is supported by AtHeath, LLC and its expert community. You can find us at www.atheath.com.

We look forward to a lively discussion!

Carey V. Azzara, Founder AtHeath, LLC