The Research Playbook

Archive for March, 2010

The use of matrix questions that could scare a respondent into dropping out is counterproductive. Yet matrix questions are in frequent use and written with less care than you would write an email.

These large and overwhelming questions are generally created to collect a large number of data points in a relatively small amount of space. The rationale is often a misguided notion that the number of questions will be fewer and therefore this is a good and acceptable approach. Nothing could be further from the truth.

Large matrix questions place a huge burden on respondents and in worst-case scenarios, (I have seen this more than once) there are multiple matrix questions in one questionnaire. One case in point had five such questions back to back each with over twenty rows and three to five columns with drop-down menus that asked for scale scores related to satisfaction or capabilities. Those five questions attempted to collect over 400 data points.

How many respondents do you think finished, and of those who did, how many provided thoughtful answers for each item? There is no way to know for sure. However, I’m guessing (and you probably are too) the odds were not in the researcher’s favor.

Much simpler matrix questions than the example given above have proven difficult to execute successfully. It is never about what you want; it’s always about what the respondent can and is willing to provide.

Remember; if you make the respondent’s task impossible or extremely difficult, (at the risk of sounding biblical), you will reap what you sow!

	Join Our Email List

This post is a revised passage from the new book Questionnaire Design for Business Research, now available!  Click on this link:  http://bit.ly/Questionnaire_Design

And, I have a Special Offer perfect for anyone serious about:
 •  Raising the bar on questionnaire design in his or her organization
 •  Finding a cost-effective way to start designing questionnaires
 •  Preparing for a new market research project
 •  Improving his or her research skills
http://bit.ly/Q_DesignSpecialOffer

It is essential that your sample be composed of people who are authorities on the topics and issues you are trying to understand better. Screening questions are a necessary step to accomplish the goal of recruiting appropriate respondents. The respondent must have the level of responsibility you desire and knowledge of your topic. You wouldn’t rely on an investment tip from a bank teller so don’t ask line managers to tell you what keeps CEOs up at night.  

Learn to think in terms of who you want “In” and who you want to keep “Out” of the study. Create screening questions that “Find” and “Exclude” people or organizations (companies) to meet the research design requirements. In addition, think of ways to create qualifying questions to help you improve the screening process, these questions must be crafted carefully to avoid annoying qualified respondents or worse insulting them.  

Finding the right people is an important step in the research process and it begins with using the best available sample frame. The sample source(s) (or sample frames) you use will have a significant impact on the quality of your data. Verifying that your sample is composed of people who are who they say they are and who have the knowledge to respond to your questions is obviously critical.  

Sampling Essentials

Like a poor questionnaire, there is no way to recover fully from a poor sample. Mountains of fancy graphical presentation will not compensate for a sample that lacks credibility. When developing your sample, carefully establish objectives for the number of target respondents, and develop a specific and focused profile of your target respondent(s).  

One of the worst situations you will encounter is standing in front of a room of stakeholders presenting  research results and getting this question from a SVP sitting in the back of the room, “Are you certain these results are accurate? How did you find these respondents? They don’t seem to have any idea of what the market or our products are all about.”  

If you hesitate and cannot defend the validity of the sample, you’ve lost the audience – they will not be interested in anything else you have to say because the voice of the respondents does not parallel who they want to reach – the day will end badly.  

If on the other hand, you reply with a definitive response, “We believe the respondents in this sample are qualified, let me tell you why.” Then give a crisp description of the screening steps taken to verify the validity of the sample – you’ve saved the day!  

Please tell a friend about this blog. Thanks! http://researchplaybook.wordpress.com/ 

	Join Our Email List

An interesting data collection option is using a hybrid approach, such as the telephone-to-web method, which combines telephone recruiting and screening with a URL for data collection. Another hybrid approach is to allow respondents to choose between a web-based instrument and a telephone interview. Only studies that can be conducted equally well with both methods should attempt this approach.

The telephone-to-web hybrid approach leverages the advantages of both data collection methods. It brings a level of control to the web-based method that is otherwise minimal. It allows researchers to use lists not constructed for web interaction, and it maximizes your ability to fill quotas. In addition, when the benefits of a web approach outweigh the telephone method, the combination allows the research team to use the best of both worlds.

Unfortunately, in practice this best of both worlds approach does not often materialize. The reason is simple: Although telephone recruiting is used for all its advantages, in the end, the percent of respondents that complete the instrument is often too low to justify the cost.

This was not always the case, but the percent of telephone-recruited, web-based completes has been dropping over the last several years. What once yielded good results at a competitive price point (especially for larger studies) now may cost more than a telephone survey.

Completion ratios of 50-60% or higher (i.e., 1 in 2 or more recruited telephone respondents complete the study) are typically cost-effective. However, ratios of 1 complete per 3 recruited respondents or worse a 1 to 4 ratio, which is not uncommon, are not tenable. This has caused a significant decrease in the use of this approach and new approaches have not been forthcoming to take the place of this hybrid method. However, some companies are experimenting with improved recruitment methods and, on occasion, this approach yields cost-effective results.

Ultimately, the research requirements should be the most important consideration when selecting a data collection methodology. A detailed evaluation of the strengths and weaknesses of each approach in relation to the goals of the study will provide the best basis for that decision.

Summary
In many of our posts we discuss the key issues researchers must address as they scope out a study and define the data collection approach to achieve high quality results. This post presents one more possible data collect option. It does not provide an in-depth treatment of the issues. However, it does provide a quick overview to help readers understand how the hybrid approach to data collection could optimize a study design.

	Join Our Email List

Can you simply listen in on social media network discussions to gain insights? Do you have to do more than casual listening? How many times have you heard someone talk about “listening” to customers (and less often prospects) as one of the opportunities that social media has created? Are you prepared to glean insights from customer feedback? 

Not Listening!

 

Do you think of it as an opportunity or as a nightmare? “OMG do I have to worry about what everyone is saying about my company on Twitter, Facebook, and a bunch of blogs?” Followed by the related question, “If I do listen should I let customers know I‘m listening?” 

Listening?

 

Are you asking yourself these questions? In addition, are you asking yourself, “Is just listening enough?” What more does the marketplace demand of me to thrive in business? How can a business find resources for an activity that is still, well unproven? 

Are you overwhelmed and asking, “How can I possibly keep up?” 

Do you see the social media movement as an opportunity or a new business burden? What have you decided to do? Will you hide in the sand or ride the potential wave? Do you really have a choice? Can you afford not to participate? 

Hasn’t the marketplace always been in a constant state of change? Perhaps, you’re thinking, yes but hasn’t change been slower in the past? Is it the pace of change that’s driving you crazy? 

Have you considered applying the practice of Organized Listening? Do you know what this involves? Do you need answers to all these questions? Do you know what’s next? 

Did you notice this entire blog was a series of questions?

Please share with me (in the comments below): 1 – The most valuable take-away you had from any of our blog posts 2 – What you want me to write about next And, please send out a tweet and tell a couple of people about this blog – Thanks!  http://researchplaybook.wordpress.com

	Join Our Email List

 

 

Web-based and telephone interviews are the major data collection methods in use today for quantitative studies, but they are not the only methodologies available. Mail surveys, mall intercept surveys, and other methods (such as social media intercepts) are, on occasion, data collection approaches worth considering.

As with other aspects of market research (e.g., analytics), the data collection approach can vary as much as the objectives. Aligning the research methodology with the study objectives is the best way to ensure your efforts yield high utility results. Carefully consider the desired breadth and depth of your research objectives. Your selection of the appropriate data collection methodology and target sample size should reflect those objectives. 

Taking an eclectic approach to data collection ensures the strengths and weaknesses of each approach will be evaluated in the context of the research objectives. For example, unaided awareness and preference studies are difficult to accomplish using a web-based approach, especially if the study is exploring multiple products types or uses probing questions. Conversely, while not impossible, it would be  undesirable to use a telephone interview to collect information when long lists of items or complex statements are part of the research instrument design (e.g., multiple response questions with “all that apply” question structures).

Is your business plan based on the results of a ten person focus group when, in fact, your board of directors expects a quantitative study to substantiate your market assumptions? Are you ignoring the use of web-based surveys and social media networks when targeting an audience that communicates electronically first and foremost? If so, you may need to rethink either your research objectives or your research design and methodology.

Download a Free Report with details on Data Collection  at: www.atheath.com/MRRC

 Please email a friend about this blog. Thanks!

	Join Our Email List

Questionnaire or Research Instrument (RI) design for any data collection approach (e.g., telephone and web-based) must be aligned with the method selected. Web-based and telephone instruments share the major requirements for good design, such as avoiding (at all costs) double-barreled questions, clear question construction, use of definitions when needed, and many other basic tenets of question writing.

However, the implementation of the RI for a web-based versus a telephone study requires different treatments of some issues. For example, the instruction sets used by interviewers versus instructions provided for respondents need to be written with each approach in mind.

In telephone interviews, the interviewer can tell the person what to expect and read definitions to the interviewee. Web-based instruments must be careful not to ask too much of the respondent with respect to long passages, the person must read to answer a set of questions.

On the other hand, a web-based study allows the respondent to go at his or her own pace and easily review a set of answers or change item percents that need to total to 100%. These examples and many other considerations need to be taken into account when designing the research instrument to optimize data collection and data quality.

In Summary

It is critical to align the data collection approach (e.g., telephone or web-based) with the questionnaire design. Implementing a web-based versus a telephone study questionnaire requires treating some design issues very differently. Ultimately, aligning the data collection method with the questionnaire design is a function of evaluating the strengths and weaknesses of each approach in relation to the goals of the study.

Download a Free Report with details on Data Collection  at: www.atheath.com/MRRC

Please email a friend about this blog. Thanks!

	Join Our Email List

A new book Questionnaire Design for Business Research provides valuable lessons to help you with an array questionnaire best practices. This book will be formally release on June 1, 2010. However, it is available now directly from the author or the publisher’s website.  Here is the link:
http://www.tatepublishing.com/bookstore/book.php?w=978-1-61566-835-9

What do you want when you send out a request for quotes (RFQ) or a request for proposals (RFP) for web-panel pricing to panel providers, besides a price?

If this question puzzles you it is probably because in our rush-to-get-it-done world we sometimes move too quickly and end up wasting time rather than using our time efficiently.  Writing a request for proposals (RFP) or more typically in the case of web panel providers a request for quotes (RFQ) should include more than asking for: “How much is it?”

The price and the cost may be two different dollar amounts.

The price you pay an individual panel provider versus the cost of the decision to use one provider over another must factor in the value received. It’s your total cost of ownership (TCO) or perhaps more specifically in this case your total cost of sample development (TCSD) that is at stake. We’re not going to create this metric, but it should be noted that the quality of your research project is directly proportional to the quality of your sample.

If you’ve ever heard me talk about research you’ve probably heard me say: “You can never fully recover from a poor sample – No amount of clever data processing, cleaning, or manipulation will save you.”  With that thought in mind, let us work through the process of writing a world-class RFQ for web-panel sample development that you can use when you launch your next project.

Apples to Apples 

To evaluate and understand the apples to apples value from each provider you’ve sent the RFQ to, you must set parameters in your request that are very clear and precise.

For example, if you don’t specify the incidence rate expected or assumed you will very likely get back quotes (bids) with different rates and prices; some high-priced with a low incidence assumption and some low-priced with a high incidence assumption.

Which is correct? The answer is both.

Which is better? The answer is neither.

However, the low incidence high price is probably more realistic – at least you are less likely to be handed a nasty surprise in the form of an unexpected invoice at the end of data collection.

What should you do?

First and foremost, you want to take control, which simply means that you must specify the incidence rate or rates (explained below).  In fact, you must learn to take the time to specify a number of key parameters. Doing so will reduce the time from when you send out the RFQ and when you receive final quotes you can compare. If you don’t take the time up front you’ll be getting questions from panel providers individually and their questions will not necessarily overlap. Therefore, you will be updating each provider with the newest clarification you developed based on the most recent question you have received from each provider – what a hassle!

Responding to these questions will require more work than creating a well thought out RFQ in the first place. It will also delay the process of having apples-to-apples proposals to compare. Why go through all that trouble?

For more in-depth How-to instruction see, ”Fast Track Your Web Panel RFQ Process” Here is a link to the abstract http://www.atheath.com/booksandseminars

	Join Our Email List

It is important to be a research historicist; remember it was not too long ago sampling panels for web-based studies did not exist. For example, two prominent panel providers both founded in 1999: GMI and e-Rewards began with consumer panels; specialized business, industry specific, and panels of professional groups (e.g., medical and pharmaceuticals) appeared later. Another case in point, WorldOne Research a European based company, only had an Information Technology (IT) professionals panel operational for a little over three to four years as of 2009.

These and many other firms developed consumer panels first and later specialized panels such as panels of medical professionals and panels of other professional groups such as Information Technology (IT) panels.

Development of one of the first IT panels took place at International Data Corporation (IDC, circa 1997) for internal use only to serve IDC clients. However, the initial mechanism for data collection was disk-by-mail, not email invitations to an external panel provided by a panel vendor.

With this in mind, we should temper our expectations about the state of the art of panel capabilities. The fact is this is a young industry. Geographic scope, as well as the quality of panels is still evolving and maturing. At the same time, the development and use of panels is changing as the world embraces Web 2.0 technology and the social media that is defining the next wave of Internet use. Keep a close watch on sampling approaches, while new innovations may provide access, they must also be valid to provide value.  

	Join Our Email List

Facebook’s success is nearly unparalleled and marketing, as well as marketing research, professionals are scrambling to find the best ways to take full advantage of this social media phenomena. However, it is this very zeal to immediately and aggressively leverage Facebook that could be its undoing.   

Crowds Flock to Facebook

As we rush in to take advantage of opportunities presented by an audience of 400 million people we could be destroying the goose before she has a chance to lay any golden eggs. The question is will these 400 million (and growing) Facebook users accept the potential promotional onslaught that Fan Pages could unleash or will they reel against this wave of marketing and research into their lives and “unFan” in droves leaving Facebook an empty shell?  

Abusive Marketing Could Cause Unfanning

It’s hard to bet against a company like Facebook, their track record is so impressive, yet I wonder if it isn’t time to be looking beyond this and other social media networks that may have peeked without us knowing it.  

The shame of it is social media has so much potential, but it could be wasted by greedy and aggressive forces in the marketplace. We have so often ruined our own opportunities. Take the fishing industry, which fished the Grand Banks into exhaustion, the greed on Wall Street that people believe caused our current recession, or the over use of natural resources, a list of cases too numerous to include in a short blog post.  

Social media networks are a resource too. Will we plunder it or will we use it wisely and help to sustain its value for the long-term?  

Please share your thoughts on this  issue. Add your comment below, and please email a friend about this blog. Thanks! 

	Join Our Email List

Poor research instrument (RI) design (e.g., Asking leading and misleading questions) is more than a pitfall. You simply cannot fully recover from fielding a poorly designed questionnaire – no degree of clever analysis will save you. To a large degree, it is all about HOW you ask the question. For example:

Q10. When was the last time you upgraded the memory and storage on your file server?

This is a double-barrel question.  If you are a respondent, you wouldn’t know how to answer this question. Memory and storage are different elements of a computer system and they aren’t necessarily upgraded at the same time. Thus, if the respondent answers the question would he or she be referring to the memory or the storage or both? When you analyze the data, how will you know which part of the question was answered? Half of the sample could be responding to memory while the other half might be responding to storage. You will have no way of knowing for sure. Consider the following question:

Q20. When did you stop providing your customers with improved service?

How would you answer this question? And, how would you interpret answers to this question? If you are having trouble describing how you would answer the question and you find it difficult to imagine how to interpret the possible answers, you understand the problem with this question structure.

Researchers must address these and numerous other questionnaire design issues and problems. Perhaps, this is a good segue to announce a new book Questionnaire Design for Business Research, (2010, Tate Publishing), which offers a full discussion on questionnaire design.

Here is the link:  http://www.tatepublishing.com/bookstore/book.php?w=978-1-61566-835-9

For email Newsletters you can trust

	Join Our Email List