The Research Playbook

Archive for the ‘Questionnaire Design’ Category

_________________________________________

Often when I recommend that a research team prepare a formal analysis plan the first response I hear is, “Why? The analysis isn’t due for weeks and I have too many other things to do.” Alternatively, I hear statements like, “That is too much extra work, I know what to do, I’ve done a lot of analysis work.”

An analysis plan is not extra work; it’s work that makes all the other project tasks flow efficiently. It will help you produce on-time project deliverables. Typically, you develop an analysis plan in parallel with your research instrument (RI). Like the RI the analysis plan is tied back to the goals and objectives of the study.

In addition to the obvious purpose of an analysis plan, producing a plan serves to improve the RI and manage project scope, these benefits alone will pay you for the time you devote to creating it.

The RI is referenced in an Analysis Plan (AP) and while there are no hard or fast rules and no one right way to structure an AP we can offer some guidelines. The approach presented here is as good as any and better than most.

The analysis plan approach described is specific to quantitative studies. The first step of the process will be familiar to those of you who read some of my other blog posts and publications.

Research has the greatest chance of success when the objectives are clearly stated and that is where we begin. Use these five (5) straightforward steps.

1. State the key study objectives clearly at the beginning of the analysis plan (AP) and refer to them throughout the process.

2. Describe the major comparisons for the analysis (e.g., major cross tabulations for the study such as: Customers versus Non-customers, Companies by size, Customers that are Satisfied, Neutral, or Dissatisfied).

3. State how each question is used to answer a specific objective of the study either on its own or in combination with other data points. Think through how you expect to present the results from each question. What statistics, if any, will you use in the analysis? Identify the independent and dependent variables.

4. Write a clear justification for including the information from the question in the study and perform a section by section “So what” litmus test.

5. When the analysis plan is finished, go back and make sure each key study objective has been addressed.

These five steps are the basic approach to the AP template. While it is straightforward it is not a trivial task. The key is to focus on objectives and think critically about how to execute on the primary goal of the study.

For a more detailed description of how to develop an Analysis Plan see Analysis Plans Made Easier, which is on the www.AtHeath.com Resource tab (scroll about halfway down the page).

For Email Newsletters you can trust

____________________________________________

Research Axiom: You can never fully recover from a poorly written questionnaire.

• No manipulation of the variables, regardless of how cleverly done
• No amount of analysis, regardless of how brilliant
• No degree of insightful interpretation, regardless of intellectual prowess

Nothing can save you from a poor research foundation. The building will collapse like a house of cards!

By Carey V. Azzara

If there is one part of the research process that I know, it is questionnaire design. It is a task repeatedly given insufficient time and attention. Clients and research professional alike often underestimate the time it will take to develop a truly well structured and concise instrument.

What amazes me most is when this task is somehow relegated to a status depicted by the attitude of:  “Once the questionnaire is done we can get on with the important stuff, like analysis and reporting.” The assumption that analysis work is the essence of the research and the expectation that interpreting the results is where the mastery of research ultimately lies is a mystery to me.

Have we not pounded the concept of garbage-in garbage-out into our heads? Can new internet tools substitute for critically thinking and the hard work of aligning the research instrument to the purpose of the study – answering the business questions that sponsors paid to learn?

If this seems like a bit of a rant, well I guess I’m guilty. My own research on research including the use of a 25 point questionnaire audit system has shown me that even well healed researchers are less diligent about quality than one would hope. Research is not only science it is a craft [perhaps an art] and if the proper fundamentals are not applied the product is less than artful.

I’ll end the ranting with an analogy [but don’t be surprised to hear more on this topic]. If you have not studied and then practiced writing poetry would you expect to publish a book of poems simply because your company’s marketing department asked you to?

Designing a good quality research instrument probably takes less talent than being a good poet, but it’s close.

For Email Newsletters you can trust

_________________________________

No, I am not going to offer you yet another webinar, podcast, free report, or video.

We overstocked hard copies of my Book:

Questionnaire Design for Business Research

Tate Publishing (2010)

So I am making this offer to my Research Playbook readers

Purchase the book Questionnaire Design for Business Research, and you will receive a paperback copy signed by the author (my wife loves this author!)

My Special Offer is a signed copy [tell me who to address it to] for $16.75 with free shipping in the USA.

Perfect for anyone serious about:

• Raising the bar on questionnaire design in his or her organization
• Finding a cost-effective way to start designing a questionnaire
• Preparing for the next market research project
• Improving his or her research skills

IMPORTANT: To reserve your signed copy of Questionnaire Design for Business Research you must: Email me carey.azzara@atheath.com or call 508 400 6837.

We have less than 100 copies and my signing hand will probably give out sooner LOL. . . 

“I want to get this book in your hands.“

It is also available from the publisher’s website: http://www.tatepublishing.com/bookstore/book.php?w=978-1-61566-835-9

_____________________________________________

The questionnaire Audit, Correct, and Enhance (ACE) approach starts with the audit. The Questionnaire Quality Control (QQC) audit identifies problems and errors commonly encountered during the task of questionnaire design.

My Questionnaire seems perfectly fine to me

No one wants to hear that what she or he created is less than masterful, but there is more to Questionnaire Design than asking questions the way you do in everyday discourse. Our audit is structured to be helpful and provide actionable guidance to the author(s) of the research instrument.

We have created a Do-It-Yourself (DIY) version of this audit process and we invite you to use this process to improve your questionnaires before submitting them for fielding. The process is described in this post and we hope you find it valuable. Read the rest of this entry »

Using Groups to Connect

Finally, we come to the discussion of how to use LinkedIn Group participation to help you connect with your target audience. As the screen shot below illustrates, Groups is one of the options for making a connection.

You can select Group, when you and the person you want to connect with are both members of one or more Groups. You can select any of the mutual groups you belong to and invite him or her to connect with you.

If you are not both members of a Group, check the person’s profile and find a group in which the person participates, preferably a group relevant to you, then join it. Once approved as a group member you can connect with people in this group.

LinkedIn as always provides the option to “Include a personal note: (optional).” However, we do not consider this optional - always Personalize Your Invitations (see examples below)
Read the rest of this entry »

We typically think of social media sites such as LinkedIn as places for networking and sharing ideas, which is certainly their primary function. Over time, however, sites like LinkedIn (and to varying degrees other networking sites) have developed an enormous global reach. The sheer size of LinkedIn’s membership has resulted in the creation of groups on nearly every topic imaginable and discussions within these groups touching on thousands and thousands of subjects.

This “user” created content has propelled networking sites and LinkedIn in particular into a realm, which transcends networking. A listening opportunity for market researchers has emerged and not just for passive listening. The opportunity to conduct active listening (a.k.a. market research studies) is possible. In fact, it is already a reality, available to anyone with the skill to extract the information.

Here are three ways to access marketing information from LinkedIn:

Discussion Groups

Discussion groups offer a wealth of content. While you may rush to start your own discussions to capture content, you might find that the information you seek is already out there. Finding the group(s) and discussions with the content you are looking for can take some time and skill. However, if you find relevant content much of the work has been done for you (assuming the topic generated a large number of comments). The trick now is to synthesize the information and pull out the themes, trends, and other pearls of wisdom.

Of course, you can start your own discussion threads within a group. The advantage is you can design the specific question or topic you want to explore. In addition, as the author of the discussion you have a little more latitude to steer the discussion, allowing you to simulate an asynchronous focus group.

Answers

LinkedIn has additional functionality such as “Answers” which allows a user to pose questions and offer them up to the LinkedIn membership (Go to the main menu click on More and select Answers). The person asking the question can grade the answers and select a “best answer.” This is the motivation for taking time to participate.

Again, you don’t necessarily need to ask a question to glean value from this function. You may find that someone, and in some cases a number of people, have already asked a question that is close enough to the question you want answered. Once again, all you have to do is synthesize the content. Well that is the hard part, but the data are often there awaiting your analysis.

Polls

The Poll function also offers a simple but potentially powerful mechanism for collecting data on an individual question. Used creatively Polls can provide market intelligence, albeit with a very limited scope. You can search existing Polls or write and launch your own. Either way there are answers to questions waiting to be discovered. For no cost or low cost you can reap a great deal of market intelligence from LinkedIn using Polls and other functions either individually or in combination with one another.

Conclusion

These and other avenues for exploring the marketplace, listening to your target audience, and performing both qualitative and quantitative market research are available to anyone who wishes to use them.

Need help with questionnaire design ? Go to http://questionnairedesign.tatepublishing.net/
 

Analysis plans are executed when data collection is completed, but should be formulated during the research design phase. Therefore, analysis planning should begin in parallel with the questionnaire design. We create questionnaire designs (or architectures) to address specific data collection objectives. The analysis plan should reflect these objectives and be related directly to research and business goals.

How does the process work? The research team (hopefully with the client’s involvement) is well advised to think “backward” from the intended report or analysis rather than “forward” from the research proposal. Thinking backward from the analysis requires knowing what hypotheses (a fancy word for beliefs or expectations), the client had going into the study: “What are they attempting to discover?”

The analysis plan and the questionnaire to collect the data for the analysis must be coordinated. If you ask questions that are not related to specific research objectives, you have wasted valuable resources. On the other hand, if you don’t cover the research objectives in the questionnaire, well good luck addressing the client’s expectations. However, luck will have had little to do with it.  

Use a clear validation process to identify questions in the questionnaire that address the objectives outlined in the proposal or statement of work. In addition, be sure questions needed to “examine the data” [e.g., demographics and firmographics] are collected. These are the questions used for a priori segmentation analysis.

For a more in-depth discussion on analysis plans see: Analysis Plans Made Easier, find it on http://www.AtHeath.com/MRRC

	Join Our Email List

In product manufacturing “form follows function” similarly in research design the approach must fit the objectives; this includes: developing an experimental or quasi-experimental design, designing an effective questionnaire, determining sample requirements, data collection methods, and the analytic approach. 

Each of the research design elements is crafted to optimize the results and to work in concert with one another. 

Only as Strong as the Weakest Link

While this may sound a little like “Apple Pie and Mom” it is easy to lose sight of the importance of integrating all the elements of a project. The quality of the research is only as strong as the weakest link!
 

	Join Our Email List

Recently, when asked the question: “Do you have any advice on how to cut corners on sampling?” I leaned over toward my inquiring friend and said, “Yes.”

He leaned closer to me his eyes wide with anticipation, and then I replied, “Don’t do it!”

We both chuckled, but I know my answer was not what he wanted to hear. Still my advice stands. Do not compromise you will not be happy and neither will your client. Scope the study in a fashion that uses the available research funds effectively. If you want to conduct a multi-regional study with N=2,000 to explore the market for companies that supply solar energy equipment and your budget is $25,000 your headed for trouble and disappointment.

Does this mean you cannot use your budget effectively to shed light on the research and business objective you have? The answer is of course not. However, you are probably not going to accomplish the same results as a multi-regional study with N=2,000. You will need to scale back the research and the scope of objectives you can address with a smaller budget. In fact, this money is likely to be spent more effectively on a regional qualitative study, syndicated research, or an appropriate MCS.

Thus, as others have said in many venues, all successful research starts with an honest assessment of the research objectives and the resources available to achieve those objectives. Starting with sample size or data collection method or some analytic approach that seems intriguing is likely to end in disappointment or failure; at the very least you will not optimize your research efforts and resources.

If your sampling plan is well though out and you have created the means to execute high quality data collection we congratulate you!

You can congratulate yourself too!

But realize that this is only a necessary step and not sufficient to guarantee success, questionnaire design, analysis, interpretation, and all the other steps in the continuum of a research project must also be accomplished and done well too.  However, if you do not get the sampling step right (or the questionnaire design step) you can never fully recover.

You have probably heard me harp on this point, but it’s worth repeating: Research Axiom # Two: 

You can never fully recover from either a poorly developed sample that lacks validity.

• No amount of analysis, regardless of how brilliant
• No degree of insightful interpretation, regardless of your intellectual prowess
• No manipulation of the variables, regardless of how cleverly done

Nothing can save you from a poor foundation. The building will collapse!

You may have read this in other writings we have published, but if you haven’t re-read it now and review this research axiom each time you start a research project.

In addition, if someone asks you “Do you have any advice on how to cut corners on sampling?” Lean over and say, “Yes.” Then, reply, “Don’t do it!”

For more guidance on sampling see our eBook “Sampling Dilemmas and Solutions” www.AtHeath.com/MRRC

	Join Our Email List

I recently launched a poll on LinkedIn please participate. After you vote, you’ll be able to see results and analysis of how different types of professionals answered the question, which is:

Several critical skills are needed to conduct research. Which of the following skills do you consider your weakest?

Take poll:   http://polls.linkedin.com/p/89542/xelvm

Please add your comments on the general topic of research skill levels and requirements among marketing professionals.

	Join Our Email List