The Research Playbook

Archive for the ‘Quantitative Methods’ Category

_________________________________________

Often when I recommend that a research team prepare a formal analysis plan the first response I hear is, “Why? The analysis isn’t due for weeks and I have too many other things to do.” Alternatively, I hear statements like, “That is too much extra work, I know what to do, I’ve done a lot of analysis work.”

An analysis plan is not extra work; it’s work that makes all the other project tasks flow efficiently. It will help you produce on-time project deliverables. Typically, you develop an analysis plan in parallel with your research instrument (RI). Like the RI the analysis plan is tied back to the goals and objectives of the study.

In addition to the obvious purpose of an analysis plan, producing a plan serves to improve the RI and manage project scope, these benefits alone will pay you for the time you devote to creating it.

The RI is referenced in an Analysis Plan (AP) and while there are no hard or fast rules and no one right way to structure an AP we can offer some guidelines. The approach presented here is as good as any and better than most.

The analysis plan approach described is specific to quantitative studies. The first step of the process will be familiar to those of you who read some of my other blog posts and publications.

Research has the greatest chance of success when the objectives are clearly stated and that is where we begin. Use these five (5) straightforward steps.

1. State the key study objectives clearly at the beginning of the analysis plan (AP) and refer to them throughout the process.

2. Describe the major comparisons for the analysis (e.g., major cross tabulations for the study such as: Customers versus Non-customers, Companies by size, Customers that are Satisfied, Neutral, or Dissatisfied).

3. State how each question is used to answer a specific objective of the study either on its own or in combination with other data points. Think through how you expect to present the results from each question. What statistics, if any, will you use in the analysis? Identify the independent and dependent variables.

4. Write a clear justification for including the information from the question in the study and perform a section by section “So what” litmus test.

5. When the analysis plan is finished, go back and make sure each key study objective has been addressed.

These five steps are the basic approach to the AP template. While it is straightforward it is not a trivial task. The key is to focus on objectives and think critically about how to execute on the primary goal of the study.

For a more detailed description of how to develop an Analysis Plan see Analysis Plans Made Easier, which is on the www.AtHeath.com Resource tab (scroll about halfway down the page).

For Email Newsletters you can trust

_________________________________

No, I am not going to offer you yet another webinar, podcast, free report, or video.

We overstocked hard copies of my Book:

Questionnaire Design for Business Research

Tate Publishing (2010)

So I am making this offer to my Research Playbook readers

Purchase the book Questionnaire Design for Business Research, and you will receive a paperback copy signed by the author (my wife loves this author!)

My Special Offer is a signed copy [tell me who to address it to] for $16.75 with free shipping in the USA.

Perfect for anyone serious about:

• Raising the bar on questionnaire design in his or her organization
• Finding a cost-effective way to start designing a questionnaire
• Preparing for the next market research project
• Improving his or her research skills

IMPORTANT: To reserve your signed copy of Questionnaire Design for Business Research you must: Email me carey.azzara@atheath.com or call 508 400 6837.

We have less than 100 copies and my signing hand will probably give out sooner LOL. . . 

“I want to get this book in your hands.“

It is also available from the publisher’s website: http://www.tatepublishing.com/bookstore/book.php?w=978-1-61566-835-9

Written in collaboration with Dr. Ralph Finos

In an earlier post, we talked briefly about near-term and long-term forecasting – Why Do We Forecast?

Near-term Forecasting and Planning
In near-term planning we’re trying to be as accurate as possible about the outcome – regardless of how we get there.  The premium is on prediction vs. explanation. 

Problems like:

“What are my sales for established products likely to be in the next 6 months?”

Or demand pull considerations in the supply chain:

“How many toilet seat covers is Walmart going to need from me in the next 2 weeks?”

This type of forecast enables short-term resource deployment and benchmarks to facilitate near-term decision making “now decisions.”  Recent history is a key factor because where you have recently been is likely to be highly correlated with where you will be in the near future. Methods that project from recent tracking data are the most useful.  Moving averages and cyclical factors (i.e., seasonality) are important.  Theories of market behavior are generally less important (unless you’re forecasting emerging products or markets.

Near-term planning is also vital for long-term planning.  If you’re not predicting the near-term effectively, then assumptions in your long-term forecast might be incorrect.  Near-term forecasting can serve as a benchmark/hypothesis test for a long-term forecast.

Long-term Forecasting and Planning
In most industries, confident forecasting can typically be accurate for 6-18 months.  It should draw on the recent past, but must focus on prediction related to likely customer behavior (i.e., adoption behavior) and market context (i.e., substitute technology, competition, economic factors).  As opposed to near-term forecasting, the forecaster must create a theory or set of assumptions about market adoption in the context of value as well as market context (i.e., other products and services).

For example, technology markets are subject to evolution and possible revolution in technology itself that can redraw the landscape of a market – or create entirely new ones.  The technology market forecast confidence horizon doesn’t typically go much beyond 12-18 months. Obviously, the more inclusive the market segment being forecasted (i.e. all application software), the more stable and amenable to forecasting a market will be.

However, the interesting forecasting problems (new products, new markets, new technologies – and their interplay) often involve narrow segments as entry points and complex factors that govern adoption, making prediction a challenge.  Therefore, any forecast going out 2 years or more requires solid theories of adoption as well as solid metrics that are correlated with adoption allowing the forecaster to create predictive models that lave a longer half-life and are robust, yet flexible.

Useful (i.e., good) forecasts must be:

1. Robust enough to enable repeat calibration of core market drivers
2. Flexible enough to adapt to new market realities and drivers

These conditions are easier to state than to accomplish. Any forecaster who has created and then validated his or her forecasts against actual events can testify to the challenges.

Written in Collaboration with Dr. Ralph Finos

This is a short collection of quotes related [in some cases loosely] to market forecasting. Forecasting is a difficult and challenging endeavor. It can be a dirt job, but someone has to do it!

We’ll kick it off with a quote from Mr. Churchill, he is always to the point.

The Basic Challenge
The future is just one damn thing after another. – Winston Churchill

Prediction is very difficult.  Especially when it’s about the future – Anonymous

Our business is prophecy and if prophecy were certain, there would not be much credit in prophesying – Max Radin

Never mind the noise in the market, pay attention to the price of the fish – Bahamian saying

Forecasting Tools and Processes
Anything worth doing is worth doing poorly – GK Chesterton

If we knew what we were doing, it wouldn’t be called “Research” [or in our case forecasting!] – Albert Einstein

Sources and Data Quality
Every tool carries with it the spirit by which it has been created –  Heisenberg

Torture the data long enough and it will confess – Anonymous

Staying on Top of Your Forecast
None of us really understands what’s going on with these numbers. –  David Allen Stockman, Director US OMB, on the U.S. Budget, 1981

Every model, no matter how detailed or how well conceived, designed, and implemented, is a vastly simplified representation of the world, with all the intricacies we experience on a day-to-day basis. – Alan Greenspan

Be Open to Novel Outcomes
We do not know what the future will bring, except that it will be different from any future we could predict –  John Maynard Keynes

Normal science often suppresses fundamental novelties because they are necessarily subversive to its basic commitment. -  Anonymous

We can’t solve problems by using the same kind of thinking we used when we created them – Albert Einstein

We make progress in economic theory one academic funeral at a time.

While it is easy to poke fun at ourselves as we endeavor to create useful forecasts, the forecaster’s job is a difficult one. Fortunately, there are best practices and a body of work we can draw upon.

Research results are usually delivered as a presentation and/or report per the requirements of the project’s stakeholders. However, the form factor is typically less important than how the information is organized. An executive summary is typically required, but if not it is an excellent idea, one we highly recommend. All executives and managers are busy professionals who want the bottom line. Afterward, they are likely to entertain the results that support the conclusions.

Executive Summary
An executive summary typically consists of text, tables, graphs, images, and figures that allow the research team to offer its insights. Documentation to support the insights, implications, and conclusions should be available to provide further detail and to show how higher-level implications were derived.

The presentation would do well to start by answering the questions posed by stakeholders, when they decided to fund the study – seems obvious, but often not the focus of the presentation.

Address the objectives of the study and organize the results around the answers that speak to these objectives – these could be divided into the sections of the questionnaire, but often to tell a good story some reshuffling needs to take place.

Using the Questionnaire as a Guide
The questionnaire architecture is a good starting point. You already made decisions about what is important and that’s what the questionnaire content covers.

Since the questionnaire was designed hopefully to address objectives, using it as a guide to organize your results can be a good short cut. However, the goal is to provide insight to clients so look for market changes, trends, and themes that will help clients prepare for what is ahead.

Therefore, you are not limited to the structure of the research instrument and you can certainly move sections around to tell a clear story, which is exactly what you want to do. The format you choose is less important than the story you tell. You need to guide your readers through the evidence once you have stated the implication or conclusion.

Ex. “Gaming habits in the youngest cohort studied are likely to change direction next year, they are moving to …..” Then give the supporting evidence.

Permission to be Wrong
Give yourself permission to be wrong – some of what you say won’t come true. The market will change after the study is complete and you cannot report on what happens next only on what you observed. Take some risks (not crazy risks) based on the data and plot a course to attach the market that your clients can readily use.

Telling them what people in their target market are doing is a great start (ground clients in the facts about the market), but that’s not enough. You have to tell them what people are likely to do in the future too. Remember, they are typically trying to make business decision, give them direction and support it with facts and if possible, a few sound bites from the open-ended questions that fit your assessment of the direction of the market.

Researchers should “listen for direction” from the data. What seems to be changing and what seems to be static (at least for now)? Examine the data for convergence, i.e., two or more trends that create a theme. Listen for agreement within and across groups or market segments. Your clients want to know what to do so tell them how market conditions are changing or might change. Tell them what messages are likely to resonate best with each age group (or another market segment view, e.g., male vs. female).

Summarizing
Summarize the results within each section – put the summary (a.k.a. the conclusion) first and then the supporting data. Use representative quotes in the summary sections to make your points come alive – if you have qualitative data use sound bites to punctuate your points.

Finally, use short paragraphs, be succinct – you know the saying “if I had more time I would have written you a short letter.” It’s your job to write a short letter!

Are you planning to deploy an online survey, conduct telephone interviews, or recruit focus group participants? A key differentiator for your team is the ability to manage the process of providing a sample frame that is appropriate to the research objectives.

Explore the various panels of respondents available as well as the list options at your disposal. You may want to tap one or both for sample development. To make the process as effective and efficient as possible, provide clear parameters and as much information about screening and estimates of incidence rates as you can.  

Thinking through the specifics related to the sample parameters ahead of time will help create a clear request for quotes (RFQ). The bids you get back will be based on your specifications and completeness and clarity are the best tools you have to avoid surprises.

If you need help creating an RFQ I highly recommend our eBook titled “Fast Track Your Web Panel RFQ”, an AtHeath Publication.

	Join Our Email List

Put More Punch in Your Surveys and get the “Fast Track Your Web Panel RFQ” eBook FREE!

Purchase  “Questionnaire Design for Business Research” it will help you create questionnaires using innovative best practices.

Find it at our dedicated website where you can read excerpts and get a special offer (two bonus eBooks valued at $59.90).   http://questionnairedesign.tatepublishing.net/

Just want the book go to a bookstores or to www.barnesandnoble.com   or www.amazon.com

Recently we asked the question What Exactly is a Research Panel? [see What Exactly is a Research Panel?]. Now we answer two related questions.

What is a Panel Not?

A panel is not a simple list of emails or names. Panels typically achieve a 5% – 20% response rates, which is much higher than the typical list response rate. The reason panel response rates are higher is panel providers actively manage the panelists and delete inactive respondents. Lists on the other hand are often old, overused, and typically provide under 1% and sometimes less than 0.5% response rates. 

Why Do I Need Panels? 

Panels allow you to better control your sample than lists. In the absence of high response rates, we are often left with quota sampling and a panel provides the means to fill quotas.

In addition, when you buy a list the cost is per number of names. Typically, lists are sold as a minimum purchase of 5,000 names, with discounts for larger pulls. Depending on the list, you may be able to pre-select names based on profile information. However, the quality of these profiles is often in question and rarely verified.

Panel companies charge per completed survey. The advantage is you know what you will pay (assuming you’ve correctly estimated the incidence rate) and the profiling information is typically of a higher quality and more current than from a list. With that said there is a good deal of variability in the quality of panels and panelists and the Latin phrase Caveat Emptor (Let the buyer beware) is a most appropriate caution.

This not because panel providers are disreputable, in fact, most are highly contentious. The caution is a warning not to assume anything, be an educated buyer and user. Understand the strengths and limitations of the lists and panels you use and you will be in a much strong position when clients ask tough questions.

	Join Our Email List

Panels are sampling frames, designed as sources for sampling specific populations as part of market research studies and used for both business-to-consumer and business-to-business projects. Sample development is an essential part of any research project.

Panels are divided into two basic types: public panels and private panels. Public access panels are public (hence the name), which means that anyone can access them for conducting research. Anyone, that is, who is willing to pay for the privilege of gaining access. A Public Access Panel is a database of people with profiles (some short others reasonably extensive) and emails, which is actively managed and accessed to conduct online market research.  

Panelists have actively signed up, usually through a double opt-in process (see Note below for definition).  In return for giving their opinion during online surveys, they receive rewards typically in the form of points or cash equivalents, which participants can convert to cash or other incentives. Other forms of incentives include (but are not limited to) airline miles, a chance to win a drawing, gift vouchers, and in some cases donations to charity.

Private panels are owned by a company or organization and used for their proprietary and explicit data collection requirements. These panels may be managed internally or the management may be outsourced to a third party. Private panel owners do not, in general, make their panelists available for research outside their company or organization. There are of course exceptions, but most exceptions are limited to a very close group of partners or alliances.

NOTE: Definition of Double Opt-in

First Opt-in           Person opts in to participate [typically online, but could be in any form]

Second Opt-in       Person is then sent an email asking them to confirm agreement to participate [the double opt-in]

	Join Our Email List

The sample plan is typically straightforward. However, on occasion when the sample is multinational or uses a complex stratification scheme the research team will need to examine its options; paying close attention the how complexity translates into cost. It’s very important once again to set expectation appropriately.

Use this checklist to think through the sample plan process and structure you sampling plan to maximize quality and minimize cost.

Data Collection and Sampling

1. Match the Data Collection method to research objectives

• Phone interviews
• Web-based interview
• In-person, face-to-face
• Email w/ phone
• Mail [postal]

2. Sample source options for web-based studies

• Bartered lists – No control, low risk, opportunity costs
• Paid lists – No control, high risk, RR management
• Panels – High control, CPI based cost, guaranteed sample size
• Internal respondent pools – Hard to develop and maintain, good control, limited size

3. Sample source options for phone-based studies

• Internal lists – High control, low risk, uncertain biases, opportunity costs
• General business lists [D&B] – Good control, low risk, Response Rate management
• Panels – High control, CPI based cost, guaranteed sample size
• Respondent pools – Hard to develop and maintain, good control, limited size

4. Sampling frame criteria issues to consider

• Feasibility to supply the sample is essential
• Filling quotas is the next most important
• Time in field and meeting deadline is third

5. Sample bias issues

• Assume that all sources have a bias

6. Mitigate bias through the use of:

• Multiple sources
• Response rate management
• Incentives
• Avoiding respondent fatigue
•  Translation – localization

7. Stratification and quota setting

• Use a Random Stratification
• Set Quotas

For more guidance on sampling see our eBook “Sampling Dilemmas and Solutions” www.AtHeath.com/MRRC

	Join Our Email List

One of the most powerful approaches you can use to better understand the market is a sales cycle analysis. Gaining a strong appreciation for the good and bad news about your market positioning in relationship to your competition isn’t always easy and if there’s bad news it’s difficult to hear. However, not knowing where you stand is at best dangerous and could prove fatal.   

Sales cycle analysis is about understanding why you do or don’t get on the short list of your prospects (or stay on the short list of you customers) and what will facilitate or impede your progress toward becoming a preferred supplier – the position we obviously all want to be in.  

Smoking HOT!

If knowledge is power than sale cycle knowledge is a supercharged Muscle Car.
 

	Join Our Email List