The Research Playbook

Archive for May, 2010

Recently, when asked the question: “Do you have any advice on how to cut corners on sampling?” I leaned over toward my inquiring friend and said, “Yes.”

He leaned closer to me his eyes wide with anticipation, and then I replied, “Don’t do it!”

We both chuckled, but I know my answer was not what he wanted to hear. Still my advice stands. Do not compromise you will not be happy and neither will your client. Scope the study in a fashion that uses the available research funds effectively. If you want to conduct a multi-regional study with N=2,000 to explore the market for companies that supply solar energy equipment and your budget is $25,000 your headed for trouble and disappointment.

Does this mean you cannot use your budget effectively to shed light on the research and business objective you have? The answer is of course not. However, you are probably not going to accomplish the same results as a multi-regional study with N=2,000. You will need to scale back the research and the scope of objectives you can address with a smaller budget. In fact, this money is likely to be spent more effectively on a regional qualitative study, syndicated research, or an appropriate MCS.

Thus, as others have said in many venues, all successful research starts with an honest assessment of the research objectives and the resources available to achieve those objectives. Starting with sample size or data collection method or some analytic approach that seems intriguing is likely to end in disappointment or failure; at the very least you will not optimize your research efforts and resources.

If your sampling plan is well though out and you have created the means to execute high quality data collection we congratulate you!

You can congratulate yourself too!

But realize that this is only a necessary step and not sufficient to guarantee success, questionnaire design, analysis, interpretation, and all the other steps in the continuum of a research project must also be accomplished and done well too.  However, if you do not get the sampling step right (or the questionnaire design step) you can never fully recover.

You have probably heard me harp on this point, but it’s worth repeating: Research Axiom # Two: 

You can never fully recover from either a poorly developed sample that lacks validity.

• No amount of analysis, regardless of how brilliant
• No degree of insightful interpretation, regardless of your intellectual prowess
• No manipulation of the variables, regardless of how cleverly done

Nothing can save you from a poor foundation. The building will collapse!

You may have read this in other writings we have published, but if you haven’t re-read it now and review this research axiom each time you start a research project.

In addition, if someone asks you “Do you have any advice on how to cut corners on sampling?” Lean over and say, “Yes.” Then, reply, “Don’t do it!”

For more guidance on sampling see our eBook “Sampling Dilemmas and Solutions” www.AtHeath.com/MRRC

	Join Our Email List

Multi-Client Studies (MCS) fit the classic good-news bad-news scenario.

The good news is the sponsors (clients) receive proprietary-like research results with robust sample sizes at a fraction of the cost of conducting a proprietary study. The bad news is so will all the other sponsors.

However, there is a model for conducting a MCS that minimizes the bad-news portion of this scenario, it is the semi-custom approach. This approach requires an extra effort from the research firm to apply the results in meaningful ways to the specific market conditions each sponsor has to face. For example, an enterprise with the largest market share is interested in protecting that position. A midsize company or new entry to the market will be concerned with how to grab share from competitors. Results from a carefully constructed study can be semi-customized to produce analyses that address each specific but different need.

Back to basics and the traditional MCS.

Of course, MCS come in a variety of shapes and sizes. There is no official definition of what constitutes a MCS, but in practice, these studies share a few basic parameters. First, quite obviously the study is conducted for more than one sponsor or company. This shared cost model is attractive especially when a market has several midsized and small players that would be hard pressed to fund a large-scale study alone. However, there are no defined minimum or maximum limits on the number of sponsors. Practical limits for managing sponsor requirements and the review process for tasks such as questionnaire approval come into play, but the research firm conducting the study will need to determine the optimal approach.

Shared costs between only two sponsors may work to everyone’s advantage, especially if the sponsors are not competitors (e.g., one provides products and the other provides services to the same market – they might be partners).   

When do MCS make the most sense?

Look for changes in technology or a market shift like some form of “tipping point” in the market, especially situations where two suppliers create a shift in the market. MCS that explore these changes and follow the trend for at least a year are typically valuable.

In addition, you can use MCS for competitive advantage; while it is sponsored by multiple companies you are still likely to be one of a few companies in the market to participate giving you an advantage over all the non-sponsors.  MCS are also useful and a relatively inexpensive mechanism to verify what you believe is happening in the market.

One last word about MCS.

Don’t buy a study based on fear. Fear is a poor substitute for needing to know. Results from studies bought to avoid being left behind are rarely used. As with all market research, to provide value there must be an internal user (stakeholder) who has an identified “need to know.” Without such a person or group of people, the deliverable is likely gather dust and you will have wasted valuable resources.

	Join Our Email List

I recently launched a poll on LinkedIn please participate. After you vote, you’ll be able to see results and analysis of how different types of professionals answered the question, which is:

Several critical skills are needed to conduct research. Which of the following skills do you consider your weakest?

Take poll:   http://polls.linkedin.com/p/89542/xelvm

Please add your comments on the general topic of research skill levels and requirements among marketing professionals.

	Join Our Email List

The entire set of research tasks to this point in a project relate directly or indirectly to the work required to develop a high quality questionnaire. This topic is of the utmost importance and we have published a book Questionnaire Design for Business Research, 2010, Tate Publishing, which focuses on the work required to develop a high quality research instrument (RI). One well suited to the research requirements and designed to be as user friendly as possible. Instrument design also requires the expertise to design custom questions to meet specific and unique requirements.

Most well established research firms have developed a library of questions to capture information on a variety of concepts (e.g., messaging, mindshare, short-list probabilities and many others). Instrument design also requires the expertise to design custom questions to meet specific and unique requirements.

RI Development Checklist

Checklists Rule! This checklist summarizes points you need to cover, but it is not an endpoint. It is a tool for you to use and develop further in your practice. Use checklists to help with project management and stay focused on the most critical tasks.

Research Instrument Design Issues

A. Architect the RI in relation to the study objectives

B. Develop the RI in conjunction with the data collection method (i.e., Phone versus Web)

C. Know the sample and quota requirements

D. Decide on the approach to sample development

E. Audit the questionnaire – test and retest timing and accuracy

• QQC Audit: Questionnaire quality control (Use AtHeath’s 30 plus audit process)
• Check timing use a rule of thumb and true timing tests (See Power of a Questionnaire QC Plan: Timing and Testing)

F. General design standards

• The RI includes a clear and concise introduction of the study that builds rapport with respondents to increase the likelihood of participation
• The RI both reads and “sounds” well (when read aloud)
• Clear instructions are used to specify Quotas for the research team (e.g., programmers, data collection partners) in the Screener Section of the RI.
• Use a set of “S10-Sxx” questions to screen respondents, terminating those who do not meet study requirements
• All interview prompts and probes are clearly specified and programming instructions are consistently formatted (typically in BOLD)

Here are a few resources that will help you:

Questionnaire Design for Business Research

Questionnaire Quality Control (QQC) Audit process

Power of a Questionnaire QC Plan: Timing and Testing   www.atheath.com/booksandseminars

	Join Our Email List

Here are five possible sampling options, no one option is best for all situations. Think of these options as a starting point for creating a sampling plan that will optimize the studies you are working on.

Option One: Rely on a single source of web-based sample from a panel sample provider you’ve used (hopefully with success) for all future studies and cycles of any ongoing studies. However, introduce an aggressive level of over sampling (15-20%) to mitigate sample quality issues you might have encountered in the past.

Option Three: Use a telephone-to-web approach. This is a good option in web-challenged countries. It is also an option to consider when studying markets where panels are insufficient or introduce too much bias. This approach has cost implications for your projects, but it may be necessary regardless of other options selected due to a lack of reputable panels in some countries or markets.

Option Four: Rely completely on high quality [but limited scope] panel providers. Quality in this case is defined by respondent verification, typically by phone. This is a great approach. However, the availability of sample is restricted to the major country markets, probably only seven or eight countries in total.

In the short-term, this alternative is limited with respect to use for all possible studies, but could provide telephone like results with respect to quality and become a stable sample platform for the future. One notable risk is the continued availability of this more costly sample. Sample customers will need to pay the higher price to purchase this source of sample to make the business model for panel providers that verify respondents feasible.

Option Five: Move to or continue to use a telephone approach for your studies. This side steps the web issues entirely, but still requires rigorous field management. If a consistent and a stable sample are the requirements, this might very well be one of the best choices to consider. Another driver for this alternative is to integrate the use of mobile devices and put your work on the fast track for what is likely to be the next wave on innovation in sample development.

	Join Our Email List

Companies stress the use of different types of market research depending on economic conditions. When businesses are struggling in an atmosphere of unemployment and budget cuts, they act differently than they do when the economy is booming.

During growth years, companies look for market opportunities often relying on syndicated research to provide guidance and direction. When budgets are tight and the research must help navigate very choppy seas, the research dollar is more likely spent on some form of consulting to address a specific concern the company is facing.

As we recover from hard economic conditions the need for market data and an analyst’s view of the market, which never disappeared, will become more important. The continuing need for this form of guidance was simply subsumed by the more immediate issues companies were grappling with as they tried to counteract very specific market conditions. Companies focus on point solutions rather than on broader market views when navigating through the uncertain waters of a recession or transformational change within an industry.

Market Sizing

Syndicated research is a wonderful business model for the thousands of research and analyst firms that employee this one-to-many approach. It provides a great service to companies interested in understanding (typically from supply-side analysis) the size of a market. While sizing a market can be challenging it is typically possible to arrive at a good estimate.

Market Forecasting

Buyers also want the analyst firm’s opinion of market growth or decline typically over a 3-5 year period in the form of a Compound Annual Growth Rate (CAGR). Forecasting a market is much trickier than sizing a market.

How the research analyst determines the growth rate, i.e., what assumptions he or she applies and how much the rate is simply a linear extension based on actual rates from past years are questions that should be addressed in the forecast methodology. However, these and other details are not always available. The reasons for less than a full disclosure can be vague and often revolve around the issue of some proprietary approach, which some firms claim they cannot make public.

Be wary of these proprietary claims, especially if you are a client and have paid for the privilege of obtaining the forecast. If you cannot evaluate the validity of the forecast, you may not want to place too much at risk based on its lack, or potential lack, of varsity. Caveat emptor – buyer be ware!

	Join Our Email List

Usually the concepts of Type I and Type II errors apply to statistical comparisons. However, we can also apply these concepts to sampling issues related to removing potentially bogus cases.  

Type I and Type II errors refer to: The probability of being wrong in one of two possible directions.

Specifically, a Type I error refers to a situation where you have determined there was a “real” difference when in fact there was no true difference. That is, a comparison that shows a statistically significant difference, but is not repeatable, it occurred by chance

You will probably use heuristics (e.g., a set of rules) to delete cases from your sample. If you determine that some responses came from invalid or “gamer” respondents when, in fact, the respondents were authentic, you have a Type I error. This might occur if your criteria for deleting sample cases are too strict and tend to eliminate valid respondents.

Conversely, a Type II error occurs when you accept the Null hypothesis, that is, you determine no real difference exists when in fact a real difference does exist (note: this can occur if your sample size is too small to support detection of true differences – a separate but related topic).

If you are using heuristics to delete a case in the sample and you determine that the case is from an authentic respondent, but, in fact, the respondent created a false or invalid case, you have a Type II error. This could occur if your criteria for deleting sample cases are too lenient and thus do not eliminate invalid respondents.

Which is the better error to make? In general, it is probably better to allow some bad cases to remain undetected. Therefore, our recommendation is to make Type II errors rather than be overly restrictive and make Type I errors. Throwing out valid cases is wasteful and can cause as much damage in the form of sample bias as keeping false cases. It is best to err on the side of caution – make more Type II errors than Type I errors.

Stop Making Questionnaire Design Errors!
Click Here:  http://bit.ly/Questionnaire_Design

	Join Our Email List

“The most powerful force in the Universe is compound interest”- Albert Einstein

It is likely that Einstein would agree that this powerful force also applies to compound growth rates.

In the world of marketing and market research, sizing and forecasting a market, usually with estimates of varying degrees of accuracy is a part of life. Here are some basics on how to calculate a compound annual growth rate (CAGR) that might come in handy.

The simple form of a compound annual growth rate is calculated by taking the n^th root of the total percentage growth rate, where n is the number of years in the period (typically 3-5) being considered.

The calculation is written as follows:

CAGR = (Ending Value / Beginning Value) ^ (1 / # of years) -1

In this calculation, which is relatively easy to set up in a spreadsheet, the ending value is the expected size of the market for the last year of the compound period (e.g., 5 years). The beginning value is the (actual or estimated) size of the market for the starting year in the period (often the year prior to when you are publishing the CAGR,  for which hard data rather than estimates exist), and the number of years is the period for the CAGR calculation.

An example will help:

Assume a market is estimated at $500,000 and is expected to grow to $1 million in five years:

($1,000,000 / $500,000) ^ (1 / 5) -1 = 0.14869

Breaking it down the calculation is:

($1,000,000 / $500,000) = 2

(1/5) = 0.2

(2^0.2)-1 = 0.14869 or 14.87%

Therefore, the CAGR for a five-year period in this example is equal to 14.87%, representing the smoothed annualized growth of the market estimated over this time horizon. The year-by-year growth may fluctuate with some years predicted to achieve higher growth than other years. In fact, it could conceivably include negative growth years. However, in this example the overall five-year results are expected to be positive.

You can apply this approach to any market and develop a spreadsheet to explore how plausible  your estimates are for each year in a forecast using different assumptions. Have fun!

	Join Our Email List

Do you want to:

• Raise the bar on questionnaire design in your organization?
• Find a cost-effective way to start designing questionnaires?
• Prepare for a new market research project?
• Improve your research skills?

If you answered Yes to any of these questions we have a Special Offer perfect for you.

 http://bit.ly/Q_DesignSpecialOffer

True market research professionals understand that the value of primary research is in your ability to use it effectively. Whether it’s a customer satisfaction survey, product assessment, ROI analysis, or another primary research initiative, you want people who are results-driven.

Look for research expertise in the area of methodology first; research design skills are a necessity. It’s a big plus if there is a qualified industry analyst on the team who can leverage his or her expertise to deliver actionable results. You want specific implications, conclusions, and recommendations that will have an impact on your business. This is or should be the goal of every market research project.

Ultimately, you will meet your objectives if you work from a position of knowledge. Avoiding common pitfalls when conducting market research will help you achieve your objectives. Dangerous Pitfalls to Avoid

Checklist: for Selecting a Research Partner

• Market research expertise
• Knowledge of your business or marketplace
• Diverse methodologies
• Understanding of business and technology issues
• Ability to make specific recommendations from the data
• Sound and thorough analysis
• Ability to LISTEN
• Skilled project management

The final step of a research study is to transform the market intelligence you achieve into insights you can act on to advance your business. Your research partner should help you work backward from the desired deliverable. Work with your vendor to design the research methods, research instruments, and analytic plans from that perspective.

	Join Our Email List

Do you want to:

If you answered Yes to any of these questions we have a Special Offer perfect for you.  http://bit.ly/Q_DesignSpecialOffer

Whenever, someone tells you they are going to share a “secret” in a publication that could reach hundreds or thousands of people you should suspect either to hear no true secrets or mostly ad copy with secrets (if any) hidden behind the “door” through which you pass after giving your credit card information.

The three secrets about Article Marketing I want to share with you are more like insights gleaned from months of active work and actively using article directories to conduct my own article marketing campaign. It was also an article marketing research project to help others with their campaigns. Okay so let’s get to it.

Secret Insight #1: The advice you will get on article marketing from many “gurus” is wrong. The idea that you can reach 400 or 1,000 directories with one blast and cover the world with your content and get a “gazzillion” (yes this is a technical term) back links and website hits, is nonsense. How does that fit into the basic tenant of being authentic in your social media marketing efforts? It doesn’t. Moreover, there are consequences that come with using inappropriate short cuts.

Secret Insight #2 Quality trumps quantity – it may feel like it takes longer to reach as wide an audience as you would like, but your reputation is worth it – don’t spin articles! For those of you who don’t know, clever programmers have created article spinners. Change a few words using synonyms and move paragraphs around, next thing you know you have a unique article – just change the title. Again, what are they thinking? Does that approach really fool anyone or just damage your reputation?

Secret Insight #3 Achieve expert status in the eyes of the article directory editors – it will pay dividends. A friend of mine who used spinners and automated blasts was complaining that he had not achieved platinum status with one of the premier article directories after submitting  over 40 articles. When I told him I had done it with ten (which is the minimum possible) he was furious! “How?” He asked. Simple I said – quality!

Remember, not all directories are created equal – I know I told you this already, but it is worth repeating.

Please share with me the most valuable take-away from the posts on Article Marketing
What would you like us to write about next?

And, please send out a tweet and tell a couple of people about this blog – Thanks!  http://researchplaybook.wordpress.com 

	Join Our Email List