The Research Playbook

Archive for February, 2010

Knowing how stakeholders plan to use research results facilitates progress and success; not knowing could, and usually does, spell trouble.

Here is an example of the type of question you probably want to ask. “Do you expect to use the results of your research to drive a product development strategy, create presentation materials for a board meeting, or serve as the backbone for a promotional campaign?”

Although each of these uses is a great reason to conduct primary research, the scope, deliverables, and timetables can be radically different depending on which objective is really driving the research effort.

If your research team is aware of what is creating urgency around a study (especially deadlines that are event driven) you and/or your client are more likely to get the results you need in the timeframe you need them. This information helps to refine the scope of the research program and determine design methodology, selection of sample sources, sample sizes and quotas, the analysis plan, deliverables, and ultimately, the preparation of results and implications or recommendations.

Please tell a friend about this blog. Thanks! http://researchplaybook.wordpress.com

	Join Our Email List

Part of a series on: Approaches to Data Collection

We will start with the premise that data collection methods should align with research objectives. No one methodology is the best approach for all research designs. The criterion for deciding on a method should relate directly to the research requirements.

Consideration of secondary factors should influence the decision only after the primary objectives are satisfied. On occasion lower priority criteria, such as time in field, or a concern about professional survey takers, are used to decide which data collection approach to employ. These types of issues need to be addressed, but should not dominate the decision on data collection method for a study.

Taking an eclectic approach to data collection ensures the strengths and weaknesses of each approach will be evaluated in the context of the research objectives. For example, unaided awareness and preference studies are difficult to accomplish using a web-based approach, especially if the study is exploring multiple products or product types and needs to use probing questions.

Conversely, while not impossible, it would be highly undesirable to attempt a conjoint study using telephone interviews. Conjoint studies are much better suited to a web-based approach.

Summary
The issue is not whether a study using a specific data collection approach is superior or inferior to another approach, but rather whether the research objectives are served better by one approach over another. Some research designs clearly align with specific data collection methods while in other cases more general strengths and weaknesses can be used to decide the approach that should be used. In some cases, the differences between data collection approaches are less dramatic and issues such as availability of field resources, cost factors, and time in field are the deciding factors.

Please share with me (in the comments below):
1 – the most valuable take-away you had from any of our blog posts
2 – what you want me to write about next
And, please send out a tweet and tell people about this blog. Thanks!  http://researchplaybook.wordpress.com

	Join Our Email List

Part Two of Two Written with Co-Author Charley Spektor   

 A website “Performance Tune-up” is multi-dimensional and web pages are only part of the task required.

During a recent engagement, a site owner hoped he could make performance improvements, but was uncertain where to begin. Charley Spektor, one of AtHeath’s website-analysis professionals and I stepped in and started a review of the site’s search engine optimization (SEO) results. The problem was that even on the search engine results pages where the site ranked high, click-through rates were quite low.

While evaluating the company’s website pages, we recognized several interconnected naming-convention issues.  First, his title tags were far too generic.

For reasons of confidentiality, we won’t mention our client’s website, but here is an analogous example we found recently while searching for Pinot Noir wines from the Pacific Northwest.

Here is the HTML title meta tag and Google search results for the home page of the Major Creek Cellars winery in Oregon:

 Note that only the words “Welcome page” are used in the title tag, instead of a more descriptive “Pinot Noirs and More – Major Creek Winery of Oregon”– or something to that effect. Our client was naming about 50% of his pages with the equivalent of “Welcome page.” His search engine results ended far from the top in “no-man’s-land” nowhere close to a page one (1) listing.

After we identified this issue, the website owner made a single change to his content management system’s title-tag process, and all page titles were retagged retroactively. Google crawlers have now picked up the newly minted descriptive title tags, and have incorporated them into their index. This led to hundreds of new first-page rankings for the site, particularly on long tail searches, whose keywords are matching many of the new title words. (See  Amazon’s Long-tail Search Advantage: A recent test)

Our client was compounding the title-tag issue by not including meta-description tags in his HTML. The meta descriptions, of course, are often picked up by the search engines and used as the “snippets” that appear on search pages under the title tag. Once again, our client fixed the meta description problem with an easy adjustment in his CMS system, and strong meta descriptions now display on his search results entries.

We worked with the site owner to make some painless fixes on both issues, and to date, site visits from organic search have jumped by 33%, which is a conservative estimate.

What is different about the approach we used? It began from the business side not the technical one. We used business experience to understand the needs of a business owner. We advise you to look at website performance issues in the same way – the business view should educate the technical tasks. Than take aim at improvements and set up tests to make sure what you change helps rather than hurts your site and business performance metrics. 

Of course, no one can absolutely guarantee that you will find ways to accelerate your website’s performance. However, a systematic approach is highly likely to make a material difference in the performance of your website.

Please tell a friend about this blog. Thanks! http://researchplaybook.wordpress.com/

	Join Our Email List

During the next few weeks, we will publish a series of topics related to data collection (one per week). The first entry is Data Collection an Eclectic Approach and will appear later this week.

Other topics include:
Advantages of Web-Based Methods, When to Use Telephone Interviewing, Questionnaire Design and Data Collection Alignment, Using a Hybrid Approach, and Telephone versus Web-based Study Cost Structures.

Stay tuned this series starts very soon!

Please let me know if there are topics related to data collection you would like us to add.

P.S. Part Two on Website Performance Tune-ups will be published this week too! It will be a busy week.

	Join Our Email List

Research objectives that are poorly defined or unrealistic is one of the most dangerous market research pitfalls. Your research objectives need to be well defined, realistic, and static once the study work begins (objectives that are moving targets will play havoc with you project outcomes). Sounds pretty basic right?   

Well, many market research initiatives travel well down the path to execution, but haven’t given nearly enough consideration to the basic business and market research objectives up front. The research objectives drive the methodology, including: Instrument design, data collection, analysis, and ultimately the recommendations. Without a well-defined set of research objectives, you run the risk of generating invalid or inappropriate results that can mislead your decisions rather than inform them.   

To be effective your research objectives must also align clearly with your business objectives. Establish a two-tiered objectives plan, business and research, to guide the process.    

Please share with me (in the comments below):
1 – the most valuable take-away you had from any of our blog posts
2 – what you want me to write about next
And, please send out a tweet and tell a couple of people about this blog – Thanks!  http://researchplaybook.wordpress.com
 

	Join Our Email List

In a recent blog post, we explored the problem of Mixed Mode Scales and used a very specific example to point out the issues. If you have not read that posting, please do so before continuing, it will make this blog entry easier to understand.  Mixed Mode (double concept) Scales

If you have read it, you will remember the problem.  In short, the problem with the question was a respondent could score ‘Effectiveness’ for an item as “higher than expected” when his or her original expectation was low or score ‘Effectiveness’ for an item ‘lower than expected’ while starting with an expectation that was high. Without knowing the original expectation level, the data are nearly impossible to interpret.

As we discussed in the previous posting, the solution is to separate out expectations from effectiveness. In addition, one could improve the data collection by providing some form of definition for effectiveness. Here is one approach worth considering.

Introductory Paragraph

The next two questions relate to the “effectiveness” in achieving business goals from the use of social media networks. For the purpose of this study, we define effectiveness as an above average outcome regarding the marketing of your brand, product or service.

Q10. What were your starting expectations regarding how well the following social media networks could help your company?

Our company had:

Very Low                                                         Very High
Expectations                                                  Expectations
                   1       2       3       4       5       6       7

 [Note the items to be rated are not necessary to the example, fill in your own]

Now we can ask the Mixed Mode Question we original posed:

Q20. Please rate the effectiveness in achieving your business goals compared to your expectations for each of the following social media networks your company uses. Please use the scale below, where: 1 = much lower than expected and 7 = much higher than expected.

Effectiveness in achieving business goals was:

Much lower                                                      Much higher
than expected                                                 than expected
                   1       2       3       4       5       6       7

 [Note the items to be rated are the same as the items in Q10]

Using this approach respondents are at least in the same ballpark with respect to how they define effectiveness (the definition is still a bit vague, but our goal is not to measure how respondents view effectiveness, although a study on that topic would be interesting).

We now know whether the respondent had high or low expectations for each social media networking item we are testing. This information allows us to examine the four scenarios we outlined in the previous posting and understand more about the relationship of expectations and outcomes. Finally, we can more accurately examine how effective (or not) the company’s social media networking efforts were.

This approach provides a much richer set of data and is one of the principles discussed in the book Questionnaire Design for Business Research, 2010, Tate Publishing.

Please share with me (in the comments below):
1 – the most valuable take-away you had from any of our blog posts
2 – what you want me to write about next
And, please send out a tweet and tell a couple of people about this blog – Thanks!  http://researchplaybook.wordpress.com

	Join Our Email List

Part One of Two  Written with Co-Author Charley Spektor   

A simple “Performance Tune-up” may be just what the doctor ordered. Are you ready to conduct, what is probably, a long-overdue website evaluation?   

Engine Tune Up

If your answer is yes or even maybe, read on and see what you might be able to achieve with a little work and a bit of luck.   

Concrete examples will help. Let me describe two recent engagements where we successfully helped business sites implement some dramatic performance improvements.   

Charley Spektor, one of AtHeath’s website-analysis professionals and I were recently asked to help a website owner find a way to boost the conversion rates of some poorly performing lead-generation sign-up pages. We studied the click-through funnel-process stats in Google Analytics, and identified several key roadblock pages that were driving the abandonment rate sky-high.   

Working with the site owner, we revamped the content of the pages and made a few simple design changes. One of the most powerful areas for optimization is the registration page and that is where we focused on attention. Results to date are the conversion rate has improved by over 25%, enough to make a material difference in the site’s performance. All you need to do is determine the “pressure” point of a website. Locate this pressure point, fix the problem and test.   

The practice of testing is extremely important. Never, assume the change you have made is for the better. Always demonstrate it to your own satisfaction empirically – test and test again.   

In a forthcoming blog post, we’ll explore the second example related to search engine optimization (SEO). A good site evaluation is multi-dimensional and web pages are only part of the task.   

Please tell a friend about this blog. Thanks! http://researchplaybook.wordpress.com/ 

	Join Our Email List

We found another picture of the Giant Squid caught in the Gulf of Mexico by the National Oceanic and Atmospheric Administration (NOAA) team on the Gordon Gunter and we want to share it with you.    

Aboard the Gordon Gunter

 

 Now that’s a lot of calamari!   

Please tell a friend about this blog. Thanks! http://researchplaybook.wordpress.com/
 

	Join Our Email List

  

   

What is the one social media activity a business owner must do each day of the week? This question was recently posed on LinkedIn and it generated a wonderfully rich set of answers. I will summarize some of the comments, guidance, suggestions, and tips that came from the collective wisdom of over 20 professionals. Unfortunately, a blog post is too short to provide you all the great advice offered, but I will provide you a good sample of the thoughts these people generously offered.  

Perhaps, the one theme repeated most among the responses I received was that “There is no ONE thing.” In affect the answer to my question was that the premise of the question was essentially flawed. This is no doubt true and in retrospect, I admit that trying to boil down a business owner’s activities related to social media to one thing is a bit like asking a sailboat captain during a race to tell you the one thing the crew must do to win. Impossible, you must do many things and do them well.   

Photographed by Alyson J. Azzara

With this in mind, here are eight of the suggestions people provided:  

1. It depends on your goals and objectives for the business. Decide what you want to accomplish through social media and then find the best strategy for using social media to support your business goals.  

Several of the people who participated echoed this comment. While it is a higher-level suggestion than the question asked it is good sound advice not only about social media but other business activities as well.  

2. Once you start a conversation, keep it going do not let conversations run out of steam. This can be as simple as checking email or as complex as responding to multiple posts and forums. Once you have started a conversation, be there to keep it going.  

It may seem obvious, but too few companies consistently use social media to listen to customers and prospects. They start something but don’t follow through and when they don’t see immediate results they abandon the approach.  

Here are a few of the various social media resources recommended:  

3. LinkedIn is your best bet! Linkedin.com is the single most important social network for business. [Keep in mind this was a LinkedIn discussion]  

4. Twitter was recommended by a number of people as a good way to develop interest, target people, and keep your finger on the pulse of the market.  

 5. Forget all the other stuff.  

• Blogging is best way to establish yourself as an expert by offering content that provides insight.
• If you have to choose one – blog. – you should write a blog entry or have other people in the company do it.
• Yes, write a blog. However, you cannot fully engage in social media by using only one approach.

6. First, listen. Read blogs, twitter messages and go to LinkedIn groups related to your business. Only than should you get involved in contributing.  

Obviously, not one thing!  

7. Business owners should post content (preferably original) to their site each day – search engines pick it up and it will create activity on their site. For example, travel agents could post the latest travel restrictions.  

8. The most important and the first analysis you should do is to find out what social media channels your customers use. Once you know the answer to this question, you can efficiently communicate with your prospects and customers on the social media networks they use.  

Clearly, there are a number of strong opinions on the topic of best practices for using social media. Expect to see more postings on this topic in the future.  In the meanwhile, go to http://www.atheath.com/MRRC to find two papers that I am certain you will find interesting.  

Please share with me (in a comment below):
1 – the most valuable take-away you had from any of our blog posts
2 – what you want me to write about next
And, please send out a tweet and tell a couple of people about this blog – Thanks!  http://researchplaybook.wordpress.com
 

	Join Our Email List

Data from mixed mode questions are hard, if not impossible, to interpret accurately. Here is an example of a mixed mode question and my problem analysis related to interpreting the results the mixed mode scale would produce. In a future blog posting, I will provide a solution to the dilemma posed in the problem analysis provided here.

The Mixed Mode Question

Qx. Please rate the effectiveness in achieving your business goals compared to your expectations for the following social media networks your company uses. Please use the scale below, where: 1 = much lower than expected and 7 = much higher than expected.

Effectiveness in achieving business goals was:

Much lower                                                      Much higher
than expected                                                 than expected
                   1       2       3       4       5       6       7

[Note the items to be rated are not necessary for the example, fill in your own items]

You may already see the problem, but allow me to point it out. The problem is a respondent could score ‘Effectiveness’ for an item ‘higher than expected’ but his or her original expectation was low or score ‘Effectiveness’ for an item ‘lower than expected’ with a starting expectation that was high. Without knowing the original expectation level the data are nearly impossible to interpret.

Let’s explore this further by looking at extreme cases.

1. If all respondents had, low expectations and they responded: 1= much lower than expected, what would be your interpretation?

2. On the other hand, if all the respondents had high expectations and they responded: 1 = much lower than expected, what would be your interpretation?

3. Conversely, if all the respondents had low expectations and they responded: 7= much higher than expected, what would be your interpretation?

4. And again, if all the respondents had high expectations and they responded: 7 = much higher than expected, what would be your interpretation?

In scenario # 1 People start out with low expectations and their experience is lower still – Wow! What does this say? Social Media was not expected to do much and it did even less!

In scenario # 2 People start out with high expectations and their experience is lower than expected – well perhaps the expectations were unrealistically high! This tells us something entirely different from the first scenario.

In scenario # 3 People start out with low expectations and they experience higher value than expected – well good, but it wasn’t hard to meet or exceed a low bar – right?

In scenario # 4 People start out with high expectations and they experience value higher than even their originally high expectations – Wow, now that’s a winner! And, a very different interpretation from any of the other scenarios.

As you can see the mixed mode question or scale is important to avoid. I will develop a research instrument to show how to avoid this serious design flaw. One that explores the relationship between effectiveness and expectations you can interpret accurately.

P.S. Mixed mode scales are one of the 25-30 quality control (QC) items checked in our standard Questionnaire Audit.

Please share with me (in the comments below):
1 – the most valuable take-away you had from any of our blog posts
2 – what you want me to write about next
And, please send out a tweet and tell a couple of people about this blog – Thanks!

	Join Our Email List