Improving your clarity: Using focus groups to test website designs

skydeck1IT IS GREAT that so many companies test new website designs using focus groups.

IT WOULD BE EVEN BETTER if more companies did it well.  With that in mind, I thought I’d share the methods I’ve developed, collected, and refined to gather actionable data for website redesign projects.

Desired Focus Group Outcomes

Focus groups for site redesigns share desired outcomes. Using an optometric approach (“this or this”) the goal is to test a number of design characteristics, one at a time over 90 – 120 minutes until the following outcomes are known:

- best overall design;
- web design layout preferences/trends;
- design characteristics (patterns and colors) preferences/trends;
- text and copy preferences/trends;
- image and graphical preferences/trends;
- interactive element preferences; and
- whether additional testing is required;

Focus Group Score Card

Although the product of every focus group is opinion (qualitative data), that doesn’t mean you’re unable to clearly identify opinions and trends. The key is to determine HOW STRONG opinions are. And HOW PREVALENT trends are. Enter the scorecard:

1

2

3

4

5

1

O

O

O

O

O

2

O O O O O

3

O O O O O

4

O O O O O

5

O O O O O

6

O O O O O

7

O O O O O

8

O O O O O

9

O O O O O

10

O O O O O

Like its standardized-testing based counterpart, the scorecard is the place where focus group participants will record their ratings and answers.  This ensures that everyone’s opinion is recorded quickly and accurately before  discussion begins.  I prefer providing pens to a #2 pencil so that it is more difficult for participants to change their answers after discussion begins.

While some marketers pre-assign rows on the scorecard for each focus group question, I prefer assigning scorecard rows dynamically and providing 10-12 extra rows.  This allows the facilitator to insert or add related questions “on the fly” and use the scorecard to collect responses as they are needed.

Best Overall Design

One of the most important outcomes for any site redesign focus group is determining the best overall design.  This remains important to justify the project and its related cost.  To do this effectively, you often have to test five or more designs. Why five or more? I like to include the following:

  • Unbranded sites from two or three top competitors
    To prove your new web site is better than the competition, you need to test it. Specifically, you should test web designs from leading competitors against the new design. Prior to testing each competitor site, removed each company’s logo and any identifying product/company terminology.
  • An unbranded version of your existing site
    Your new web design needs to be better than the existing one. This is why I always included an unbranded version of the existing website against new designs.  One of the worst potential outcomes of a redesign project is rolling out a site that produces worse results than the design it replaced.
  • Two or three new unbranded design layouts
    Often, you’ll want to consider more than one alternatives to the current web design. I’ve found that it’s possible to test two or three new concepts within the allotted 90-120 minutes.  If you want to test more concepts, you need more time — and you need to test for fewer outcomes.

It’s a good idea to test two pages from each design: the home page and a product page.  Testing the home page alone has limited value.  Via search engines, many site visitors will bypass your home page directly once the site is live. And, even for those who visit your home page, it is unlikely that they’ll spend a significant amount of time there before moving onto the appropriate inside page.

Each design of the five (or more) designs you test need to be scored by focus group participants on predefined criteria before discussing designs.  For example, participants might be asked to rate each design on a scale of 1-5 (where 5 = ‘excellent’ and 1 = ‘poor’) on each of the following characteristics:

1

2

3

4

5

General appearance

O

O

O

O

O

Easy to use/navigate

O O O O O

Clarity and breadth of information

O O O O O

Credibility of company

O O O O O

Overall impression

O O O O O

Testing five characteristics for five designs will provide you with 25 rows of scorecard data.  Since it only takes a few minutes for focus group participants to rate each design, the bulk of the time can still be used for conversation. One of the advantages of this approach is that you know which designs are best, and why, before the discussion begins.  As a result, you can have a more targeted, qualitative conversation.

During the conversation, ask participants to write down a unique keyword (such as “friendly” or “professional”) that describes each design.

One Thing At a Time

For testing other elements (such as inside pages, colors, photos, and navigational styles), it’s important to test one element at a time.  Present two options at a time to focus group participants and ask them to rate each option, or indicate their preferred option.  The goal is always to force choices by asking focus group participants to do one of the following:

  • Rate elements on a scale of 1-5;
  • Identify which of the elements is better; or
  • Provides a ‘yes’ (mark 1) or ‘no’ (mark 2) response.

Lead discussions about each option after participants have marked their score card.

Online Works Fine

Although a face-to-face environment is typically the preferred method for focus group participation, it’s not always possible for regional, national or international companies to gather participants locally.  In these instances, online presentation tools such as WebEx or LiveMeeting can work nicely for managing geographically dispersed groups — or groups larger than 8-10 participants.

Using these tools is typically a two-person job, where one leader presents options on-screen and leads discussion by telephone while another leader presents multiple-choice questions  (in lieu of a score card) and tabulates the results in real time.  Based on results, the tabulator can often send instant messages to the discussion leader to introduce observations and followup questions.

Undesirable Outcomes

When you conduct enough focus groups, you’ll inevitably run into situations where focus groups have undesirable outcomes such as proposed designs that rank lower than competitor websites or your company’s existing website. In these instances, there are typically three ways to proceed:

  • Make revisions to the proposed design and conduct additional focus groups.
    Since focus groups provide opinions, rather than facts, I typically avoid this in favor of the third option.
  • Accept undesirable outcomes as opinion and proceed as intended.
    Focus groups are helpful, but they aren’t all knowing. You’ll see in the video example below that new concepts — even great ones — don’t always test well.  Recognize concerns raised in focus groups and proceed cautiously. I typically forgo this option in favor of the third option.
  • Take the time for Quantitative Study.
    Use focus group questions  (already worded for Y/N and multiple choice responses) as the basis for an online survey with a larger group of target customers.  Validate or debunk focus group results and proceed accordingly.

Focus group encourage customers to share their thoughts, feelings, attitudes and ideas. They also provide a very useful way of getting buy-in within the company for website revisions.  Using the described process will provide the best qualitative results and the foundation for gathering quantitative results, if needed.

Credits: Cartoon by Tom Fishburne. See more of his work at Skydeck Cartoons.

Focus Group Dynamics

I’ve found that its MUCH EASIER to test aspects of website usability using focus groups than it is to test the quality of advertising creative.  This was EXPERTLY demonstrated for the opening of the 2007 Hatch Awards (New England’s regional show) when put the best TV commercial ever produced (Apple’s ’1984′) before focus groups and it failed miserably. See and share the result:

  • Mark Anderson

    I’m a big fan of anything with a chimpanzee in it. LOL! Dance moves?

    This sort of feedback is what has always made me skeptical of qualitative focus group feedback. You get lots of creative feedback from non-creatives. Your approach is a great combination of both quantitative and qualitative and forces people to own their own reactions via a scoresheet and pen (no changing your mind when people don’t agree with you).

  • http://www.hubpages.com Kesinee

    This is very good article and I really like the it. It’s very important nowadays that to apply research approach to develop the websites. Focus group is one of the best method to find out new ideas, creativities for websites. But the cons are costing/budgeting and the quality of RDs.

  • Pingback: bezzicante