Do You Make These Common Card Sorting Mistakes?
Card sorting is designed to allow you to view your product through the eyes of your users. It also helps you to understand how they think about navigation and what they expect. This type of research may seem like a fairly easy thing to do, but if you forget about a few important things, it can be a waste of time. Here are some common card sorting mistakes.
Too much testing
A big mistake is to think the more users you test, the better. After all, testing more users can give you more data – more data leads to more info which allows you to see the bigger picture more clearly, right? This is one of the pitfalls that is easy to fall into.
There is a bit of a debate going on around the web about how many participants a card sorting study should have in order to get accurate results. The general conclusion is that you don’t need an entire army.
A study by Tullis and Wood suggests that if you want to obtain accurate results and stay on budget, you don’t need more than 20-30 people. Their theory is based on an observation that when they have tested more than 30 people, the results were very, similar to those where the study had 40, 50 or more participants.
Their basic conclusion was that if you conduct a study with 5 people and a study with 15 participants your results will be totally different from each other. But if you test 30+ participants, the correlation is pretty much the same, so it would be waste of time and money to test more and get the same results.
Knowing the findings of the Tullis-Wood study, Jacob Nielsen goes even lower than that, suggesting that even 15 participants is enough, especially if you want to stay on budget.
Choosing the wrong target group
A common mistake made during card sorting studies is not choosing the right audience. Your first priority should be focusing on user personas for your product and choosing the audience accordingly. It is a mistake to ask people involved with a project to be test subjects, select labels and create navigation. Even if they have the best intentions, they are not the people who will use it because they are not the target group.
Here is what the Tullis-Wood study has to add:
It is important to note that even the trees based on the smallest sample sizes are probably closer to the one for all 168 participants than might be obtained from speculation by a designer who is not a potential user of the content or application for which the organization is being developed.
Not listening to the participants
Numbers will tell you everything? Not exactly. If you base your conclusions and any future AI improvements solely on card sort number results, you’re making a mistake. Why? Because you will know the numbers and you won’t know users’ motivations. And it’s the motivations you need in order to fully understand the audience and build information architecture that makes sense to them. In the words of Jacob Nielsen:
Much of the value from card sorting comes from listening to the users’ comments as they sort the cards: knowing why people place certain cards together gives deeper insight into their mental models than the pure fact that they sorted cards into the same pile.
Too many options
As a researcher, you may want to get results fast as you are pushed for time. Or you didn’t exactly think the project through and decided to do one big study instead of separating it into smaller chunks. This is when you might want to make the mistake of tossing too many cards and labels for your testers to sort. And this can backfire. Why?
In general when we are confronted with too many options, our decision making processes tend to be slower. We lose the big picture and we don’t do well when asked to decide. This inability to decide is called the paradox of choice, and has been brilliantly presented by Barry Schwartz on TED Talk stage.
People’s attention span is 8 seconds. When we’re given a time consuming task and long pieces of information we tend to abandon it. This has been confirmed by Tullis and Wood in another study. They found that people are willing to spend only 20-30 minutes on a card sorting study or even less if the card sorting study is done online: 15-20 minutes.
Here’s what Donna Spencer has to add based on her own experience. She is an information architect with many years of experience and one of the organizers of UX Australia, yet this didn’t stop her from making this bad call.
I first made the mistake of trying to cover the whole intranet in one card sort, with the result that, when users went to sort the cards, they really looked at them and said, “But none of this really goes together. There is too much here.” Of course, they tried, which meant that my results weren’t very good quality.
As you can see, no matter if you’re a novice or a pro like Donna – mistakes happen. It’s how we learn. Do you have any research mistakes on your account and advice how to avoid them? Share your thoughts in the comments.