Comparing Closed-Ended and Open-Ended Questions
A key part of creating excellent online surveys is in the proper uses of both open-ended and closed-ended questions. First, we must look at the nature of both question types, their strengths and weaknesses, and when they should be used. So let’s jump right into things!
Closed-ended questions come in a multitude of forms, but are defined by their need to have explicit options for a respondent to select from. There are a wide variety of closed-ended question types for survey creators to choose from, including: Multiple choice, semantic differential, drop down, check boxes, ranking, and many more. Each question type does not allow the respondent to provide unique or unanticipated answers, but rather, they have to choose from a list of pre-selected options. It’s like being offered spaghetti or hamburgers for dinner, instead of being asked “What would you like for dinner?”.
How to Use Closed-Ended Questions
Questions that are closed-ended are conclusive in nature as they are designed to create data that is easily quantifiable. The fact that questions of this type are easy to code makes them particularly useful when trying to prove the statistical significance of a survey’s results. Furthermore, the information gained by closed-ended questions allows researchers to categorize respondents into groups based on the options they have selected.
Demographic studies can illustrate a good use for closed-ended questions. Imagine that the manager of a designer clothing store believes that certain types of people are more likely to visit their store and purchase their clothing than others. To decipher which segment groups are most likely to be their customers, the manager could design a survey for anyone who has been a visitor. This survey could include closed-ended questions on gender, age, employment status, and any other demographic information they’d like to know. Then, it would be followed by questions on how often they visit the store and the amount of money they spend annually. Since all the questions are closed-ended, the store manager could easily quantify the responses and determine the profile of their typical customer. In this case, the manager may learn that her most frequent customers are female students, aged 18-25. This knowledge would allow her to move forward with an action plan on how to cater to this niche better or break into other target demographics.
The major drawback to closed-ended questions is that a researcher must already have a clear understanding of the topic of his/her questions and how they tie into the overall research problem before they are created. Without this, closed-ended questions will lead to insufficient options for respondents to select from, questions that do not properly reflect the research’s purpose, and limited or erroneous information.
For example, if I asked the question “Do you get to work by driving, busing, or walking?” I would have accidently omitted carpooling, biking, cartwheeling or any other form of transportation I am unaware of. Instead, it would have been better for me to ask the open-ended question of “How do you get to work?” to learn all the different types of answer before forcing the selection based on a list of several options.
Open-ended questions are exploratory in nature. As discussed with the “How do you get to work?” question, it allows for the respondent to provide any answer they choose without forcing them to select from concrete options.
How to Use Open-Ended Questions
Questions that are open-ended provide rich qualitative data. In essence, they provide the researcher with an opportunity to gain insight on all the opinions on a topic they are not familiar with. However, being qualitative in nature makes these types of questions lack the statistical significance needed for conclusive research. Nevertheless, open-ended questions are incredibly useful in several different ways:
1) Expert Interviews: Since questions that are open-ended ask for the critical thinking and uncut opinion of the respondent, they are perfect for gaining information from specialists in a field that the researcher is less qualified in. Example: If I wanted to learn the history of Ancient China (something I know very little about), I could create my survey for a selected group of historians whose focus is Ancient China. My survey would then be filled with broad open-ended questions that are designed to receive large amounts of content and provide the freedom for the expert to demonstrate their knowledge.
2) Small Population Studies: Open-ended questions can be useful for surveys that are targeting a small group of people because there is no need for complex statistical analysis and the qualitative nature of the questions will give you more valuable input from each respondent. The rule here is the group must be small enough for the surveyor to be able to read each unique response and reflect on the information provided. Example: A supervisor who is looking for performance feedback from his/her team of six employees. The supervisor would benefit more from questions that allow the respondents to freely answer rather than forcing them into closed-ended questions that will limit their responses.
3) Preliminary Research: As stated in the closed-ended questions section, conclusive research usually requires preliminary research to be conducted in order to design the appropriate research objects, survey structure and questions. Open-ended questions can reveal to the surveyor a variety of opinions and behaviours among the population that they never realized. It is therefore, incredibly useful to use open-ended questions to gain information for further quantitative research.
4) A Respondent Outlet: It is usually a good idea in any survey, no matter how large, to leave an open-ended comments question at the end. This is especially in the case of a survey asking closed-ended questions on attitudes, opinions, or behaviours. Forcing respondents to answer closed-ended questions asks them to fit in your box of options and can leave them with extra information or concerns that they want to share with you. Providing respondents with the outlet of a comment box is showing them the respect they deserve for taking the time to fill out your survey.
There are a few drawbacks to open-ended questions as well. Though respondent answers are almost always richer in quality, the amount of effort it takes to digest the information provided can sometimes be overwhelming. That is why open-ended questions work best in studies with smaller populations. Furthermore, if your survey sample is a fraction of the population you are studying, you will be looking to find data which can be inferred on the overall population as statistically significant. Unfortunately, open-ended questions cannot be used in this manner, as each response should be seen as a unique opinion.
Free Survey Q&A
Join our survey & research expert Rick Penwarden as he answers all of your questions every Wednesday at 1PM EST!
When conducting usability studies or field studies, it’s a great idea to ask lots of open-ended questions. Typically, researchers ask questions before, during, and after research sessions. It’s easy to focus on what you want to know rather than on how you ask, but the way you ask questions matters a lot in terms of what and how much you can discover. You can learn unexpected and important things with this easy technique.
Open-ended questions are questions that allow someone to give a free-form answer.
Closed-ended questions can be answered with “Yes” or “No,” or they have a limited set of possible answers (such as: A, B, C, or All of the Above).
Closed-ended questions are often good for surveys, because you get higher response rates when users don’t have to type so much. Also, answers to closed-ended questions can easily be analyzed statistically, which is what you usually want to do with survey data.
However, in one-on-one usability testing, you want to get richer data than what’s provided from simple yes/no answers. If you test with 5 users, it’s not interesting to report that, say, 60% of users answered “yes” to a certain question. No statistical significance, whatsoever. If you can get users to talk in depth about a question, however, you can absolutely derive valid information from 5 users. Not statistical insights, but qualitative insights.
How to Ask Open-Ended Questions
Are you satisfied?
How satisfied or dissatisfied are you with this process?
Did it act as you expected?
What would (did) you expect to happen when you ... ?
Did you find it?
Before a task:
After a task:
Do you think you would use this?
How would this fit into your work?
How might this change the way you do that today?
Does that work for you?
What do you think about that?
Have you done this before?
What kinds of questions or difficulties have you had when doing this in the past?
What happened when you did this before?
Please describe your level of experience with …
Is this easy to use?
What’s most confusing or annoying about … ?
What worked well for you?
Did you know … ?
How do you know ... ?
Do you normally … ?
How do you normally ... ?
Did you see that?
What just happened?
What was that?
Do you like this?
What would you most want to change about … ?
Which things did you like the best about … ?
Did you expect this kind of information to be in there?
Before a task:
After a task:
Why Asking Open-Ended Questions Is Important
The most important benefit of open-ended questions is that they allow you to find more than you anticipate: people may share motivations that you didn’t expect and mention behaviors and concerns that you knew nothing about. When you ask people to explain things to you, they often reveal surprising mental models, problem-solving strategies, hopes, fears, and much more.
Closed-ended questions stop the conversation and eliminate surprises: What you expect is what you get. (Choose your favorite ice cream: vanilla, strawberry, or chocolate.) When you ask closed-ended questions, you may accidentally limit someone’s answers to only the things you believe to be true. Worse, closed-ended questions can bias people into giving a certain response. Answers that you suggest can reveal what you are looking for, so people may be directly or indirectly influenced by the questions. Don’t ask, “Does this make sense?” Ask, “How does this work?” and listen closely to discover how well the design communicates its function. Note users’ word choices, because it might help to use their terms in the interface.
Start open questions with “how” or with words that begin with “w,” such as “what,” “when,” “where,” “which,” and “who.”
Don’t start questions with “was” (an exception to the “w” tip) or other forms of the verbs “to be” and “to do.”
In general, avoid “why” questions, because human nature leads people to make up a rational reason even when they don’t have one. We normally ask “why” only about ratings, to tease out more open-ended feedback. Say “Please tell me more about that,” instead.
Aim to collect stories instead of one- or two-word answers.
Even when you must ask closed-ended questions, you can ask an open-ended question at the end, such as, “What else would you like to say about that?”
Adding Other __________ to a set of multiple-choice answers is also a good way to get open-ended feedback.
When to Ask Open-Ended Questions
- In a screening questionnaire, when recruiting participants for a usability study (for example, “How often do you shop online?”)
- While conducting design research, such as on
- Which problems to solve
- What kind of solution to provide
- Who to design for
- For exploratory studies, such as
- During the initial development of a closed-ended survey instrument: To derive the list of response categories for a closed-ended question, you can start by asking a corresponding open-ended question of a smaller number of people.
When To Ask Closed-Ended Questions
- In quantitative usability studies, where you are measuring time on task and error rates, and you need to compare results among users
- In surveys where you expect many (1000+) respondents
- When collecting data that must be measured carefully over time, for example with repeated (identical) research efforts
- When the set of possible answers is strictly limited for some reason
- After you have done enough qualitative research that you have excellent multiple-choice questions that cover most of the cases
Whenever possible, it’s best to ask open-ended questions so you can find out more than you can anticipate. Test your questions by trying to answer them with yes or no, and rewrite those to find out more about how and what. In some cases, you won’t be able to accommodate free-form or write-in answers, though, and then it is necessary to limit the possibilities.