This is a mobile version, full one is here.

Yegor Bugayenko
28 November 2023

Be Indirect in Your Research Questionnaire to Gain More Honesty

Let’s say you are conducting research to discover programmers’ opinions about their work environments: whether they appreciate their office spaces or not. Preparing a survey with a few questions is essential. Their responses will reveal their thoughts and feelings. After working with several student groups, I’ve noticed a common mistake in questionnaire design—they are too obvious with their questions, simply asking, “How do you feel about this?” There’s a more effective approach.

Typically, to understand people’s thoughts and feelings, we might ask directly, “What do you think and feel?” This is akin to a doctor inquiring, “What is your disease? What kind of pill should I prescribe?” While straightforward, this method suits a doctor less concerned with patient recovery.

Asking directly also exposes the survey’s intent. Savvy respondents may realize our research goals, potentially skewing results by conforming or sabotaging the study. Some might claim they enjoy their work environment, while others might express dissatisfaction. However, few will be entirely candid, feeling more like researchers than participants.

Here’s an example of an ineffective survey structure:

Q1: Is your work environment comfortable?
  - Agree
  - Neutral
  - Disagree
Q2: Do you feel tired at the end of an office day?
  - Agree
  - Neutral
  - Disagree
Q3: Do you enjoy working in the office?
  - Agree
  - Neutral
  - Disagree

A skilled doctor, rather than directly asking about diseases, inquires about symptoms: “How often do you urinate?” or “Are you thirsty upon waking?” Similarly, in empirical computer science studies, we can engage respondents with hypothetical scenarios.

By asking respondents directly, we inadvertently shift our research responsibilities onto them. Our role is to determine if they enjoy their office space. We should observe their behavior, symptoms, and reactions to draw conclusions. Simply asking, “Do you feel comfortable?” suggests a lazy or inexperienced interviewer. Responding to such a generic question, respondents will have to put together their entire experience of being in the office, analyze it, make some conclusions, and then summarize them for us—this is the work we researchers have to do, not our interviewees.

Consider this revised questionnaire:

Q4: With a looming strict deadline, where would you 
prefer to work on a critical software module?
  - At home
  - In a café
  - In the office
Q5: When did you last feel exhausted at the end of 
an office day?
  - A few days ago
  - A few weeks ago
  - Don't remember
Q6: How would you rate the office coffee 
machine's quality?
  - Excellent
  - It's OK
  - Poor

The first two questions, Q4 and Q5, are situational, placing respondents in specific scenarios. We then interpret their reactions to deduce answers to our primary question: Do they like their work environments? This interpretation method should be clarified in the research paper.

Question Q6, while not situational, is superior to Q1-Q3. It avoids asking respondents to self-diagnose, subtly probing their opinions about office coffee machines. The responses indirectly indicate their overall satisfaction with the work environment.

In summary, avoid directly inquiring about illnesses; instead, ask about symptoms to discreetly pursue your research objective. This approach elicits more honest responses.

When the list of questions is ready, you can draw a table in your research paper, listing all questions on a vertical axis and possible answers on the horizontal one. Under each answer you mention the impact it makes to one of your research questions, for example:

This table clearly explains to readers of your research, why did you ask these questions and how the responses provided by the respondents helped you answer your research questions.