Citation
Vogt, W. Paul et al. When to Use What Research Design. New York: The Guilford Press, 2012.
What is the author's argument?
In Chapters 1, 7, and 13 of When to Use What Research Design, the authors discuss the use, application, and analysis of survey design. As with previous chapters, they do not uphold one type of survey method over another, but instead use the chapters to explore options related to survey administration and analysis and assess their value to specific research questions.
Key Points
Regarding surveys, it is essential to ask: 1) if you can learn what you need to know by asking people 2) how should the survey be conducted? 3) Whether you intend to study change over time, and 4) which question formats will be effective
One of the distinguishing features of survey design, as opposed to most other forms of interview designs, is that surveys usually employ structured questions. These are often forced-choice questions in which the respondents select a pre-determined response. Surveys work best with structured questions but can also include unstructured or open-ended questions.
Challenges in survey design include participant dishonesty and memory.
The response rate for surveys should be at least 50%, but researchers should aim for the highest possible outcome.
Methods used to survey a population include face-to-face, telephone, and self-administered (on paper or electronically).
Whom should you survey and how many respondents do you need are the two basic questions of survey sampling.
Regarding ethics (consent, harm, and confidentiality/privacy), survey methods tend to have fewer issues due to the structured nature of the design. The chief issue concerns ensuring privacy/confidentiality. But issues can also arise if the researcher has significant influence over subjects which could be construed as pressure or coercion (i.e. a professor and their students).
Some Key Terms/Concepts:
Structure refers to the degree of control you want to exercise over your answer.
Social desirability bias – when respondents answer in a way that they think is socially acceptable.
Event-history analysis – an approach to studying change over time using only a cross sectional survey. Instead of being prospective (repeated surveying planned for the future), event-history analysis is retrospective, meaning participants are asked to remember biographical data.
Semantic Differential Scaling – a form of survey question format in which respondents are asked to locate their attitudes on a group of scales composed of opposing adjectives.
Likert Scale – the most widely used format for survey questions in which respondents are asked to answer given a series of options from “Strongly Agree” to “Strongly Disagree.” It is good for assessing the degree of agreement for a belief, policy, or practice.
The two main categories of survey sampling are probability and nonprobability. In probability sampling, each respondent has a known probability of being selected for inclusion in the study. In nonprobability sampling, the probability of inclusion is not known.
Probability sampling can be further divided into four categories: random, systematic, stratified, and cluster. Each lends itself to benefits. For example, random sampling maximizes a study’s external validity. For a complete breakdown, see page 140.
Nonprobability sampling should be used when you have no choice and when representativeness is not important to your research. The three most common types of nonprobability samplings are convenience samples (or opportunity samples), quota samples, and judgment samples (also called purpose samples). For a complete breakdown of subtypes and their application, see page 140.
Population statistics are called parameters.
Nonresponse bias arises from the fact that if a substantial number of those in your planned representative do not respond, those who do respond no longer constitute a representative sample. (It is therefore ideal to have a large sample size.)
Key Quotations
“Deciding on the mode you will use to administer your survey requires making tradeoffs. The methods that are cost effective (cheapest) and methods that are the most effective for answering your research question may not be the same.” (2)
“If you want to study change over time, you need to measure your variables at more than one time.” (23)
“Most survey researchers hope to make valid generalizations from samples to populations. If you are choosing a sample in order to generalize to a broader group, then the main issue is whether your sample is representative.” (121)
“As a researcher, you will want to design your survey for flexibility, both to make informed consent as meaningful as possible and to avoid alienating respondents with rigid rules about how they must answer questions.” (244)
Strengths and Weaknesses
When to Use What Research Design begins each chapter with a helpful outline of the chapter’s objectives, a summary of the chapter’s key points, and suggestions for further reading. As a result, there is a continuity and consistency to the chapters that I appreciate. The chapters discussing survey design and sampling were well organized and articulated. However, I found that the section concerning ethics in Chapter 13 may not have taken some factors into account. For example, the authors state harm is rarely an issue in survey design. To illustrate, they discuss a participant angered by the insinuation that the survey they were subjected to was about homosexuality. He was angry and the authors suggest the only harm was to his own ego. I think there are other ways in which surveys could be damaging – namely surveys containing questions that could be triggering for the respondent. For example, a survey about sexual assault given to an assault survivor or a survey about death given to a combat veteran with PTSD. These instances could pose a risk of psychological harm that is far greater than the experience noted.
How does this relate to your research?
Although, my personal research will most likely utilize archival or observational research, survey design is applicable to many studies concerning museum engagement. I administered many surveys as a former museum professional (through methods including face-to-face and web-based questionnaires). The results of these surveys provided helpful information for the development of future exhibitions and events. The chapters assigned in When to Use What Research Design are an extensive examination of the benefits and challenges of utilizing surveys and provide useful guidance for the construction of meaningful questionnaires.
What connections, if any, can you make to other authors?
The authors provide a useful framework for the development of surveys. They also insinuate the ways in which the structure of the design can create the potential for bias and coercion. Although the authors unfortunately do not spend significant time on this aspect, its insinuation ties survey design to a greater responsibility – one that is highlighted in Writing Community Change: Designing Technologies for Citizen Action by Jeffrey Grabill. Grabill argues that infrastructures (aka the standards we set in place) are powerful tools for social action. This understanding is relevant to the way we structure research and the questions we ask of target populations.