Survey Design: Meaning, Concepts, Planning

A survey is the most used abs best tool for data collection for any type of research. However, it must be properly designed to make each question relevant to the research question and objectives.

Let’s learn the meaning, concepts, and planning of survey design.

What is Survey Design?

Survey design is the process of preparing a complete plan of operations to be followed in conducting a survey and disseminating its intended results.

Specifically, it includes, among others, decisions on such factors as variables to be included in the survey (called survey variables), the method of data collection (whether by direct interview, telephone interview, or self-administered questionnaire), construction of questionnaire, organizing fieldwork, management of non-sampling error, data processing, and data analysis.

It seems obvious that the survey objectives covered under the survey design determine the sample design. In practice, the sample design must be developed as an integral part of the overall survey design.

Survey design and sample design are thus two interrelated concepts, and one is complementary to the other. Several decisions must be made to obtain a sample from a population. Each requires unique information.

Here we raise a few questions that must be addressed in the process of obtaining a sample.

7 Concepts To Clear Before Planning a Survey Design

Population

For sampling to be representative, the population from which the sample must be drawn is defined unambiguously.

No confusion should arise about whether the population consists of individuals, households, or families, or a combination of these.

Objectives

The objectives of the survey need to be clearly stated. A clear statement helps to select the appropriate study design, decide on the sample size, and prepare suitable survey instruments.

Parameters of Interest

Population parameters are summary descriptors (e.g., mean, variance, proportion, prevalence, etc.) of variables of interest in the population.

Statistics are descriptors of relevant variables computed from the sample data.

They are estimators of population parameters. Sample statistics are the basis of our inference about the population because they are the best estimates of the population parameters.

Sampling Frame

The sampling frame is the list of population units from which the sample is drawn.

The theory of probability sampling is heavily dependent on the existence of a good sampling frame. Ideally, a sampling frame is a list of units or groups of population units to be sampled.

To be useful, a sampling frame should be organized and arranged so that every unit occurs once and only once in the list, and no unit is excluded from the list.

Type of Sample

Whether a probability or non-probability sample is to be adopted must be decided in advance. A probability sample is feasible if the resource permits if a sampling frame is available.

Size of the sample

One of the most important problems in planning a sample survey is determining how large a sample is needed for the estimates to be reliable. The decision is important for several reasons.

Too large a sample involves huge costs, additional manpower, more materials, and prolonged time, while too small a sample invalidates the results. Then the question is: what is the optimum size of the sample?

The absolute size of a sample is much more important than its relative size. The eventual sample size is usually a compromise between what is ‘desirable’ and what is ‘feasible.’

Cost

Cost considerations influence decisions about the size and type of sample and data collection methods. Almost all studies have some budgetary constraints, which can lead to using a non-probability sample.

The probability sample incurs call-back costs, listing costs, and a variety of other costs that are not necessary when a non-probability sampling scheme is adopted.

Survey Design Planning and Structuring

Now let’s look at a 5 part or section survey design structure. These sections cover all stages of survey development and administration.

Each section has a series of questions intended as a guide to facilitate addressing the main issues so that ample data is collected for proper data analysis.

Section 1: Survey Groundwork

Do you have an external advisory group to assist you in the whole survey work? Non-experts benefit from expert advice, and experts benefit from peer review.
Do you need ethical approval? (if in doubt-ask, if you want to publish the results in peer-reviewed journals, they may require ethical approval). What arrangements will be made to get ethical approval?
What are the questions you wish to answer? What are the specific objectives of the survey to address these questions? Consider:
- What new information will this survey provide?
- What policy, program, funding, or other kinds of decisions will be made based on the survey findings?
- What data are relevant to the research questions, hypotheses, and decisions? What are the possible sources of these data in addition to the survey designed for the study in question?
- What are the geographical/political/demographic domains for the investigation? What might be the implications of each of these for interpreting the results and applying the findings?
It is very seldom true that a questionnaire is the only means available to gather data. What are other possible means of gathering relevant data to address your questions? What are the pros and cons of conducting this survey compared to other possible means of data collection?
What type of design will be used for the project? Do you want descriptive information, or do you want to compare groups? Have you arranged for help with the design and planning phase?
What type of survey will be conducted (e.g., telephone survey, self-administered, or face-to-face interview)? Consider the advantages and disadvantages of each survey method. Why will this type of survey be chosen over another type?
Has there been a thorough search and review of other relevant studies? Have you looked for existing tools with known reliability and validity?
Have you defined the target population for your survey? How will this target population be sampled (e.g., simple, stratified, systematic, focus group, convenience, or snowball)?
How many responses are needed? What steps will be taken to increase the response rate? How will a low response rate be handled? What kind of follow-up contacts might be taken to increase returns? How does your follow-up plan fit your promise of confidentiality and anonymity?]
Have you developed a timeline with deadlines and contingencies for piloting, revisions, and possible delays? Have you allowed time for administrative review, ethics approval (if needed), and inevitable delays? Have you shared this timeline with stakeholders?
What staffing will be needed to collect the responses, enter and analyze the data, and write the report? Who will these people be (staff, contract positions)?
What potential criticisms might be made of the survey and its administration? For example, people sometimes argue: “The sample was not representative because it didn’t include any of us.” Which criticisms might be plausible? How will you handle each of these possible criticisms?

Other points pertinent to the groundwork include, among others, relevance, feasibility, applicability, and urgency of the data needed.

Section 2: Instrument Development

Before beginning the development of the questionnaire/interview schedules, the objective of the survey must be very clearly articulated. Each question on the survey must relate to the objectives.

The analysis for each question must be considered, as each question is designed.

How does each question meet its objective? Does each provide information that is not otherwise available?
Can you use questions developed and tested in another project? If you intend to compare responses from two sources, are the questions the same?
Which question format are you going to use to collect data? Do you want numeric information, text information, or both?
For numeric questions, what type of data will be most useful? Do you need nominal, ordinal, interval, or ratio data for your planned analysis and tables? Do you want to leave space for comments?
Consider the possible responses to each question. Are response categories clear and mutually exclusive?
For each question, what might a non-response mean? How will non-responses be handled?
Are any questions going to be open-ended? What strategy will you use to analyze open-ended data?
At each draft, consider the questionnaire as a whole. Are the questions categorized in sections or sub-groups (e.g., demographic information, preferences, etc.)? Can the survey questions be organized in a better way to improve flow and maximize responses? For example, it is usually best to place sensitive questions at the end of a section rather than at the beginning of a survey.
Have the questions been verified by a group of content experts for clarity, intent, and appropriateness?
Have all important concepts and terms been given good working definitions that are clear and easy to understand? Has a naive layperson read and understood them?
Is the reading level appropriate for the group of content experts for clarity, intent, and appropriateness?
How long is your questionnaire? Does it look like it will take a long time or be difficult to do? Can any questions be removed? Are there lots of blank spaces, particularly on the first page? Or is the font large enough for the participants to read? Are all the pages numbered?
Have the questions been carefully sequenced? Ordered? Have the skipping patterns been checked for consistency?
Did you make sure there aren’t any spelling or grammatical errors in the questionnaires?
How will the questionnaire/interview schedule be piloted? How many people need to be in the pilot?

Section 3: Conducting the Survey

Does the cover page convey the rationale and objectives of the survey clearly and succinctly so that the target group can easily understand them? Does it make them want to complete the survey?
Does the cover page indicate who is responsible for the survey, why and how the information will be used, and who to contact for more information?
Does the cover page clearly state how issues associated with confidentiality and anonymity are handled? Does it indicate what will be done with the results?
How is the participants’ informed consent obtained?
For questionnaires, are there clear instructions? Do the instructions indicate how long it takes to complete? How to complete the questions, and what to do with the completed forms?
If you are using interviews, are interviewers well trained to ensure that the data are reliable and valid?
What will be done to help people with language barriers or disabilities, or any other shortcomings?
How will participants be able to access the results of the survey?

Section 4: Dealing with Data

Before you start:

Consider how the data are to be reported. Prepare sample tables and graphs for numeric data.
What information will be available? What information will not be available?
Consider possible response patterns. What decisions would be made based on each possible pattern?
What are additional data available to supplement your data? How much do these data support, confirm, or contradict your findings?
What possible weaknesses might there be in the data that could compromise strong conclusions?
What might additional information be needed to address these weaknesses?

Once you have the data:

How will the response data be entered? For interviews, will you collect tape recordings or interviewers’ notes? How will you deal with tape recordings? (Transcriptions? Time required? Cost?)
Do you have individuals skilled in using the computer analysis packages needed?
How will the data be analyzed? How long will it take to do the analyses? Do you have access to others to help with the analyses?
For numeric data, how will missing cases be handled? (For example, will the record be deleted? Will the entry be coded as ‘O’?
How will the data be handled once the survey and final reports are completed? What are the arrangements for archiving questionnaires?

Section 5: Disseminating the Results

Who will receive the results of the survey? What is the most appropriate method to present the results? Written reports, personal presentations, a telephone conference?
Will different forms of the presentation be prepared for the stakeholders and the public?
If there is a final report, does it include all the survey elements, including a copy of the survey instruments?
Is the report easily readable? Is it written and at the language level of the reader?
Will the survey be published in newsletters, annual reports, and other publications? (If the survey is published in a peer-reviewed journal, ethical approval may be one condition for publication).