SURVEY ADMINISTRATION

  I.                 Distributing self-administering questionnaires via the mail.
                A.    The cover letter.
                                                1.      State the nature and purpose of study clearly and simply.
                                                2.      Tell respondents exactly what is expected of them and what they can expect in the questionnaire.
                                                3.      Request participation, making sure respondents are aware that their participation is voluntary.
                                                4.      Tell respondents if the questionnaire contains questions that might be troubling or sensitive, and tell them if they find any questions troubling they may skip them.
                                                5.      If you are providing a referral, put that information in your cover letter.
                                                6.      If the study is anonymous tell them so and what is being done to maintain anonymity.  If it is not anonymous, tell them why it is not and what is being done to
                             protect their confidentiality.
                                                7.      Provide an accurate time estimate.
                                                8.      If you are offering any kind of incentive for returning the questionnaire, discuss it in the cover letter.
                                                9.      Express your appreciation in advance.
                                            10.      Remember, always be polite and courteous, but to the point.
                B.    Instruction page.
                                                1.      Provide a general discussion of how answers are to be marked.
                                                2.      Discuss each type of question and how it is to be answered.
                                                3.      If there are large blocks of questions that only apply to certain groups, that should be pointed out.  Give instructions on how to follow contingency questions.
                                                4.      If a separate answer sheet is used, discuss how it is to be marked.
                                                5.      Tell them there also will be instructions in the body of the questionnaire.
                                                6.      Include instructions on how to return the completed questionnaire.
                                                7.      Incorporating instructions into the cover letter.
                C.    Return envelope and postage.
                                                1.      Make it easy for them to return the questionnaire.
                                                2.      Include a self-addressed stamped envelope.
                                                3.      Alternatives to the self-addressed envelope.
                D.    Practical matters.
                                                1.      Mailing options.
                                a.       Special delivery.
                                b.      First class.
                                c.       Bulk mail.
                                d.      Business return.
                                                2.      Postage regulations.
                                a.       Size and weight
                                b.      Packaging.
                                                3.      Timing.
                                a.       Avoid mailing surveys so that they arrive around holidays or vacation periods.
                                b.      Shortly after the first of the year is a good time.
                E.     Collecting questionnaires.
                                                1.      Previewing, numbering, recording, and filing returned questionnaires.
                                                2.      Monitoring responses rates.
                                a.   Response rates have to do with the proportion of surveys that get returned.
                                b.       Low response rates threaten the representativeness of samples.
                                c.      Generally, a 50% response rate is generally considered adequate if sample size was large enough. A 60% response rate is considered good. A response rate
                                      of 70% or more is great.
                                                3.      Improving response rates.
                                a.       The importance of appearance.
                                b.      Appeals and inducements.
                                c.       Keep things simple and clear.
                                                3.      Using follow-ups.
                                a.       Tracking returns with response graphs.
                                b.      Identifying non-responders.
                                c.       Letter only follow-ups.
                                d.      Letter and a new survey.
                                e.       Follow up phone calls.
                                f.      No more than 2 follow up mailings two or three weeks apart.

II.              Distributing self-administering questionnaires by hand.
                A.   Door to door delivery.
                B.   Distribution to groups.
                C.  Having others assist in the distribution.
                D.  General guidelines for hand delivery.
                                                1.      Always appear serious and professional with respondents.
                                                2.      Make sure oral explanations and instructions are clear.
                                                3.      Have a clear strategy for collecting surveys.
                E.   While hand delivery can increase response rates, it can be more costly and more time consuming, especially if large representative samples are used.
                F.   Anonymity, confidentiality, and informed consent.
 
III.     Developing and administering electronic/online questionnaires.
                A.  Most everything above about developing self-administering questionnaires applies to electronic/online questionnaires as well, but there are some unique issues.
                        1.  Questionnaire construction.
                                a.  Software.
                                        i.  Word processing and PDF software.
                                       ii.  Web authoring software.
                                       iii.  Specialized online survey software.
                                b.  General structure.
                                        i.  Single page document respondents will scroll through.
                                       ii.  Multi-page document with each section on a separate page.
                                       iii.  One question at a time.
                                c.  Answer types.
                                        i.  Radio button (select only one).
                                       ii.  Check boxes (select more than one).
                                      iii.  Drop down menu (for long lists of answer choices).
                                      iv.  Single and multi-line text boxes.
                                       v.  Numerical data boxes.
                                d.  Dealing with varying platforms and web browsers.
                                e.  Use of color, graphics, sound, and video.
                                f.  Cover page/introduction.
                                g.   Instructions and the importance of consistency.
                        2.  Administration/distribution.
                                a.  Emailing the questionnaire.
                                b.  Posting the questionnaire on the web.
                B. Advantages of electronic/online questionnaires.
                        1.  Collect large amounts of data quickly and inexpensively.
                        2.  Automation.
                                a.  Automating contingency questions and controlling navigation through the questionnaire.
                                b.  Checking and verifying responses.
                                c.  Automatic data entry.
                C. Disadvantages of electronic/online questionnaires.
                        1.  Requires subjects have access to and knowledge of computers and the internet.  Can bias the sample toward younger, more educated respondents.
                        2.  Technical problems and glitches.
                        3.  Limited control over who has access to and completes questionnaire.

IV.          Conducting face-to-face interviews.
                A.  Designing the interview schedule.
                                                1.      The interview schedule is developed before interviews are actually conducted. It is similar to the self- administering questionnaire except it is addressed to the      
                              interviewer.
                                                2      The interview schedule should contain:
                                a.       Instructions for the interviewer as to how to conduct the interview.
                                b.      A list of all the questions to be asked, instructions for asking them, and a list of the available answer categories.
                                                3      It might also contain:
                                a.       Information about permissible specifications, clarifications, and probes.
                                b.      Guidelines for dealing with problems that might come up during the interview.
                                                4      Questions in the interview schedule.
                                a.       The same types of questions used in a self-administering questionnaire can be used in an interview, but some may require slight modifications.
                                        i.         Closed-ended questions.
                                        ii.       Open-ended questions.
                                        iii.      Contingency questions.
                                        iv.     Matrix questions.
                                b.      As in the self-administering questionnaire, keep questions and answers simple and clear.
                                                5      Questions order.
                                a.       Putting demographic and "get-to-know-you" questions first.
                                b.      Put threatening or challenging questions near the end.
                                c.       Use logical groupings.
                                d.      Make the interview schedule logical, connected, and consistent.
                                                6      Marking and recording respondents answers on the interview schedule.
                                                7      Make sure the interview schedule is neat and easy to follow.
                B.  General guidelines for conducting interviews.
                                                1      Familiarize interviewers with the interview schedule.
                                                2.      Conduct practice interviews.
                                                3.      Maintain a pleasant appearance and demeanor.
                                a.       Dress professionally, but similar to interviewees.
                                b.      Be neat and clean.
                                c.       Be pleasant and friendly.
                                d.      Minimize personal qualities to reduce interviewer effect.
                                                4.      Be an interested, yet neutral listener.
                C. Specific guidelines.
                                                1.      Convincing the people to be interviewed.
                                a.       Others they know have participated.
                                b.      Their opinion is highly valued.
                                c.       The project is important.
                                d.      Personal appeals.
                                                2.      Establishing rapport.
                                a.       Being open and friendly.
                                b.      Establishing trust.
                                c.       Casual conversation.
                                                3.      Asking Questions.
                                a.       Follow the interview schedule exactly.
                                b.      Try to make the interview seem as much like natural conversation as possible. Make the transition and flow from one question to another smooth and logical.
                                                4.      Recording responses.
                                a.      Allowing them to come up with their own answer then checking the most appropriate answer on the interview schedule.
                                b.     Reading answer choices.
                                        i.         Reading a list of possible answers and having respondents stop the interviewer when correct answer is read.
                                        ii.       Reading all available choices then having respondent select the best one.
                                        iii.     Using reminder cards.
                                c.       Marking fixed choice answers.
                                d.      Recording open-ended responses.
                                                5.      Expressing thanks and appreciation.
                D.  Ethical issues.
                                                1.      Informed consent and voluntary participation.
                                                2.      Sensitive questions.
                                                3.      Anonymity and confidentiality.
                E.   Coordination and control of multiple interviewers.
                                                1.      Training.
                                a.       Explain the study.
                                b.      Offer general guidelines.
                                c.       Go through the interview schedule.
                                        i.        Ask for questions or problems.
                                        ii.      Provide probes and specifications.
                                d.      Demonstrate and practice interviewing.
                                                2.      Supervision and evaluation.
                                a.       Following up and checking.
                                b.      Contacting interviewees.
                F.   Advantages of face-to-face interviews over questionnaires.
                                                1.      More personal and perhaps more valid.
                                                2.      Better response rates.
                                                3.      Decreases the number of "don't know" answers.
                                                4.      May allow extra explanation and clarification.
                                                5.      Observations can be made.
                G.  Disadvantages of face-to-face interviews.
                                                1.      Lack of anonymity.
                                                2.      Interviewer effect and social desirability.
                                                3.      Coordination and control of multiple interviewers.
                                                4.      Reliability of multiple interviewers.
                                                5.       More time consuming and costly.
                                                6.      More risky.
 
 V.         Telephone interviewing.
                A. The increasing popularity of telephone interviewing.
                B. Computer assisted telephone interviewing.
                                                1.      Random digit dialing.
                                                2.      Teleprompting.
                                                3.      Automatic data entry.
                C. Advantages of telephone interviews.
                                                1.      Sampling through random digit dialing.
                                                2.      Easy, quick, safe, and inexpensive.
                                                3.      Can be anonymous.
                                                4.      Reduces interviewer effect.
                                                5.      Easier to supervise.
                D. Disadvantages of telephone interviewing.
                                                1.      Only people with phones who happen to be at home when you call can be in study.
                                                2.      Unlisted numbers and invasion of privacy.
                                                3.      Non-household numbers.
                                                4.      Sampling within households.
                                                5.      Bogus interviews and telephone solicitations.
                                                6.      Ease of termination.

VI.  Sampling

A.       The goal of sampling.

1.      Quantitative researchers typically use probability samples that allow them to generalize their findings to a large population. 

a.       Probability samples are constructed by ensuring that every possible combination of elements of a given size has an equal probability of being the selected sample. 

b.      Since they are based on probability, probability samples allow researchers to use statistics calculated from the sample to draw conclusions about the larger population.

c.       The concept of inference.

i.                     Populations and samples.

ii.                   Parameters and statistics.

iii.                  Estimation and hypothesis testing.

iv.                 Probability theory and the importance of probability samples.

v.                   Probability samples and sampling error.

2.      However, if it not possible or feasible to use proability sampling, quantitative researchers may rely on non-probability, recognizing that this reduces the generalizability of their findings.

 

VII.   Probability sampling.

A.     Probability samples are based on random selection in which every element in the population has an equal probability of being selected for the sample. 

B.     Types of probability samples.

1.      Simple random samples.

a.       In a simple random sample, a list of every element in the population is obtained or produced (the list is called a sampling frame) and elements are randomly selected ensuring that each element on the list has an equal chance of being chosen.

b.      Populations and sampling frames.

c.       Using random number tables to select elements.

d.      Using computers to randomly select elements.

2.      Systematic random sample.

a.       In a systematic random sample, a list of every element in the population is obtained or produced and elements are randomly selected by randomly selecting a starting point, then skipping a pre-determined number of cases (the sampling interval) to select the next case, and so on. 

b.      Determining a starting point using a random number table or computer generated random number.

c.       Calculating the sampling interval.

3.      Stratified random sample.

a.       In stratified random samples, the sampling frame is divided (or stratified) along selected criteria (such as race, age, and gender) then elements are selected from the stratified lists using either simple random or systematic random sampling.  The goal is to ensure proportional representation of the selected criteria.

b.      Stratifying the sampling frame.

c.       Selecting cases using simple random or systematic random sampling.

d.      Over-sampling and weighting.

4.      Multistage cluster designs.

a.       Unlike the three sampling designs discussed above, multistage cluster designs do not require that a sampling frame listing every element in the population of interest be produced or obtained up front.  Rather, it samples naturally occurring groups elements (you will need a list of the groups), sometimes at multiple levels, and then samples elements from within these groups.

b.      Identifying and sampling naturally occurring groupings.

i.                     Using multiple levels of naturally occurring groups.

i.                     Obtaining or producing sampling frames.

ii.                   Sampling groups and elements.

iii.                  Probability proportionate to size adjustments.

iv.                 Example:  Sampling high school seniors.

 

VIII.  Non-probability sampling.

A.     While quantitative researches tend to use probability sampling whenever possible, in some instances it is difficult, if not impossible, to do probability sampling.  In these instances, non-probability techniques may be used.

1.      Small or difficult to identify populations.

2.      Limited time and resources.

B.     Types of non-probability samples.

1.      Convenience samples.

2.      Quota samples.

3.      Snowball samples.

4.      Self-selected samples.

5.      Purposive or theoretical samples.

 

IX.  Doing sampling in quantitative research.

A.     Define your population and, if possible, obtain a sampling frame.

B.     Choose a sampling technique.

1.      Consider your population and the availability of a sampling frame.

2.      Consider feasibility and cost.

3.      Use probability sampling if at all possible.

C.     Determine sample size.

1.      Factors affecting sample size.

a.       Size and variability in the population.

b.      Degree of accuracy desired.

c.       Number of variables and type of analyses to be performed.

d.      Expected return rate.

e.       Calculating sample size.

2.      General rules for determining sample size.

a.       For populations under 1,000, sample 30%.

b.      For populations around 10,000 sample 10%.

c.       For populations over 100,000, sample 1.5%.

d.      For populations over 1,000,000, sample .2%.

e.       For populations over 10,000,000 sample .025%.

f.        Large populations require smaller sampling ratios, not smaller samples.

g.       Samples of 2500 have been shown to be adequate for even the largest of populations.

  D.      Draw your sample.