Choosing a Method of Data Collection

I.       Choosing the right method of data collection is essential.  You cannot choose a method just because you like it or it sounds easy.  You must choose a method that is right for the study you are developing.

II.    Issues to consider when choosing a method of data collection.

A.    The goal and purpose of the research.

1.  As stated earlier, research is typically conducted to describe, explore, or explain.

2.  Some methods are better suited to some of these goals than others.  For example, experiments are excellent for investigating causal explanations, but are not well suited for descriptive or exploratory research.  Surveys and the analysis of existing data on the other hand are great for developing descriptions of groups and organizations or exploring relationships between variables.

B.     The unit of analysis and unit of observation.

1.   The unit of analysis in a study is the thing or things you want to know something about or draw conclusions about.  It is what you are studying or comparing.  In sociological research it may be individuals, couples, small groups, large organizations, institutions, cities, states, nations, cultures, etc. 

2.   The unit of observation is the thing or things you actually observe and from which you collect data.  The unit of observation may also be individuals, couples, small groups, large organizations, institutions, cities, states, nations, cultures, etc., but may also include objects and artifacts such as trash or furnishings or communication media such as TV advertisements or song lyrics. 

3.   The unit of analysis and unit of observation are often, but not always the same.  In fact, the phrase “unit of analysis” is often used to refer to both when they are the same. 

4.   Some methods are better suited to certain units of analysis and observation.  For example, both surveys and experiments tend to collect information from and about individuals, while secondary data analysis is often used to study larger social collectives like cities.  Content analysis, by definition, uses communication media as the unit of observation, although the unit of analysis in such studies is usually the social collective that produced communication.

C.     Types indicators to be observed and measured.

1.   What is being observed and measured in this study?

a.       The characteristics or features of individuals or groups.

b.      Attitudes, opinions, perceptions, and orientations of individuals or groups.

c.       Actions and behaviors of individuals or groups.

2.   What types of indicators will be used?

a.       Direct observations.

b.      Self reports.

c.       Secondary reports/existing records and data.

d.      Recorded communication

e.       Physical traces/artifacts.

3.   Methods vary as to what they can measure and the type of indicators they can address.  For example, experiments can employ direct observation, but surveys rely almost exclusively on self reports.  Analysis of existing data often relies on second-hand reporting, while content analysis relies on recorded communication.

D.    Change over time.

1.   Change over time is an essential part of almost all studies, but in some studies time itself is an important variable that must be measured.

a.       Time as an independent variable.

b.      Time as a dependent variable.

c.       Time as an intervening or control variable.

d.      Time order and causality.

2.   Cross-sectional and longitudinal research.

a.       Cross-sectional research measures phenomena at one point in time.

b.      Longitudinal research measures phenomena at different points in time.

                                                                                                        i.            Panel designs.

                                                                                                      ii.            Cohort designs.

                                                                                                    iii.            Trend studies.

c.       Approximating longitudinal designs.

3.   Most methods can address change over time, but some do so more effectively and efficiently than others.  For example, longitudinal research using existing data is often relatively easy since the groups and organizations that produce such data are collecting it over time as a part of their normal operations.  On the other hand, longitudinal research involving surveys can be costly, time consuming, and wrought with methodological difficulties such as attrition.

E.     Causality.

1.   The criteria for establishing causality revisited.

a.       Association/correlation.

b.      Time order.

c.       Controlling for alternative explanations.

2.   Experiments are explicitly designed to evaluate causality.  While other methods may be used to assess causality, they have more difficulty doing so.  For example, while cross-sectional surveys can demonstration association and possibly control for alternative explanations, they often have difficulty establishing time order.

F.      Ethics.

1.   It goes without saying that all social research, regardless of the method employed, must be ethical.

2.   Research should seek to avoid harming or exploiting human subjects.

a.       It should avoid physical, psychological, or emotional harm.

b.      It should avoid social, economic, or legal harm.

c.       It should avoid deception, coercion, and exploitation.

d.      It should avoid the use of captive and/or dependent populations whenever possible.

3.   Reducing risks and protecting subjects.

a.       Informed consent and voluntary participation.

b.      Protecting privacy, anonymity and confidentiality.

c.       Removing or reducing the threat of harm.

d.      Providing remedy and compensation.

e.       Institutional Review Boards.

4.   Some methods put subjects at greater risk of harm than others.  For example, experiments frequently require deception and manipulation and are seldom anonymous, while analysis of existing data and content analysis typically pose few risks to human subjects since they rely on previously collected or recorded information. 

G.    Available resources.

1.   You must consider the resources available for your study.

a.       Time.

b.      Money.

c.       Personnel

d.      Equipment.

e.       Data sources.

2.   Different methods require different resources.  While large scale surveys can be expensive, they are relatively time efficient.  On the other hand, content analysis is typically inexpensive, but can be time consuming.  Secondary data analysis can be quick and cheap.

III.   Some specific advantages and disadvantages of each method.

1.   Advantages and disadvantages of experiments.

a.       Advantages.

                                                                                                        i.            Almost all extraneous factors are controlled, so that change in the dependent variable is almost certainly due to change in the independent variable, making time order and causality relatively easy to demonstrate.

                                                                                                      ii.            Existing measurement instruments can be used to measure the dependent variable saving time and increasing reliability.

                                                                                                    iii.            Multiple and composite measurements of the dependent variable are possible, increasing validity and allowing the measurement of complex multidimensional dependent variables.

                                                                                                    iv.            Allows behavioral outcomes to be observed and measured.

                                                                                                      v.            Data are typically easy to code and quantify for statistical analysis.

                                                                                                    vi.            They are relatively safe, fairly easy to carry out, and generally time efficient.

                                                                                                  vii.            They can be relatively inexpensive, depending on how elaborate the design is and whether subjects are paid to participate.

b.      Disadvantages.

                                                                                                        i.            Since experiments are usually based on small random samples, problems arise if sampling is done incorrectly.  Also, small samples make inference to larger populations more difficult.

                                                                                                      ii.            Experiments generally take place in contrived settings, raising questions about their validity in the real world.

                                                                                                    iii.            Experiments require significant researcher involvement, raising questions about reactivity and researcher effect.

                                                                                                    iv.            Because experiments typically involve manipulating people and their environment, they can present numerous ethical problems.

                                                                                                      v.            If experiments are not carefully designed or require a great deal of time between the pre- and post-tests, numerous problems can occur that reduce their validity.

                                                                                                    vi.            They are typically limited to data by and about individuals or small groups.

2.   Advantages and disadvantages of surveys.

a.       Advantages.

                                                                                                        i.            They allow the collection of large amounts of data from large numbers of individuals in a relatively short period of time.

                                                                                                      ii.            Existing measurement instruments can be incorporated easily saving time and increasing reliability.

                                                                                                    iii.            Multiple and composite measurements are easy to incorporate increasing validity and allowing the measurement of complex and multidimensional concepts.

                                                                                                    iv.            Many variables can be measured simultaneously, allowing complex causal models to be investigated and facilitating statistical control.

                                                                                                      v.            Data are typically easy to code and quantify for statistical analysis.

                                                                                                    vi.            They can easily be administered to representative samples, allowing statistical inference to larger populations.

                                                                                                  vii.            Self-administering questionnaire and telephone surveys are relatively safe, fairly easy to carry out, and time efficient. (Face-to-face interviews can be more risky and time consuming.)

                                                                                                viii.            Self-administering questionnaire and telephone surveys can be conducted anonymously and pose few ethical risks. (Face-to-face interviews obviously cannot be anonymous.)

b.      Disadvantages.

                                                                                                        i.            They are limited to self-reported data. (Limited direct observations sometimes can be made in face-to-face interviews.)

                                                                                                      ii.            They generally are limited to individuals as the unit of observation.

                                                                                                    iii.            They do not allow the careful controls that experiments do, making it more difficult to demonstrate time order and to evaluate causal relationships.

                                                                                                    iv.            They have some difficulty addressing change over time.

                                                                                                      v.            They can be expensive if done on a large scale.

                                                                                                    vi.            They are dependent on representative samples, problems arise if sampling is done incorrectly or response rates are low. (Face-to-face interviews can significantly increase response rates.)

3.   Advantages and disadvantages of analyzing existing data.

a.       Advantages.

                                                                                                        i.            Secondary data analysis is unobtrusive and non-reactive.

                                                                                                      ii.            Depending on the nature and quality of the data, it can allow multiple and composite measurements, the simultaneous measurement of several variables, and statistical inference.

                                                                                                    iii.            It tends to be well suited for the analysis of change over time.

                                                                                                    iv.            It is typically the best source of data about large social collectives, but can provide data about individuals as well.

                                                                                                      v.            Since it relies on previously collected data, secondary analysis is usually relatively quick, easy, cost efficient, and safe.

                                                                                                    vi.            Secondary analysis presents few ethical problems.

b.      Disadvantages.

                                                                                                        i.            Secondary data analysis is only as reliable as the data being analyzed.

                                                                                                      ii.            Secondary analysis depends on being able to locate an existing data set that contains indicators of the concepts you want to measure.

                                                                                                    iii.            Since existing data are often about collectives rather than individuals, caution must be taken to avoid the ecological fallacy.

4.   Advantages and disadvantages of content analysis.

a.       Advantages.

                                                                                                        i.            Content analysis is unobtrusive and non-reactive.

                                                                                                      ii.            It is excellent for addressing broad cultural values and beliefs, especially as they change over time.

                                                                                                    iii.            It is relatively easy, cost efficient and safe, often being conducted in a library.

                                                                                                    iv.            Content analysis presents few ethical problems.

b.      Disadvantages.

                                                                                                        i.            It can be very time consuming.

                                                                                                      ii.            The quality of the research is affected by the quality of the data.

                                                                                                    iii.            The reliability and validity of coding can be difficult to demonstrate.

                                                                                                    iv.            The unit of observation is limited to previously "published" data, limiting its application to specific types of research questions.