Time Sharing Experiments for the Social Sciences (TESS)

Introducing TESS Since 2001, TESS has been supporting general population experiments on behalf of investigators throughout the social sciences. Faculty and graduate students from the social sciences and related fields (such as law and public health) propose survey-based experiments. A comprehensive, on-line submission and peer review process screens proposals for the importance of their contribution to science and society. A diverse team of leading scholars assist the co-PI's in administering the review process. For successful proposals, TESS pays standard data collection and data dissemination costs. In other words, TESS is free for investigators. This strategy enables researchers to concentrate more on the production of ideas and less on acquiring the means to pay the often high fixed costs of conducting experiments with representative national samples. What are the overall goals of TESS? What sampling methodology is used for TESS surveys? What kinds of proposals are appropriate? Who pays for TESS? Who runs TESS? What is a general population experiment? What are advantages of general population experiments? Why is time-sharing beneficial? What are the overall goals of TESS? 1. to provide social scientists with a new opportunity for original data collection and discovery 2. to promote innovative experimentation in social science 3. to increase the precision with which fundamental social, political and economic dynamics are measured and understood 4. to increase the speed and efficiency with which advances in social scientific theory and analyses can be applied to critical social problems 5. to maximize financial efficiency by combining otherwise separate studies, thereby radically reducing the average cost per study 6. to provide an online collection of exemplary experiments for teachers and students who want to learn more about social science experimentation What sampling methodology is used for TESS surveys? To achieve a representative sample, we contract with NORC, which conducts surveys using its AmeriSpeak Panel. AmeriSpeak is a nationally representative, probability-based panel based on NORC’s National Sample Frame, an area probability sample funded and managed by NORC and used for several NORC studies. These other studies include the General Social Survey funded by the National Science Foundation and the Survey of Consumer Finances sponsored by the U.S. Federal Reserve Board in cooperation with the U.S. Treasury Department. For the recruitment methodology, A summary of the AmeriSpeak Panel sample design used for the TESS projects can be accessed here. While the AmeriSpeak Panel does include online and telephone interviewing, for all TESS surveys, data will be collected using online interviewing only. NORC will select the most representative sample available from panel members who have elected to participate in online surveys. The final data will be weighted to the appropriate population benchmarks for the target sample of interest in order to provide each investigator a representative sample. What kinds of proposals are appropriate? Proposals may come from any substantive area within any discipline in the social sciences so long as they utilize experimental designs and seek to make a valuable contribution to knowledge. In the most common experimental design in social science, different subjects are randomly assigned to groups that receive some different stimulus, and then differences between groups in some outcome are assessed. Designs that involve within-subject experimental manipulations are also acceptable. TESS does not consider proposals that involve presenting the same survey questions to all respondents and comparing non-assigned subgroups (for example, men vs. women, or lower vs. higher educated respondents). Experimental research often involves testing specific hypotheses, but this is not required of TESS proposals. Who pays for TESS? TESS is funded by the Social, Behavioral, and Economic Sciences Directorate of the National Science Foundation. Who runs TESS? Jeremy Freese of Stanford University and James Druckman of Northwestern University are the current Principal Investigators. A multidisciplinary team of Associate PI's assists Freese and Druckman in managing TESS. The team includes an accomplished and diverse roster of over 65 Associate PIs from across the social sciences. Team members span several generations and multiple disciplinary boundaries, and each member has established a reputation in his or her respective field. Most importantly, they share our enthusiasm for this project. What is a general population experiment? General population experiments allow investigators to assign representative subject populations to experimental conditions of their choosing. TESS uses Internet technology to present randomly selected respondents with experimental stimuli of some kind. These stimuli can take any form; thus far, general population experiments have used systematic variation in the information that is given to respondents, alterations in question wording, pictorial stimuli, and differing incentives and instructions, just to name a few possibilities. What are advantages of general population experiments? General Population Experiments are intended to combine the strengths of experimental and survey designs in supporting causal inferences in the social sciences. Surveys provide an efficient and well-studied means of gathering descriptive information about populations. There are, however, often substantial obstacles to drawing strong causal inferences from conventional survey data. Over the years, many have hoped that advances in statistical methods would help scholars use such survey data to "partial out" or control for all plausible rival interpretations of a potentially causal relationship. Despite massive advances in statistical methods over the years, few people are as optimistic today that statistics can solve all such causal inference problems. Instead, throughout the natural and social sciences employ experimental designs in order to combat the challenges of causal inference, experiments are widely regarded as the most decisive design for adjudicating competing hypotheses about what causes what. Social science has conducted a broad variety of different kinds of laboratory experiments. While these laboratory experiments provide strong tests of causal propositions, scientific audiences, policy makers, and the public sometimes request more than a causal demonstration. In many cases, science and society benefit from knowing that our laboratory observations survive exposure to myriad conditions outside of the lab. Moreover, some critics have questioned the extent to which the usual subjects in social science experiments resemble broader, more diverse populations (see, e.g., Sears 1986). General population experiments offer a powerful means for researchers to respond to such critiques. They allow researchers to assign large subject populations to experimental conditions of their choosing. TESS uses Internet technology to engage randomly selected respondents with randomly selected stimuli. By moving the possibilities for experimentation outside of the laboratory in this way, we can strengthen the internal validity of social science research and interest a much broader group of social scientists in the possibilities of experimentation. As such, general population experiments offer many advantages to social scientists. Laboratory experimenters, for example, can use general population experiments to show that observations generated in a laboratory can be replicated in very different conditions. They can also test new hypotheses that emerge from their work with smaller groups of subjects. Alternatively, investigators who use the internet to run experiments on "knowledge networks" (e.g., markets, which aggregate diffuse individual behavior into prices and social systems that aggregate many individual characteristics into social hierarchies) can reinforce their research agendas by using general population experiments to evaluate individual-level hypotheses that inevitably follow from network-level observations. Likewise, scholars can use general population experiments to clarify the causal implications of findings from conventional surveys. Additionally, for a wide range of scholars, general population experiments offer new opportunities to innovate. A special advantage of general population experiments is the broad and diverse subject pools that they allow researchers to contact. Such experiments are particularly effective at documenting differences in the status of causal hypotheses between the type of people who are usually selected for laboratory experiments and those who are not. Though not all social scientists require large and diverse subject populations to accomplish their research goals, many do. Finally, TESS also provides opportunities to strengthen and improve a wide range of measurement issues. For example, those interested in how to assess race and ethnicity in an increasingly diverse society can use experimental methods to understand how the method of data collection affects response attributes. Moreover, although the philosophy of early survey research was to attempt to create a social vacuum in which people could express their "true" beliefs and opinions, the more recent acknowledgment of attitudes, beliefs and preferences as a function of both the person and situation has led to an interest in the systematic study of how context alters the opinions and preferences that are expressed. Why is time-sharing beneficial? A critical element of the TESS research strategy is multiple studies from different disciplines sharing common observational platforms, all exploiting the inferential power and measurement efficiencies of experimental designs. This time-sharing on data collection platforms is the key to economic efficiencies of TESS. By distributing the costs of sampling and fielding over a large number of studies, the marginal cost of each study can be greatly reduced. The start-up costs for launching any kind of large data collection effort are substantial. Time-sharing allows social science to experience the considerable economies of scale that come from paying such start up costs only once. In this respect, time-sharing follows an established scientific tradition. The natural sciences, for example, promote progress in many areas by instituting time-sharing on expensive instruments (e.g., particle accelerators and telescopes). Time-sharing of university computer resources allowed many researchers to experience the benefits of computation before PC's became feasible and cost effective, and it is still used for many especially computing-intensive scientific applications. Our time-sharing strategy also promotes efficiency in data collection by collecting demographic information that all investigators can share. If the experiments were conducted independently, each investigator would use time to collect such data on their own - increasing redundancy and reducing overall efficiency. TESS provides other economic efficiencies by reducing waste. Most national surveys collect responses from some relatively large, round number of respondents. Because most surveys are not designed to test a small number of specific hypotheses, there end up being more cases than was actually needed for some purposes, and perhaps too few for others. The flexibility of our instruments allow people to roll on for as many or few cases as they need, and they must justify the sample size requested as part of the proposal. Flexibility in the instruments will allow more investigators to be served more efficiently.