WERA1010: Reduction of Error in Rural and Agricultural Surveys
Statement of Issues and Justification
A quiet crisis has developed in the ability of agencies, sciences and businesses to do high quality surveys in the U.S, including those aimed at providing important information about rural and agricultural populations. In the last quarter of the 20th century, telephone surveys became the dominant way of doing general public surveys needed for estimating behaviors such as employment rates, effects of rural development initiatives on entrepreneurial efforts, and consumption of food products. During this time, the telephone also became the dominant mode for surveying opinions, e.g. community and employment satisfaction, desire for new products and services, and satisfaction with rural and urban development activities. Face-to-face interviews continue to be used for the nation's most critical national surveys (e.g. the monthly Currently Population Survey used to estimate employment rates and the USDA's Agricultural Resource Management Annual Survey of Farm operators) and mail surveys are used for surveys of list populations (participants in Extension programs whose names and addresses were available.)The telephone is now losing favor as a data collection methodology, because of low response rates, and greater use of cell phones, which has resulted in fewer households having landlines. For a variety of reasons, cell phones are not generally accessible for the conduct of general public surveys. In response to this trend, many surveyors have turned to the web as an alternative data collection strategy. The difficulty with this response is that only about 70% of the U.S. households have access to the web, and many who do have access seem unwilling to respond to web surveys. Those without web access have significantly less education, lower incomes, are less likely to be married, have less stable employment and differ in other significant ways (Rookey and Dillman, 2007).
Many private sector firms have increasingly turned to web only panels for data collection, whereby volunteers are sought and surveyed repeatedly in order to lower costs of doing surveys. It is statistically inappropriate to generalize results to any larger population, as is done through scientific means for random sample surveys. The nature of this problem is summarized elsewhere (ESOMAR, 2006). Government agencies, as well as private sector organizations are searching for viable alternatives to the telephone in order to maintain the accuracy of data sets while being able to take advantage of lower data collection costs offered by web survey.
In addition to the above problems Rural America now faces a unique data crisis because of the loss of the long form from the Decennial Census, which was the primary source of information for rural communities and counties. In 2000, a minimum of 1 in 6 households in every location in the U.S. were required to complete the long census form (as opposed to the short one that asks only for name, sex, race, ethnicity, marital status, housing ownership) in order to estimate with precision the percent of households with different levels of income, education, commuting status, and many other questions important to the formation of rural policy. The long-census form has been replaced by the American Community Survey for use after 2000. This survey obtains "long-form" information from a few hundred thousand households each year. However, from a statistical standpoint, it is necessary to accumulate data over several years (up to five) in order to make reliable estimates for these demographics in smaller rural communities. This means that detailed demographic information for a particular year, is no longer available, past 2000, for most rural counties, especially in the more sparcely populated west. This change in how demographic data are collected puts significant pressure on surveyors to develop methods that will make it possible to gather information that describes situations for a particular time or season (e.g. the poverty rate in a particular county for a given year) that will no longer be available from the US Census.
Since 2002, WERA 1001 and its predecessor Regional Project W-183 have been studying ways of improving survey methodologies for rural and agricultural surveys, including a replacement for the telephone. Members of WERA, who are now requesting continuation of this coordinating committee, have for many years, researched ways of improving mail, web, and telephone methods, including use in mixed-mode surveys, whereby some people are surveyed via one mode (e.g. mail) and others by a second mode (e.g. web or telephone). For ten years preceding 2002, this work was carried on under Western Regional Project W-183.
In 2005 work by WERA 1001 members commenced in what may be the most promising replacement for survey methods that are losing their effectiveness. The U.S. Postal Service now makes available a listing of postal delivery addresses, known as the Delivery Sequence File or DSF. WERA participants have been quite active in researching the possibilities this list provides as a replacement for telephone methods. Todd Rockwood, who joined WERA a year ago from Minnesota and his colleagues have done fundamental investigation of whether use of the postal list can provide demographically representative samples. Virginia Lesser from Oregon, and a long-time member of WERA and W-183 that preceded it, has investigated blending web and mail methods together to obtain improved responses in addition to comparing the quality of responses between the two modes (Lesser and Newton, 2007a). Building upon this work, Don Dillman, also a long-time member of WERA and W-183, is now researching new procedures for using the list to contact a representative general public population and encouraging them to respond by web as a means of reducing survey costs, while developing a procedure to replace the telephone methods advocated in his books (Mail and Telephone Surveys, 1978; Mail and Internet Surveys, 2000) which are now inadequate as a basis for resolving problems stemming from the telephone revolution of the 1990's.
Other work of WERA 1001 has positioned this committee to be particularly effective in developing a replacement strategy for the telephone. Several committee members have been studying the effects of visual vs. aural communication on how people answer survey questions, making this committee a leader in such research. For example, numerous experiments on how different visual layouts affect answers have been carried out by Oregon, Florida, Montana and Washington members of WERA-1001.The results of these experiments provide an essential foundation for attempting to develop effect procedures for using combined mail and web approaches as a possible replacement for telephone survey methods.
WERA 1001 members as well as members of its predecessor, W-183, have developed a long tradition of interacting intensively on a wide variety of specific survey design issues on response rate improvements (including, Cornell, Pennsylvania, Iowa, New Hampshire, replicating experiments and reporting these to professional audiences in ways that go beyond the individual state work. For example, in August 2007, four members of WERA (Lesser from Oregon, Israel from Florida, Swinford from Montana, and Dillman from Washington) presented papers at the annual meeting of the American Statistical Association in a session devoted to developing more effective open-ended questions (Lesser and Newton, 2007b; Swinford, 2007; Smyth and Dillman, 2007; Israel, 2007). This work needs to be continued, because of one of the most promising features of web surveying is to be able to improve the quality of open-ended questions, which were virtually lost for cost reasons in the telephone survey era. This builds upon previous work by WERA members Israel (2005) and Dillman (Christian and Dillman, 2004) that showed ways of improving the quality of answers in open-ended questions. We know of no other researchers in the U.S. who have done as much research aimed at improving the quality of open-ended questions as WERA has accomplished in the last two years.
Members of WERA and its W-183 predecessor have developed a procedure of working jointly on publications that reports results from replicating and extending one another's studies. One of these outcomes is an article being published in December 2007 in the scientific journal, Rural Sociology, which includes data from five WERA and W-183 states (New Hampshire, Idaho, Oregon, Pennsylvania and Washington). It confirms the importance of personalization for improving response rates in general public surveys, but questions its effectiveness in group-specific surveys. This work is quite relevant to the work now being done on findings ways of improving the effectiveness of an initial mail contact to complete a web survey, being done now by committee members. It's unlikely that these data would have been published had it not been for the replication of studies across states.
Renewal of WERA 1001 will allow current members, as well as new members, to continue and initiate new coordinated scientific work across states on the design and conduct of research to improve a variety of survey methods, including but not limited to effective alternatives to the telephone. The committee will also continue its work in topic focused surveys of importance to rural areas, e.g. environmental and rural recreation surveys by Gentner (National Marine Fisheries) Brown and his colleagues (Cornell), Robertson (New Hampshire) and Mertig (Middle Tennessee State University). The addition of Allen (Utah State) also adds a community development perspective to our work that will allow testing of ideas in another setting, important to rural people and places.
Back to Top