Improving Your Survey Data

Most people are pretty careful about what to believe on their social media feed, but do they apply the same care to their internet surveys at work? It’s an important issue. The clinical trials industry uses internet surveys for competitor analysis, customer satisfaction/feedback, or even to measure patient outcomes. Internet surveys are ubiquitous and overwhelming so it is easy to respond carelessly.

The purpose of this blog is to describe what careless responding is and how we can avoid and identify careless responses.

Careless responding

Careless responding refers to survey responses that are random or do not reflect the degree to which the construct is present in the respondent. Carelessness is increased when there is a lot of surveys, or when surveys are forced on us, not relevant to us, or force us to respond to items for which we have no opinion. (Boto et al. 12/21 ACT;Duamis & Raymond, 12/21 ACT; Galwicki ACT 11/11). Careless responding can be due to inattention (either partial or complete), fatigue, time pressure, or socially desirable responding. Careless responding is common, varying between 3% to 50%, depending on the context. Measurement problems begin to occur with as little as 10% careless responding (Meade and Craig 2012). The technical consequences of careless responding includes decreasing the reliability of measures, obscure validity, and increase the risk of Type 2 error.

Recommendations: Survey construction and sampling

• Make your surveys short (Dillman et al. 2009).
• Instructions have a small but significant impact
• Reverse coded items
• Instructed response items. (e.g. Fill in a 3 for response.’ Ward 2015)
• Self-reported items - directly asking respondents at the end of the survey whether they think their survey responses are of adequate quality for use in the study Ward (2015). “... self-report alone is not sufficient to identify careless responses.” (Meade & Craig p.8)
• Sample only those potential respondents for whom the survey will be relevant. Not only will this increase your response rate but also reduce careless responding, although you still may get people who want to peruse the survey to get information on your project.
• Don’t use forced responses unless you are sure the question is relevant and the respondent has the information. Still, if they don’t want to give you that information, you should let them progress through the survey.

Recommendations: Analytic procedures to detect careless responding

• response times—either long or short. Meade and Craig 2012 did not find an association between time and careless responding.
• consistent responding
• patterned responding
• abandonment

Conclusion—don’t believe everything you read on the internet

• The clinical trials must rely on internet if it is to remain responsive to industry changes and patient needs.
• The clinical trials industry is being impeded by noisy data from careless responding on internet surveys

Michael Howley PA-C, MBA, PhD, and Peter Malamis MBA