In part one of our Quality Fit for Purpose series we spoke about how data quality is dominating online research conversations. In this article, we continue with the subject and detail some common research tradeoffs.
In reality, we make quality tradeoffs all the time in research. Sampling itself is a compromise: to get absolute data accuracy we require a perfect census.
We regularly accept these tradeoffs:
- Sample size, coverage or time in field vs. cost
- Online vs. in-person or phone methodology
- Qualitative responses vs. quantitative data
- Weighting vs. achieving quotas
And these are some more nebulous tradeoffs we make:
- Asking all the questions we want vs. more engagement in a shorter survey
- Specific quotas vs. time available to finish
Source breadth vs. engagement rates - Breadth of frame vs. cost
- Time spent in QC vs. time for analysis
Sample selection tradeoffs
Today we have the option to make even more potential tradeoffs, especially in sample selection. Some sample sources are more expensive to recruit, keep and reward and we can make a cost tradeoff to omit those people. We can usually meet demographic quotas without them but miss the breadth of psychographic and lifestyle diversity that would improve frame coverage.
Re-examining the ‘Holy Laws’ of research
Using a tradeoff framework to think about quality means rethinking some of the ‘holy laws” of sampling and research. Some of these allow us to ‘tick a box’ and have a comfort level without true quality benefits.
- N=1000. Accuracy loss from stopping at 950 is minimal
- In-survey confirmation of demographics. Here we risk loss of engagement due to annoyance and fatigue
- Category lockouts. Dynata’s research shows almost no data inconsistencies without lockouts – and none after 12 weeks
- Removing all straightliners. There are false positives in even the most sophisticated methods
- Demographic quotas. Depending on the topic of the research, strict quotas on age, gender, income and education may be unnecessary
- Question wording precision. Wordiness decreases engagement.
Tradeoffs and Quality Fit for Purpose
The limit to this quality tradeoff is defined by ‘fit for purpose’ so the two concepts must go hand in hand.
To take an extreme example, a senior politician is caught up in a scandal and a journalist demands an interview. It’s 9 a.m. and the politician has four hours to decide how to respond. As a researcher you know what you can do in that time – and it’s not optimal.
- It takes an hour to write, program and test the survey – no pilots or second opinions
- With only 30 minutes to prepare the data, the questions must be few and simple
- Only people available between 10 a.m. and noon will be included
- Online is the only practical methodology and you’ll pay almost any price for sample
- You have to weigh the data because you’re short of some demographics.
You make many tradeoffs but deliver usable data on time – fulfilling the ‘fit for purpose’ requirement.
More information: Dynata and quality data.
Dynata is kennispartner van Daily Data Bytes. Word ook kennispartner en bereik onze lezers via web, socials en nieuwsbrief met jouw content. Neem contact op met Tjitske Buurman voor meer informatie.