Insensitivity to sample size

The insensitivity to sample size bias arises because humans often rely on heuristics or mental shortcuts when assessing statistical data. They focus more on the content or story presented by the data rather than the statistical reliability. The representativeness heuristic, where people judge probabilities based on how much one event resembles another, often overrides considerations of how the reliability of that information might change with different sample sizes.

• Suppose a researcher presents data showing a high incidence of a rare disease from a small-town hospital. Observers may incorrectly assume that this is indicative of a larger trend, ignoring the fact that small sample sizes can easily lead to statistical anomalies.
• In marketing, a company might test a new product with a small focus group and then prematurely assume widespread market success from positive feedback because they underestimate the importance of a larger, more diverse sample size.

While insensitivity to sample size is a well-documented bias, some critics argue that in real-world decision-making contexts, individuals may have constraints such as time and resources that necessitate the use of heuristics. There is also debate about how frequently this bias significantly impacts high-stakes decisions versus smaller scale or everyday scenarios.