Sample Size Constraints, Non-Response
Effects of Small Sample Size
In the formula, the sample size is directly proportional to Z-score and inversely proportional to the margin of error. Consequently, reducing the sample size reduces the confidence level of the study, which is related to the Z-score. Decreasing the sample size also increases the margin of error.
In short, when researchers are constrained to a small sample size for economic or logistical reasons, they may have to settle for less conclusive results. Whether or not this is an important issue depends ultimately on the size of the effect they are studying. For example, a small sample size would give more meaningful results in a poll of people living near an airport who are affected negatively by air traffic than it would in a poll of their education levels.
Effect of Large Sample Size
There is a widespread belief that large samples are ideal for research or statistical analysis. However, this is not always true. Using the above example as a case study, very large samples that exceed the value estimated by sample size calculation present different hurdles.
The first is ethical. Should a study be performed with more patients than necessary? This means that more people than needed are exposed to the new therapy. Potentially, this implies increased hassle and risk. Obviously the problem is compounded if the new protocol is inferior to the traditional method: More patients are involved in a new, uncomfortable therapy that yields inferior results.
The second obstacle is that the use of a larger number of cases can also involve more financial and human resources than necessary to obtain the desired response.
In addition to these factors, there is another noteworthy issue that has to do with statistics. Statistical tests were developed to handle samples, not populations. When numerous cases are included in the statistics, analysis power is substantially increased. This implies an exaggerated tendency to reject null hypotheses with clinically negligible differences. What is insignificant becomes significant. Thus, a potential statistically significant difference in the ANB angle of 0.1° between the groups cited in the previous example would obviously produce no clinical difference in the effects of wearing an appliance.
When very large samples are available in a retrospective study, the researcher needs first to collect subsamples randomly, and only then perform the statistical test. If it is a prospective study, the researcher should collect only what is necessary, and include a few more individuals to compensate for subjects that leave the study.
In designing a study, sample size calculation is important for methodological and ethical reasons, as well as for reasons of human and financial resources. When reading an article, the reader should be on the alert to ascertain that the study they are reading was subjected to sample size calculation. In the absence of this calculation, the findings of the study should be interpreted with caution.
An appropriate sample renders the research more efficient: Data generated are reliable, resource investment is as limited as possible, while conforming to ethical principles. The use of sample size calculation directly influences research findings. Very small samples undermine the internal and external validity of a study. Very large samples tend to transform small differences into statistically significant differences – even when they are clinically insignificant. As a result, both researchers and clinicians are misguided, which may lead to failure in treatment decisions.
BA lot of things can go wrong in a survey. One of the most important problems is non-response. It is the phenomenon that the required information is not obtained from the persons selected in the sample.
The consequences of non-response
One effect of non-response is that is reduces the sample size. This does not lead to wrong conclusions. Due to the smaller sample size, the precision of estimators will be smaller. The margins of error will be larger.
A more serious effect of non-response is that it can be selective. This occurs if, due to non-response, specific groups are under- or over-represented in the survey. If these groups behave differently with respect to the survey variables, this causes estimators to be biased. To say it in other word: estimates are significantly too high or too low.
Example: surveys of Statistics Netherlands
Selective non-response is not uncommon. It occurs in a number of surveys of Statistics Netherlands. A follow-up study of the Dutch Victimization Survey showed that persons, who are afraid to be home alone at night, are less inclined to participate in the survey. In the Dutch Housing Demand Survey, it turned out that people who refused to participate, have lesser housing demands than people who responded. And for the Survey of Mobility of the Dutch Population it was obvious that the more mobile people were under-represented among the respondents.