Wow! Getting There Needs Improvement

Statistical question

The students developed a statistical question that lends itself to many threats needing to be addressed.

The students developed a statistical question that was somewhat conducive to developing a faulty research plan.

The students developed a question that was not suited for this particular project.

Selection bias

The research plan lays out clearly and specifically how information will be gathered from a non-representative sample of the population.

The research plan lays out how information will be gathered from a non-representative sample of the population, but is lacking in detail and specificity.

The research plan does not address how information will be gathered from a non-representative sample of the population.

Bias in the study design

The study is clearly biased, with “loaded” questions that lead to misrepresentation of true opinions.

The study includes some bias, but it is unclear how the plan will lead to misrepresentation of true opinions.

The study design does not include bias.

Poor choice of summary data

The research plan includes extreme outliers that will shift the results of the analysis in ways that do not represent the population as a whole.

The research plan includes some outliers, but they are perhaps not adequately extreme to shift the results and misrepresent the population.

The research plan does not include extreme outliers.

Confounding variables

The study overlooks factors that clearly influence a relationship, leading to invalid claims.

The study overlooks factors that might influence a relationship, perhaps leading to invalid claims.

The study does not include any confounding variables.

Concluding discussion

Students clearly articulated how all of the threats to validity cause problems to the study’s conclusions. Students explained what changes must be made in order to minimize the threats.

The discussion of how the threats caused problems is lacking in detail and evidence / analysis. Students explained the changes that must be made to minimize threats, but need to provide more explanation.

The concluding discussion is lacking in detail and evidence. It is unclear if the students understand how threats to validity influence a study. Students do not appear to understand how to revise the study to minimize threats.

These materials were developed partly through support of the National Science Foundation, (awards 1042210, 1535276, 1648684, and 1738598). Bootstrap by the Bootstrap Community is licensed under a Creative Commons 4.0 Unported License. This license does not grant permission to run training or professional development. Offering training or professional development with materials substantially derived from Bootstrap must be approved in writing by a Bootstrap Director. Permissions beyond the scope of this license, such as to run training, may be available by contacting contact@BootstrapWorld.org.