Referenced from lesson Threats to Validity (Spring, 2021)

The issue

It seems like “false facts” are on the rise today. People share conclusions in the media – including social media – that may be a misrepresentation of the data. Graphic representations of data can often be misleading. To understand the factors that undermine our confidence in conclusions, some important questions to ask are:

  1. How was the sample that was surveyed selected?

  2. How many people were surveyed?

  3. Did the sample selected cover all represented in the population we want information on?

  4. What other factors could be at play in the situation that could impact the conclusion made?

Objective

Students will demonstrate understanding of the various types of threats to validity and their impact on conclusions made.

Procedure

With a partner, decide on a research question from a topic of your choosing. Check the topic with your teacher. Create a claim based on faulty sampling techniques which incorporate: Selection bias, Sample size problems, Sampling errors, Confounding variables

Produce

Create a presentation (could be Google Slides, poster, or be creative) that addresses:

  1. Your research topic & question

  2. Your faulty plan for selecting individuals for the study

  3. Be sure to incorporate all four threats listed in the procedure section

  4. Explain what the problems will be with the validity of your conclusion based on the threats you allowed in your sampling

  5. Explain how you could change your study to minimize these threats

Have fun – pretend to be terrible data scientists – costumes encouraged!

Rubric

Got it! Mostly Getting There Somewhat Nope

Statistical Question: Clearly outlines the research topic and statistical question. The question lends itself to many threats needing to be addressed.

15

13

11

4

0

Threats: Developed to clearly be a threat to validity. Includes a threat from each of the four types listed in assignment.

35

27

21

10

0

Conclusion: Explained how the threats to validity cause problems to the conclusion. Explained what changes should be made to minimize the threats.

40

31

24

12

0

Neatness of Product: Project is neat, organized. Clear that effort was put in.

10

9

6

3

0

(Project by Joy Straub)

These materials were developed partly through support of the National Science Foundation, (awards 1042210, 1535276, 1648684, and 1738598). CCbadge Bootstrap:Data Science by the Bootstrap Community is licensed under a Creative Commons 4.0 Unported License. This license does not grant permission to run training or professional development. Offering training or professional development with materials substantially derived from Bootstrap must be approved in writing by a Bootstrap Director. Permissions beyond the scope of this license, such as to run training, may be available by contacting contact@BootstrapWorld.org.