Threats to Validity

Students are introduced to the concept of validity, and a number of possible threats that might make an analysis invalid.

As good Data Scientists, the staff at the animal shelter are constantly gathering data about their animals, their volunteers, and the people who come to visit.

But just because they have data doesn’t mean the conclusions they draw from it are correct!

In the example of the cat-owning survey, the claim that “people prefer cats to dogs” is invalid because the data itself wasn’t representative of the whole population (of course cat-owners are partial to cats!).

Data Scientists have several major Threats to Validity to worry about:

1) Selection bias - Data was gathered from a sample that is not representative of the population.

2) Bias in the study design - Data was gathered in such a way that it influenced the results, for example, researchers might:

3) Poor choice of summary data - Even if the selection is unbiased, sometimes outliers are so extreme that they make the mean completely useless at best - and misleading at worst. For example:

4) Confounding variables - Sometimes there’s an unaccounted for variable that is lurking in the background, influencing both of the variables we are studying and confusing the relationship between them. For example:

And there are many other threats to validity out there!

Life is messy, and there are always threats to validity.

Data Science is about doing the best you can to minimize those threats, and to be upfront about what they are whenever you publish a finding.

When you do your own analysis, make sure you include a discussion of possible threats to validity!

Why is it important to consider potential threats to validity?

Students are asked to consider the ways in which statistics are misused in popular culture, and become critical consumers of some statistical claims. Finally, they are given the opportunity to misuse their own statistics, to better understand how someone might distort data for their own ends.

You have already seen a number of ways that statistics can be misused:

1) Using the wrong measure of center with heavily-skewed data

2) Using a correlation to imply causation

3) Incorrect Interpretations of a visualization, which try to trick people who don’t know how to read charts and graphs. For example:

There are many other ways to mislead the audience, including:

4) Intentionally using the wrong chart - Suppose someone was asked to prepare a report on the demographics of the people holding positions of power in their city government. If the city had a significant Black population, and no Black elected officials, it should be cause for further investigation. But, if someone were trying to avoid addressing the issue, they might opt to display a pie chart (hiding that lack of representation) instead of displaying a bar chart (that would show an empty bar) in hopes that nobody would even notice the issue! Note: Pie charts could be used responsibly for this same scenario if a pie chart displaying the demographics of the city’s population was presented alongside a pie chart of the demographics of the city’s elected officials!

5) Changing the scale of a chart - Changing the y-axis of a scatter plot can make the slope of the regression line seem smaller ("look, that line is basically flat anyway!") or larger ("look how quickly things have changed!").

With all the news being shared through newspapers, television, radio, and social media, it’s important to be critical consumers of information!

Students are confronted with the concept of outliers: data points that stray far from the rest of the data and appear to confound observed patterns and groupings. Data Scientists take the decision of whether or not to keep outliers very seriously, as there can be profound implications for validity.

Suppose we survey the heights of 12 year olds, and almost all values are clustered between 50-70in. There’s a very low outlier, however, at 6in.

This typo could throw off our analysis completely! This one data point will destroy the mean, forcing us to use a different measure of center even if the rest of the data is symmetric.

"Junk" data is harmful, because it can drastically change our results! If we blindly keep every outlier, it can become a serious threat to the validity of our analysis!

Suppose we survey the number of minutes it takes for fans to find their seats at a stadium, and almost all values are clustered between 4-16 minutes. There’s a very high outlier, however, at 35 minutes.

If we choose to remove or keep an outlier without thinking carefully, it can become a serious threat to the validity of our analysis!

As a data scientist, an outlier is always a reason to look closer. And whether you decide to keep or remove it from your dataset, make sure you explain your reasons in your write-up!

These points are called unusual observations. Unusual observations in a scatter plot are like outliers in a histogram or dot plot, but more complicated because it’s the combination of x and y values that makes them stand apart from the rest of the cloud.

Sometimes unusual observations are just random. Felix seems to have been adopted quickly, considering how much he weighs. Maybe he just met the right family early, or maybe we find out he lives nearby, got lost and his family came to get him. In that case, we might need to do some deep thinking about whether or not it’s appropriate to remove him from our dataset.

Sometimes unusual observations can give you a deeper insight into your data. Maybe Felix is a special, popular (and heavy!) breed of cat, and we discover that our dataset is missing an important column for breed!

Sometimes unusual observations are the points we are looking for! What if we wanted to know which restaurants are a good value, and which are rip-offs? We could make a scatter plot of restaurant reviews vs. prices, and look for an observation that’s high above the rest of the points. That would be a restaurant whose reviews are unusually good for the price. An observation way below the cloud would be a really bad deal.