Students consider the concept of trust and testing — how do we know if a particular analysis is trustworthy?

Lesson Goals

Students will be able to…​ - Create a subset of data to verify that a given transformation works as-advertised, using attributes of the transformation and the dataset.

Student-facing Lesson Goals

  • Let’s learn how to test the trustworthiness of a data analysis.


🔗Confirming Analysis 30 minutes


Students learn how to create a Testing Table, which is small enough to reason about and can be used to test whether code does the right thing.


Samples are taken in Data Science and Computer Programming for different reasons. One of the main purposes of Data Science is to take a representative sample from a larger population, and use information from the sample to infer what’s true about the whole population.

In programming, we often extract a smaller Table from a larger one, for the purpose of testing that our code seems to do what it’s supposed to. In this lesson, we focus on the tasks of programmers, and consider best practices for setting up a Testing Table that helps us check our code.

  • Uber and Google are making self-driving cars, which use artificial intelligence to interpret sensor data and make decisions about whether a car should speed up, slow down, or slam on the brakes. This AI is trained on a lot of sample data, which it learns from. What might be the problem if the sample data only included roads in California?

  • Why might it be a bad thing to only test medicines on men (or only on women), before prescribing them to the general public?

Testing Matters!

A good Testing Table should be representative of the population, and relevant to what’s being analyzed. A good Testing Table should have…​

  • At least the columns that matter — whether we’ll be ordering or filtering by those columns.

  • Enough rows to include different circumstances that are relevant to the task at hand. For instance, if our code is supposed to extract certain cats from the animals table, our Testing Table should include at least one animal that’s not a cat.

  • Rows that aren’t already sorted, if our analysis is supposed to sort for us.

Data scientists usually think in terms of samples that best serve the purpose of performing inference: Samples should be representative of the entire population, and large enough to get us fairly close to the truth about that population.

Programmers need to think in terms of Testing Tables that best serve the purpose of verifying that their code does what it’s supposed to: The Tables should be designed to call attention to any imperfections in the code’s instructions.


Testing Tables can also be used to verify that a certain analysis is correct. Code that filters a table to show only cats can’t be verified with a Testing Table that already has only cats. (Why not?)

Code that shows only the kittens…​ sorted in ascending order by weight must be verified by a Table containing cats, non-cats, old and young cats…​ and rows that aren’t already sorted!

  • Someone hands you a function called fixed-cats and claims that, given a table of animals, it produces a table with only fixed cats.

  • Do you trust it? How could you test it?

  • Which animals would you use in a Testing Table?

  • Complete “Trust, but verify …​”.

  • Open the Trust but Verify Starter File. There are 3 versions of fixed-cats. Are they all correct? If not, which ones are broken?

  • Turn to “Trust, but verify…” (2). Using the same Starter File, construct a Testing Table and figure out which (if any) of the functions are correct!


Complex analysis has more room for mistakes, so it’s critical to think about a Testing Table that allows us to trust that our code really does what it’s supposed to!

How would you check whether or not a facial recognition system was equally accurate for everyone?

🔗When AI isn’t Intelligent…​ 20 minutes


Law enforcement in many towns has started using facial-recognition software to automatically detect whether someone has a warrant out for their arrest. A lot of facial-recognition software, however, has been trained on sample data containing mostly white faces. Why might this be a problem?



Discuss the article and/or video, revisiting the following questions:

  • What are some concerns that experts and activists have raised about Artifical Intelligence?

  • What are some solutions that would address these concerns?

  • How would you test whether or not a facial recognition system was equally accurate for everyone?

These materials were developed partly through support of the National Science Foundation, (awards 1042210, 1535276, 1648684, and 1738598). CCbadge Bootstrap by the Bootstrap Community is licensed under a Creative Commons 4.0 Unported License. This license does not grant permission to run training or professional development. Offering training or professional development with materials substantially derived from Bootstrap must be approved in writing by a Bootstrap Director. Permissions beyond the scope of this license, such as to run training, may be available by contacting