Why We Need To Think Critically About Data
Helen Lee Bouygues
For Bergstrom, the bigger takeaway is that people need to be aware of “new-school bullshit.”
Marketing teams are using the veneer of data to push weak or even plainly false narratives. Photo credit: Getty
Most critical thinking programs focus on issues of logic, emphasizing rationality and reasoned judgment. That’s an important approach.
But a new college-level course takes a slightly different tack, and the program provides people with tools to look critically at what is too often considered unassailable—data.
Titled “Calling Bullshit,” the course got its start in 2017 when two University of Washington professors launched a lecture series by the same name. In the set of talks, experts Jevin West and Carl Bergstrom covered the various ways in which news agencies, corporations, and even scholarly institutions sling, well, bullshit.
The lecture series turned into a university-sponsored course and soon had a moment of viral fame with a fleet of news stories. Although the headlines have died down, the professors continue to make a wealth of course materials available online including syllabus readings and case studies.
The “Calling Bullshit” program is both irreverent and informative. In one video, West uses a chart to brag about the “explosive growth” of their website. After a few minutes, Bergstrom interrupts his colleague with a smile, saying he’s “suspicious” of West’s data.
Bergstrom then points out that West’s growth chart doesn’t show all that much growth because the Y-axis starts at 100,000 rather than zero. In other words, the chart inflates the rate of development by using a high baseline.
But there’s an even bigger problem with the chart. As Bergstrom notes, the data measures “total unique visitors,” which by definition will overstate the number of new people on the site. “What do we expect to happen with total unique visitors?” Bergstrom asks, rhetorically. “It’s not gonna go down. So, of course, it’s going up.”
Bergstrom then gives a real-life example of a similar issue, showing a chart that Tim Cook, CEO of Apple, shared with the company’s shareholders in 2014. The chart measured “cumulative iPhone sales,” and it displays a decidedly uphill trend. But when Bergstrom overlays a graph showing quarterly sales, the story changes. “Apple’s in a slide,” Bergstrom explains. “He’s trying to hide it.”
For Bergstrom, the bigger takeaway is that people need to be aware of “new-school bullshit.” Marketing teams are using the veneer of data to push weak or even plainly false narratives, he says, and a growing number of organizations are “trying to pull something over on you using fancy graphs.”
The UW team argue that the solution is for people to gain the skill of “data reasoning,” and the website gives some helpful tips and insights. For instance, they argue that data graphics should rely on the “principle of proportional ink.”
Here’s their explanation of the principle: “When a shaded region is used to represent a numerical value, the area of that shaded region should be directly proportional to the corresponding value. In other words, the amount of ink used to indicate a value should be proportional to the value itself.”
While the principle of proportional ink might seem obvious, many designers ignore the rule when they’re creating together a chart or graphic. Practically speaking, this means that you should study a chart carefully and look to see if the axes have been tweaked in a way that makes the resulting visualization misleading.
As an example, the UW duo points out the chart below from Business Insider. The chart does not heed the principle, and so “the bars for The Diary of Anne Frank and for The Da Vinci Code differ in height by only a fraction of a percent, despite the fact that the latter has sold more than twice as many copies as the former.”
The team doesn’t just blame large companies. They also go after academics, and on the site, they unpack a study that made headlines in 2015 due to a chart that showed that certain genres of music like rap are more life-threatening for its practitioners than others.
As the Calling Bullshit team points out, the study uses a weak methodology. The authors compare genres of music without paying much attention to the sample size, and rap music is too young of a genre to measure its practitioners’ average life span against those in blues and jazz.
In the end, the experts don’t make an argument that’s all that much different than British prime minister Benjamin Disraeli, who once supposedly said: “There are three kinds of lies: lies, damned lies and statistics.”
But the humor — and timeliness — of the UW team’s efforts stand out among the dry academic tomes on data analytics, and their work is of incredible value to anyone invested in better thinking.
Helen Lee Bouygues is the president of the Reboot Foundation.