In 2006, Douglas Coupland wrote a book called JPod about a group of employees at a video game company who are assigned to work together due to a computer glitch in a human resources algorithm. The algorithm is thus the hand of God that sets the events in the book in motion, and it is typical of Coupland's brand of fiction that it appears to be a witty observation of what modern life is increasingly like.
Cathy O'Neal, in Weapons of Math Destruction, describes how algorithms increasingly control our lives: the news we see, the jobs we might be hired or fired from, how prisoners are sentenced and granted parole, whether or not we qualify for a loan. Moreover, she makes the case convincingly that the sometimes bizarre results of algorithms gone awry disproportionately affect the lives of the poor and disadvantaged. After all, those with wealth and influence are better able to connect with actual people when applying for a job or a loan; those without it are mere data points, and some data points have to be at the bottom. O'Neal calls algorithms that lead to increased inequality "weapons of math destruction", or WMDs.
The strangest part of all of this from a scientific perspective is how often algorithms are used as a modern replacement for phrenology. Their complexity, which in scientific terms makes them less useful, is instead used to argue for their infallibility. A teacher who is fired because of their low score according to a "value-added" model can't very well argue that the algorithm is mistaken if neither they nor their superiors understand how it works. The algorithm itself has no way to be evaluated, to see if its results predict the real world.
Of course, the real world is complex and so sometimes a model must be, too. But in science, every model is known to be false; the idea is to improve it by continuously incorporating new data. Without this feedback loop, it's impossible to tell good algorithms from bad. As O'Neal argues, the incentive behind a Weapon of Math Destruction is not to produce better models but to give the appearance of creating greater value by using cutting-edge techniques. Complexity, or simple opaqueness, is a weakness for creating a scientifically better model, but it's a strength in marketing it.
O'Neal provides a compelling narrative that illustrates the real human consequences of blind trust in big data. It's an argument for comprehension, for responsibility, and for compassion.