Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (28 page)

In this march through a virtual lifetime, we’ve visited school and college, the courts and the workplace, even the voting booth. Along the way, we’ve witnessed the destruction caused by WMDs. Promising efficiency and fairness, they distort higher education, drive up debt, spur mass incarceration, pummel the poor at nearly every juncture, and undermine democracy. It might seem like the logical response is to disarm these weapons, one by one.

The problem is that they’re feeding on each other. Poor people are more likely to have bad credit and live in high-crime neighborhoods, surrounded by other poor people. Once the dark universe of WMDs digests that data, it showers them with predatory ads for subprime loans or for-profit schools. It sends more police to arrest them, and when they’re convicted it sentences them to longer
terms. This data feeds into other WMDs, which score the same people as high risks or easy targets and proceed to block them from jobs, while jacking up their rates for mortgages, car loans, and every kind of insurance imaginable. This drives their credit rating down further, creating nothing less than a death spiral of modeling. Being poor in a world of WMDs is getting more and more dangerous and expensive.

The same WMDs that abuse the poor also place the comfortable classes of society in their own marketing silos. They jet them off to vacations in Aruba and wait-list them at Wharton. For many of them, it can feel as though the world is getting smarter and easier. Models highlight bargains on prosciutto and chianti, recommend a great movie on Amazon Prime, or lead them, turn by turn, to a café in what used to be a “sketchy” neighborhood. The quiet and personal nature of this targeting keeps society’s winners from seeing how the very same models are destroying lives, sometimes just a few blocks away.

Our national motto, E Pluribus Unum, means “Out of Many, One.” But WMDs reverse the equation. Working in darkness, they carve one into many, while hiding us from the harms they inflict upon our neighbors near and far. And those harms are legion. They unfold when a single mother can’t arrange child care fast enough to adapt to her work schedule, or when a struggling young person is red-lighted for an hourly job by a workplace personality test. We see them when a poor minority teenager gets stopped, roughed up, and put on warning by the local police, or when a gas station attendant who lives in a poor zip code gets hit with a higher insurance bill. It’s a silent war that hits the poor hardest but also hammers the middle class. Its victims, for the most part, lack economic power, access to lawyers, or well-funded political organizations to fight their battles. The result is widespread damage that all too often passes for inevitability.

We cannot count on the free market itself to right these wrongs. To understand why, let’s compare WMDs to another scourge our society has been grappling with, homophobia.

In September of 1996, two months before his reelection,
President Bill Clinton signed the Defense of Marriage Act. This law, defining marriage as between one man and one woman, promised to firm up support for the president in conservative patches of battleground states, including Ohio and Florida.

Only a week later, the
tech giant IBM announced that it would provide medical benefits to the same-sex partners of its employees. You might wonder why Big Blue, a pillar of the corporate establishment, would open this door and invite controversy when a putatively progressive American president was moving in the opposite direction.

The answer has to do with the bottom line. In 1996, the Internet gold rush was just taking off, and IBM was battling for brainpower with Oracle, Microsoft, Hewlett-Packard, and a host of start-ups, including Amazon and Yahoo. Most of those other companies were already providing benefits to same-sex partners and attracting gay and lesbian talent. IBM could not afford to miss out. “
In terms of business competitiveness, it made sense for us,” an IBM spokesperson told
BusinessWeek
at the time.

If we think about human resources policies at IBM and other companies as algorithms, they codified discrimination for decades. The move to equalize benefits nudged them toward fairness. Since then, gays and lesbians have registered impressive progress in many domains. This progress is uneven, of course. Many gay, lesbian, and transgender Americans are still victims of prejudice, violence, and WMDs. This is especially true among poor and minority populations. Still, as I write this,
a gay man, Tim Cook, is the chief executive of
Apple, the most valuable company on earth. And if he so chooses, he has the constitutional right to marry a man.

Now that we’ve seen how corporations can move decisively to right a wrong in their hiring algorithms, why can’t they make similar adjustments to the mathematical models wreaking havoc on our society, the WMDs?

Unfortunately, there’s a glaring difference. Gay rights benefited in many ways from market forces. There was a highly educated and increasingly vocal gay and lesbian talent pool that companies were eager to engage. So they optimized their models to attract them. But they did this with the focus on the bottom line. Fairness, in most cases, was a by-product. At the same time, businesses across the country were starting to zero in on wealthy LGBT consumers, offering cruises, happy hours, and gay-themed TV shows. While inclusiveness no doubt caused grumbling in some pockets of intolerance, it also paid rich dividends.

Dismantling a WMD doesn’t always offer such obvious payoff. While more fairness and justice would of course benefit society as a whole, individual companies are not positioned to reap the rewards. For most of them, in fact, WMDs appear to be highly effective. Entire business models, such as for-profit universities and payday loans, are built upon them. And when a software program successfully targets people desperate enough to pay 18 percent a month, those raking in the profits think it’s working just fine.

The victims, of course, feel differently. But the greatest number of them—the hourly workers and unemployed, the people dragging low credit scores through life—are poor. Prisoners are powerless. And in our society, where money buys influence, these WMD victims are nearly voiceless. Most are disenfranchised politically. Indeed, all too often the poor are blamed for their poverty, their bad schools, and the crime that afflicts their neighborhoods. That’s why few politicians even bother with antipoverty strategies. In the common view, the ills of poverty are more like a disease, and the effort—or at least the rhetoric—is to quarantine it
and keep it from spreading to the middle class. We need to think about how we assign blame in modern life and how models exacerbate this cycle.

But the poor are hardly the only victims of WMDs. Far from it. We’ve already seen how malevolent models can blacklist qualified job applicants and dock the pay of workers who don’t fit a corporation’s picture of ideal health. These WMDs hit the middle class as hard as anyone. Even the rich find themselves microtargeted by political models. And they scurry about as frantically as the rest of us to satisfy the remorseless WMD that rules college admissions and pollutes higher education.

It’s also important to note that these are the early days. Naturally, payday lenders and their ilk start off by targeting the poor and the immigrants. Those are the easiest targets, the low-hanging fruit. They have less access to information, and more of them are desperate. But WMDs generating fabulous profit margins are not likely to remain cloistered for long in the lower ranks. That’s not the way markets work. They’ll evolve and spread, looking for new opportunities. We already see this happening as mainstream banks invest in peer-to-peer loan operations like Lending Club. In short, WMDs are targeting us all. And they’ll continue to multiply, sowing injustice, until we take steps to stop them.

Injustice, whether based in greed or prejudice, has been with us forever. And you could argue that WMDs are no worse than the human nastiness of the recent past. In many cases, after all, a loan officer or hiring manager would routinely exclude entire races, not to mention an entire gender, from being considered for a mortgage or a job offer. Even the worst mathematical models, many would argue, aren’t nearly that bad.

But human decision making, while often flawed, has one chief virtue. It can evolve. As human beings learn and adapt, we change, and so do our processes. Automated systems, by contrast,
stay stuck in time until engineers dive in to change them. If a Big Data college application model had established itself in the early 1960s, we still wouldn’t have many women going to college, because it would have been trained largely on successful men. If museums at the same time had codified the prevalent ideas of great art, we would still be looking almost entirely at work by white men, the people paid by rich patrons to create art. The University of Alabama’s football team, needless to say, would still be lily white.

Big Data processes codify the past. They do not invent the future. Doing that requires moral imagination, and that’s something only humans can provide. We have to explicitly embed better values into our algorithms, creating Big Data models that follow our ethical lead. Sometimes that will mean putting fairness ahead of profit.

In a sense, our society is struggling with a new industrial revolution. And we can draw some lessons from the last one. The turn of the twentieth century was a time of great progress. People could light their houses with electricity and heat them with coal. Modern railroads brought in meat, vegetables, and canned goods from a continent away. For many, the good life was getting better.

Yet this progress had a gruesome underside. It was powered by horribly exploited workers, many of them children. In the absence of health or safety regulations, coal mines were death traps. In 1907 alone, 3,242
miners died. Meatpackers worked twelve to fifteen hours a day in filthy conditions and often shipped toxic products. Armour and Co. dispatched cans of rotten beef by the ton to US Army troops, using a layer of boric acid to mask the stench. Meanwhile, rapacious monopolists dominated the railroads, energy companies, and utilities and jacked up customers’ rates, which amounted to a tax on the national economy.

Clearly, the free market could not control its excesses. So after journalists like Ida Tarbell and Upton Sinclair exposed these and other problems, the government stepped in. It established safety protocols and health inspections for food, and it outlawed child labor. With the rise of unions, and the passage of laws safeguarding them, our society moved toward eight-hour workdays and weekends off. These new standards protected companies that didn’t want to exploit workers or sell tainted foods, because their competitors had to follow the same rules. And while they no doubt raised the costs of doing business, they also benefited society as a whole. Few of us would want to return to a time before they existed.

How do we start to regulate the mathematical models that run more and more of our lives? I would suggest that the process begin with the modelers themselves. Like doctors, data scientists should pledge a Hippocratic Oath, one that focuses on the possible misuses and misinterpretations of their models. Following the market crash of 2008, two financial engineers, Emanuel Derman and Paul Wilmott,
drew up such an oath. It reads:

~  I will remember that I didn’t make the world, and it doesn’t satisfy my equations.

~  Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.

~  I will never sacrifice reality for elegance without explaining why I have done so.

~  Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights.

~  I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.

That’s a good philosophical grounding. But solid values and self-regulation rein in only the scrupulous. What’s more, the Hippocratic Oath ignores the on-the-ground pressure that data scientists often confront when bosses push for specific answers. To eliminate WMDs, we must advance beyond establishing best practices in our data guild. Our laws need to change, too. And to make that happen we must reevaluate our metric of success.

Today, the success of a model is often measured in terms of profit, efficiency, or default rates. It’s almost always something that can be counted. What should we be counting, though? Consider this example. When people look for information about food stamps on a search engine, they are often confronted with ads for go-betweens, like
FindFamilyResources, of Tempe, Arizona. Such sites look official and provide links to real government forms. But they also gather names and e-mail addresses for predatory advertisers, including for-profit colleges. They rake in lead generation fees by providing a superfluous service to people, many of whom are soon targeted for services they can ill afford.

Is the transaction successful? It depends on what you count. For Google, the click on the ad brings in a quarter, fifty cents, or even a dollar or two. That’s a success. Naturally, the lead generator also makes money. And so it looks as though the system is functioning efficiently. The wheels of commerce are turning.