It was a gray December afternoon in Arlington, Virginia, day two of the Society for Risk Analysis’ 2019 Annual Meeting, when we first met Paul Slovic. A balding man of 82 in cargo pants, running shoes, and a well-worn wool color-blocked sweater, he was jotting down notes about a presentation on risk communications: “Incomplete. Complex. Multidisciplinary.”

“Fast, intuitive thinking doesn’t scale,” he told us later, especially when it comes to social values and humanitarian concerns. Tech leaders would do well to have more humility “about their ability to anticipate all the problems that might arise from a powerful system in which we don’t have experiential basis for decisions.” The first step, he said, is “to be aware of the tricks that our minds play on us, so we can be alert as to why systems that can protect us can also deceive us.”

Slovic has spent decades studying risk and decision-making, and what happens to organizations encountering unprecedented speed, scale, and harm. “People have to think incredibly hard to imagine undesirable and never-before-seen consequences,” he told us.

In 1957, Slovic was a teenage basketball player who’d enrolled at Stanford with plans to major in math. Soon after, he concluded he was much more interested in the powerful forces that shape human behavior — curiosity, fear, ambition, status, attention, greed — and switched majors to psychology.

While in graduate school at the University of Michigan, Slovic was introduced to research on the psychology of risk and decision-making. Eventually, he and his collaborators would identify a deep well of mechanisms central to what he calls “society’s gambles” — the risks we face every day. His work investigates the cognitive heuristics and biases, grounded in experience, identity, and emotions, that shape decision-making, developing a canon of work. Slovic’s theories and methods have been applied to sectors as diverse as public policy, energy, medicine, human rights, law, aerospace, the military, economics, and the environment.

Risk assessment, Slovic says, is challenging because it forces us to confront and overcome our own biases. “Decision-making is also coupled with the fact that if we don’t want to see certain kinds of things, we’re likely not to think of them. If there are outcomes that are problematic or unpleasant, you might not work very hard to anticipate and address them.”

“Decision-making is also coupled with the fact that if we don’t want to see certain kinds of things.”

In the 1970s, Slovic was invited to present his work at conferences of nuclear engineers and energy industry executives. “Nuclear power, as with tech today, was a technology driven by engineering science, by great technical knowledge,” Slovic says. The engineers viewed themselves as the smartest guys in the room — not unlike today’s tech engineers and CEOs. They had little understanding or interest in the psychological perspectives on risk that Slovic was describing.

At the time, the nuclear industry was “foundering on the shoals of adverse public opinion,” he and two co-authors wrote in the late 1970s. There were critical discrepancies between expert and lay judgments, he told his audiences of nuclear engineers and scientists, and they’d do well to understand them. Slovic warned that opposition to nuclear power was springing up nationwide and that nuclear scientists should not dismiss public concerns about risk as ignorance and irrationality.

Instead, Slovic says, the engineers regarded him as a troublemaker, focused on subjective unquantifiables — uncertainties, values, politics, issues of trust — in the face of “hard science” and technical expertise. The engineers in attendance could not see themselves — their homogeneity, social status, commercial incentives, feelings, and disdain for “nonexperts” — as risk factors.

“The whole process is subjective and value-laden,” says Slovic, describing his encounters and decades of research. “Defining risk is an exercise of power.”

Then the nuclear industry got a wake-up call. Around 4 a.m. on March 28, 1979, the Unit 2 reactor at the Three Mile Island Nuclear Generating Station in Pennsylvania failed, resulting in a partial meltdown. An investigative report found the plant’s design was flawed, and its staff were poorly trained for an emergency situation. Soon after, the U.S. Nuclear Regulatory Commission provided funding for Slovic, along with colleagues and Decision Research co-founders Baruch Fischhoff and Sarah Lichtenstein, to produce a guide to “acceptable risk.”

“Acceptable-risk problems are decision problems,” they wrote in the resulting report, published in 1980. “They require a choice between alternatives. That choice depends upon the alternatives, values, and beliefs that are considered.”

Risk analysis was initially grounded as an “objective” tool for engineers and statesmen who needed more facts to understand and control risks presented by new technologies, in the aerospace and nuclear industries in particular. A community coalesced around the subject through the 1970s and 1980s, in the private sector, industry associations, academia, research centers, and at the federal policy level, with engineers, researchers, and regulators building industry-specific processes and controls.

Today, a fleet of firms provides information security and risk management services to pretty much any collective, sector, or company you can imagine — HP, Nestle, the National Weather Service, the State of Michigan have all enlisted risk management services. It’s estimated that this industry pulls in upward of $130 billion every year.

Now, critics say that the tech industry’s existing approach to dealing with risk — rooted in narrowly defined practices and relegated to legal and infosecurity departments focused on the quantifiable and discrete, like copyright — is dated. Infosec’s traditional focus on stopping “attacks” on systems by known adversaries has left companies open to misuse. In 2020, critics say, the tech industry’s practice of risk assessment must also take into account the protection of social values, like access to accurate information.

Soliciting input from the public is essential for overcoming internal biases and blind spots, and avoiding problems down the road, Slovic and his colleagues argue. As they put it in 1980: “Early public involvement may lead to decisions that take longer to make, but are more likely to stick.”

Assessing the risks of a partial nuclear meltdown and assessing the risks of a social media platform being used to hack an election may appear vastly different — but there’s common ground. Both industries made big claims about transforming the world; both industries had early critics and civil society advocates warning of potential harms; and both are dominated by white, powerful men disposed to dismissing critics as subjective, emotional, unreasonable, and ill-informed.

In 2020, critics say, the tech industry’s practice of risk assessment must also take into account the protection of social values.

And both rushed their products to market, prioritizing technical and business goals over social and humanitarian concerns. Today’s tech leaders still tend to prioritize technical fixes — better algorithms, faster processors, improved features — over efforts to improve the structure of decision-making and paths to public engagement that improve outcomes.

Is tech now in the midst of its own Three Mile Island moment? There are signs that risk assessment practices are evolving to meet the crisis, as new theories and approaches emerge. The shift is being driven largely by academics and risk experts working outside the biggest tech companies. Their ideas and work — the Center for Humane Technology’s design guide for building “more humane” products and identifying “where investing in a deeper understanding of human nature will yield further benefits,” IEEE Global Initiative’s Ethically Aligned Design, Social Threat Modeling, the safety culture model, the do no harm approach, Ethical Operating Systems, Value-Sensitive Design, Cyber Resilience, and the Cyber-XR Coalition’s recently released guide to ethics and safety in cybersecurity and XR, for instance — represent burgeoning efforts to address risk that go beyond traditional, technical solutions and embrace human and social values.

Michael Coates, the former head of security at Twitter and Mozilla, says that these conversations are starting to have an impact. Civic concerns, he tells us, were “talked about for years in the industry” but “were brushed off by people saying, ‘Well, you’re being a little paranoid.’ Or, ‘Those things won’t happen at all.’ This approach was common among industry experts tackling security issues.”

Today’s tech companies don’t have the luxury of being so complacent. Those problems that all those security experts thought we’d never see? “Well, some of them are now seen,” Coates says.

Home

Viral hate, election interference, and hacked accounts: inside the tech industry’s decades-long failure to reckon with risk