It’s one of those Black Mirror episodes that blurred the lines between fact and conjecture.
“Nosedive,” the season three premiere of the popular sci-fi TV series, follows Lacie, a prim, proper woman who’s obsessed with how the world views her, as she struggles to maintain her social rating.
In her society’s social scoring system, each pleasant personal interaction earns people high ratings that send their overall scores up; each public misstep earns a low rating that drives their scores down. A higher score gives them access to the best housing, material possessions, and social company. A lower score restricts access.
We won’t ruin the ending for you. But if you want an idea, you can take a look at what’s going on in China.
In 2014, the State Council, the chief administrative authority in the country, announced its intention to create a nationwide tracking system for its people and businesses. By 2020, the goal is for all 1.4 billion people in China to have a data file attached to them biometrically.
Maintain a pristine credit score and citizens can earn top billing on the country’s largest dating site. Fail to pay a fine and they risk becoming one of the millions of people who have been barred from taking flights because of “social misdeeds.”
The government’s stated goal is to promote good behavior among its citizenry. In practice, though, experts worry the rating system is ushering in an enhanced surveillance state that could unfairly bar its unwilling participants from certain goods and services.
So how, exactly, is this high-tech, data-driven system playing out in the world’s most populous nation? And what are the chances that aspects of it can take hold here in the U.S.?
The Social Experiment
In early 2015, the People’s Bank of China granted eight private companies the permission to begin building a credit scoring system that complemented the centralized system maintained by the government.
My sense is that this is moving slower than it looks from the outside. I hope it is, at least.
Those companies had the leeway to mine consumers’ online purchases and behavior to flesh out those scores. Two companies in particular, Tencent and Alibaba, started offering pilot products that scored customers based not only on their purchases, but on factors such as education level and the scores of their friends on those platforms. Wired magazine reported that Alibaba’s affiliated credit app, Zhima Credit, is voluntary to opt into and includes language saying that it “does not share user scores or underlying data with any third party including the government without the user’s prior consent,” but even the People’s Bank has taken steps to try to slow down these companies’ efforts toward social credit scoring.
And that’s not even considering municipal government efforts to track and augment behavior in the march toward 2020. The city of Shenzhen, for example, employs a rather drastic system to prevent jaywalking: Linking facial recognition software with citizens’ biometric data files, the city displays offenders’ faces and names on a screen above the intersection.
The question in China remains just how invasive the government plans to be in creating its national database. Pam Dixon, executive director of the World Privacy Forum, says that, in her experience, opinions are mixed among government officials.
“The system is controversial even in China, and there are lots of people in the Chinese government who don’t necessarily want a social scoring system,” Dixon says. “At this time, it’s not a ubiquitous thing.”
The populace, too, has shown signs of pushing back against the mining of their personal data. When Robin Li, founder of the tech company Baidu, suggested in March that Chinese people were willing to give up privacy for convenience, he faced significant blowback.
Dixon was in China in 2015. She says the experience on the ground is a bit different from the breathless reports about an impending information apocalypse.
“We’re looking at a potential social scoring system with great horror for a lot of reasons. If true, it provides for a significant diminishment of personal autonomy,” Dixon says. “My sense is that this is moving slower than it looks from the outside. I hope it is, at least.”
A Framework in Place
There is some truth to Li’s evaluation of privacy versus convenience, though. And it’s not just a characteristic exhibited by citizens of China.
Take the recent Cambridge Analytica fiasco, for example, when a political data firm earned access to more than 50 million social media users’ private information without their knowledge. Or the Google suite of products, which tracks every email sent, document created, route taken, and search logged from their users, among other things.
Doesn’t seem like many Americans are deleting their social media accounts or going back to Dogpile, even with these privacy concerns.
“Google may know more about you than your spouse does,” says Bob Gellman, a privacy and information policy consultant based in Washington, D.C. “The private sector has it, the government can get it, if it wants it. There’s a tremendous amount of information about you that the government has, the private sector has and they share amongst themselves, either free or at a price.”
There are laws in place in the U.S. that prevent the government snooping into our private data. There are relatively few regulations, however, dictating how third parties can obtain our data and what they can do with it.
It’s not about having something to hide. It’s about what someone can do with your information, what decisions they can make with it, and what predictions they can make about you using it.
Dixon and Gellman co-authored a 2014 report called “The Scoring of America,” in which they researched the multitude of “secret” consumer scores that organizations keep on citizens, which they have no chance to “see, correct, or opt out of.”
Credit scores are fairly commonplace in the U.S., and they’re regulated by the Fair Credit Reporting Act. But what about the Medical Adherence Score, which predicts how likely you are to take medicine prescribed to you by a physician? Or how about the Consumer View Profitability Score, which predicts your potential as an earner?
Or how about something a little more simple, like the TSA SPOT scoring system? It’s a list of 92 possible points and, if you ring up six or more in the eyes of a TSA agent, you get pulled out of the line for further screening.
“There are no laws that protect this lifestyle information or marketing information. That can be used in any way that companies want,” Gellman says. “They can create scores about people without limit.”
The major difference between America and China, though, is that there are more legal and behavioral hurdles the U.S. government would have to cross in order to fold all of this scoring into one centralized system. With ever-increasing attention to government surveillance, to do so would risk major backlash.
If anyone starts talking about social credit in America we gotta squash that real quick
— Aaron Wolffis (@aaronwolffis) April 8, 2018
But that doesn’t mean we’re safe from entities using our private information to determine the course of our lives.
“People say, ‘If I’m not doing anything wrong, what’s the problem?’ That’s a very old-fashioned question,” Dixon says. “It’s not about having something to hide. It’s about what someone can do with your information, what decisions they can make with it, and what predictions they can make about you using it. You don’t want to be boxed in by some sort of data profile that you may not like and don’t really have any control over.”
As things stand now, according to Dixon, it’s a “one-way mirror” when it comes to corporations or government entities creating and curating scores about the people they serve. The citizenry can make its voice heard in certain aspects: voting against a candidate or ballot measure, for instance, or leaving an especially scathing Yelp! review, or filing a report with the Better Business Bureau.
But the tools we wield to combat the entities keeping score on us pale in comparison to the effect these ratings could have on us—even if our governmental system makes it more difficult to keep a centralized rating of individuals.
Through the Looking Glass
Here’s something to consider, though: How would a social credit system—where every interaction could earn or restrict access to certain things—play out in the way we treat each other?
After all, one of the stated goals of China’s proposed system is improving how its populace interacts on a person-to-person level.
If all the world’s a stage, and the private and public are becoming one, wouldn’t it make sense for everyone to be on their best behavior all the time?
“You have a performative persona of you. You can behave differently depending on which platforms you’re on,” says Paolo Parigi, an adjunct professor in civil and environmental engineering at Stanford and the “lead trust scientist” for Airbnb. “I don’t necessarily see it in the sense that you’re not sincere. I see it more through the lenses of reputation. You care about your reputation on this particular platform, so you behave well. You perform.”
Parigi was one the Stanford researchers who co-authored a 2017 paper that argued “that we are fundamentally living in a new social world in which the Internet mediates a growing number of our social interactions.”
All of us leave a piece of ourselves on every platform we use, be it Google, Twitter, Lyft, Airbnb, you name it. It gets to the point where, with so much information out there, strangers aren’t really strangers anymore.
That can be a good and bad thing, Parigi says. He researched data from the Couchsurfing, a homesharing service, and he found that the less the guest knew about the host before the stay, the deeper the ensuing relationship ended up being.
“You can figure out something about that person to a degree that you couldn’t before. It’s easier to interact and place some trust because you have some information,” Parigi says. “But there is a price that you pay for this ease of creating relationships or trusting. You’re creating a commodity out of that interaction. You may have many friendships, connections, ties, but they’re less relevant, each one of them.”
A Move to More Privacy?
The western world, at least, seems to be taking some steps toward swinging the pendulum back to privacy.
The General Data Protection Regulation is going into effect in European Union countries May 25. It seeks to give people the right to see and correct the information being gathered about them by data collectors, and it requires government entities to establish a legal basis for obtaining and using this information.
Gellman’s hope is that American companies who do business in the EU will follow suit.
Failing that, people who don’t want their data passed around can invest in complicated encryption services. Or stop running their entire lives through Google. Or put down their phones.
That’s a hard sell.
“The general rule is convenience trumps privacy,” Gellman says. “You have to make choices.”