The AI Police

A new company makes artificial intelligence software that鈥檚 in use at a handful of police departments. Can it make law enforcement more transparent?

By Bloomberg LP

here鈥檚 a story Brett Goldstein 鈥�96 likes to tell. It starts on a Friday night in 2010 with him sitting in a darkened Crown Victoria on a Chicago street, poring over maps. Goldstein was a commander at the Chicago Police Department, in charge of a small unit using data analysis to predict where certain types of crimes were likely to occur at any time. Earlier that day, his computer models forecast a heightened probability of violence on a particular South Side block. Now that he and his partner were there, Goldstein was doubting himself.

鈥淚t didn鈥檛 look like it should be a target for a shooting,鈥� he recalled. 鈥淭he houses looked great. Everything was well manicured. You expect, if you鈥檙e in this neighborhood, you鈥檙e looking for abandoned buildings, you鈥檙e looking for people selling dope. I saw none of that.鈥�

Still, they staked it out. Goldstein鈥檚 wife had just given birth to their second child, and he was exhausted after a day in the office. He started to doze off. Goldstein鈥檚 partner argued that the data must be wrong. At 11 p.m., they left.

Several hours later, Goldstein woke up to the sound of his BlackBerry buzzing. There had been a shooting鈥攐n the block where he鈥檇 been camped out. 鈥淭his sticks with me because we thought we shouldn鈥檛 be there, but the computer thought we should be there,鈥� said Goldstein. He took the near-miss as vindication of his vision for the future of law enforcement. 鈥淚 do believe in a policeman鈥檚 gut. But I also believe in augmenting his or her gut,鈥� he said.

Seven years after his evening on the South Side, Goldstein threw on a gray suit and some aerodynamic sunglasses and headed out from his hotel in Midtown Manhattan into New Jersey. This spring, he founded CivicScape, a technology company that sells crime-predicting software to police departments. Nine cities are either using the software or in the process of implementing it, including four of the country鈥檚 35 largest cities by population. Departments pay between $30,000 a year to use the software in cities with less than 100,000 people to $155,000 a year in cities with populations that exceed 1 million. Goldstein wanted to check in on the two clients who were furthest along鈥攖he police departments in the New Jersey towns of Camden and Linden.

Goldstein likes to harp on his own lack of charisma, but he鈥檚 well suited to be a pitchman for police departments. In Chicago, he rose from patrol officer to the city鈥檚 chief data officer over a seven-year government career and regularly drops a few war stories from the streets into his conversations with cops. He鈥檚 also peddling something that every department is after nowadays: technological sophistication. The criminal justice system produces reams of data, and new computing methods offer to turn any pool of numbers into something useful. Today, almost every major police department in the country is using or has used some form of commercial software that makes predictions about crime, either to determine what blocks warrant heightened police presence or even which people are most likely to be involved. Technology is transforming the craft of policing.

Not everyone is rubbing their hands in anticipation. Many police officers still see so-called predictive policing software as mumbo jumbo. Critics outside of law enforcement argue that it鈥檚 actively destructive. The historical information these programs use to predict patterns of crime aren鈥檛 a neutral recounting of objective fact; they鈥檙e a reflection of socioeconomic disparities and the aggressive policing of black neighborhoods. Computer scientists have held up predictive policing as a poster child of how automated decision making can be misused. Others mock it as pseudoscience. 鈥淪ystems that manufacture unexplained 鈥榯hreat鈥� assessments have no valid place in constitutional policing,鈥� wrote a coalition of civil rights and technology associations, including the ACLU, the Brennan Center for Justice, and the Center for Democracy & Technology, in a statement last summer.

A numbing progression of police shootings in the past several years serves as a reminder of what鈥檚 at stake when police officers see certain communities as disproportionately threatening. Over the course of eight days in late June, juries failed to convict officers who killed black men in Minnesota, Ohio and Wisconsin. In each case, the officer鈥檚 defense relied on his perception of danger. The worst-case scenario with predictive policing software is deploying officers to target areas with their ears raised, leading them to turn violent in what would otherwise be routine encounters.

Photo by Misha Friedman for Bloomberg

The police departments Goldstein visited in New Jersey didn鈥檛 raise any questions about fairness during his recent trip鈥攂ut there was skepticism nonetheless. He had barely started speaking to a group of top officers in the Linden Police Department when the man who handled the city鈥檚 procurement process confessed how wary he was of software vendors鈥� magical-sounding claims. Goldstein nodded along. When he was a cop, he said, he hated sitting through 鈥渢he vendor nonsense.鈥� Goldstein launched into a sing-song voice: 鈥溾€極h, you鈥檙e going to have a flying car, and it鈥檚 going to stop people, and you鈥檙e going to be Super Po-Po!鈥� They鈥檒l promise you anything.鈥�

Goldstein鈥檚 company does make one unusual promise, which it thinks can satisfy skeptics in law enforcement and civil rights circles simultaneously. Other companies that make predictive software for criminal justice settings keep their algorithms secret for competitive reasons. In March, CivicScape published its code on GitHub, a website where computer programmers post and critique one another鈥檚 work. It was an unprecedented move, and it caused an immediate stir among people who follow the cop tech industry. 鈥淭hey鈥檙e doing all the things I鈥檝e been screaming about for years,鈥� said Andrew Ferguson, a professor at the University of the District of Columbia鈥檚 law school and author of the forthcoming book The Rise of Big Data Policing.

Posting computer code online won鈥檛 erase the worries about predictive policing. There are still concerns about how CivicScape responds to perceived shortcomings, and there鈥檚 also the big question of what police departments do with the intelligence it produces. But more than any other company, CivicScape has turned itself into a test case for what it means for law enforcement to use artificial intelligence in a way that鈥檚 transparent and accountable鈥攁nd whether that鈥檚 even possible.

Goldstein, 43, didn鈥檛 start off wanting to be a cop. He was the director of information technology at OpenTable, the online restaurant reservation company, but he began to question the significance of that work after 9/11. In 2004, Goldstein saw an advertisement for the Chicago Police Department鈥檚 entry exam. 鈥淚鈥檓 like, 鈥榳hat does it hurt to take the police exam? I like taking tests,鈥欌€� he recalled. Goldstein took the test, did well, and in 2006 left OpenTable.

After 13 months as a beat cop, Goldstein began working to develop computer models to anticipate where crime would happen. He was later promoted to commander and put in charge of a new unit dedicated to this strategy. The unit was providing intelligence that far exceeded what it had been using before, according to Michael Masters, who first met Goldstein during his academy days when Masters was an adviser to Mayor Richard M. Daley, then moved to the police department and now works at CivicScape. 鈥淲e were well ahead of our time,鈥� said Masters. Goldstein was perfectly placed to build technology into the daily work of policing. 鈥淵ou don鈥檛 have people who were cops, and have ridden in squad cars, building these tools,鈥� Masters said.

Like any fast riser at a slow-moving institution, Goldstein was a polarizing figure. There were running rumors that he had some family connections at City Hall, and he had trouble developing any tough guy credibility鈥攅ven after he apprehended a shooter who killed a man in front of Goldstein鈥檚 family on his day off. Longtime officers simultaneously thought it was simple to predict broad patterns of crime, which consistently centered on the same areas of the city, and impossible to anticipate specific offenses. On Second City Cop, a popular, anonymous blog, Goldstein was dubbed Golden Boy, and his unit was called the Crystal Ball Unit. Neither was meant as a compliment. Goldstein鈥檚 critics would gloat when a shooting occurred a block from one of his target areas, and they鈥檇 occasionally berate him in person at headquarters.

Photo by Misha Friedman for Bloomberg

Goldstein admitted he failed to win over his critics, and his unit was disbanded when the head of the department stepped down in 2011. And he acknowledged that he never came up with a rigorous way to test the impact his techniques had on crime rates. Goldstein moved to City Hall, then left government in 2013. Since then, Goldstein has run a venture capital fund, held academic positions, and sat on the board of Code for America, a nonprofit dedicated to help governments use technology. With the Crystal Ball Unit gone, people on both sides of the debate in Chicago retreated to the comfort of their preconceived notions.

The Camden County Police Department鈥檚 Real-Time Tactical Operational Intelligence Center is a Rorschach test on how you feel about tech in law enforcement. The RT-TOIC, as it鈥檚 known, is a windowless room from which the department runs its technological initiatives. When Goldstein visited this month, the futuristic sheen had been undermined by the failure of the building鈥檚 air conditioning鈥攊t remained inhabitable only with a bunch of full-blast floor fans. Still, the department鈥檚 leadership thinks the RT-TOIC represents the future of policing, not just in Camden, but everywhere.

About a dozen people were at work inside, most of them sitting at stations displaying between four and six computer screens. Large screens showing maps and footage from surveillance cameras were displayed on the wall. Analysts monitored social media for accounts that have referred to known crimes.

Camden integrated CivicScape into the RT-TOIC three months ago. The company鈥檚 maps are always running, changing every hour to reflect updated data. When targets change, analysts switch their screens to the surveillance cameras pointed at those blocks. Officers translate what鈥檚 happening in the
RT-TOIC to the cops on the street. The guys in patrol cars don鈥檛 know whether an order is derived from some newfangled math, the judgment of a superior officer or a mixture. The ambiguity is deliberate, said Kerry Yerico, the department鈥檚 director of criminal intelligence and analysis.

On the day of Goldstein鈥檚 visit, Yerico and Lieutenant Jeremy Merck, the watch commander on duty, were discussing an area that CivicScape had flagged. Merck immediately recognized the area鈥攈is officers had said that drug dealers were ramping up operations there. They had deployed extra officers.

Neither Yerico nor Merck knew exactly how the department鈥檚 computers and humans had homed in on the same spot. The guts of CivicScape鈥檚 predictive system are a series of neural networks. Neural networks, named because their design mimics the structure of neurons in the human brain, examine large data sets in which the inputs and outcomes are labeled. They then determine patterns they can use to predict what will happen when presented with new data.

Neural networks are favored by computer scientists working with huge data sets, but one of their shortcomings is their opaqueness. Unlike an algorithm in which a human has consciously told the system what to think about each factor, neural networks find their own paths and can鈥檛 effectively explain to humans what they鈥檝e done. This has the potential to make CivicScape even less transparent than other predictive-policing software, which use different types of algorithms.

Scott Thompson, Camden鈥檚 police chief, said he hasn鈥檛 heard any criticism about transparency. For its part, CivicScape said its openness comes from inviting discussion about the types of data its models use.

Kristian Lum and William Isaac, researchers who have written their own statistical models for the Human Rights Data Analysis Group demonstrating how bias works in predictive policing, have examined the code. They both described CivicScape鈥檚 move as positive but withheld praise until they see how the company followed through.

I do believe in a policeman’s gut. But I also believe in augmenting his or her gut.

A significant shortcoming with CivicScape鈥檚 code repository, said Isaac, is that it has posted generic code when in practice it adapts its system for each separate client. Police departments often resist releasing data they鈥檙e not mandated to make public by law, and Goldstein acknowledged that his clients will not allow him to share some of the data that he鈥檚 using to produce predictions. It鈥檚 hard for an observer to assess what an algorithm does without access to either the final version of the code or a full set of the data.

鈥淚 think it鈥檚 a straddle鈥� between the desires of police departments and the public, said Goldstein. 鈥淚鈥檇 rather take this step and move forward than not take a step because we know there are imperfections.鈥� Isaac said that CivicScape isn鈥檛 fully in control of how its system works but that Goldstein鈥檚 attitude illustrates the problematic approach companies and government agencies take toward predictive tech. Goldstein rightly pointed out that secrecy has been established as the baseline, then demanded credit for any steps he takes in the other direction. 鈥淪hould it be up to him?鈥� said Isaac. 鈥淚 think that鈥檚 kind of a false choice.鈥�

Linden, like many of CivicScape鈥檚 clients, is a small department going through a transition. Its chief, Jonathan Parham, started in late 2016. He is a lifelong Linden resident, a 25-year veteran of the force, and its first African-American chief. He also thinks police departments have overemphasized arresting people.

Parham said he sees predictive policing as a way to offset brain drain. A lack of experience in the department has left Linden鈥檚 officers without a basic understanding of its communities. 鈥淲e鈥檙e looking at the absence of personal knowledge of your area and supplementing that with technical knowledge of the frequency of the crimes,鈥� said Parham.

This is a common argument for adding computers. Camden鈥檚 department cited a similar need. All the cameras, social media analysis, and automated forecasting are supposed to help the department cover the most ground with the fewest officers.

To Isaac, this is a myopic way to approach criminal justice. Even the act of updating target areas hourly distorts the situation by ignoring any potential service that can鈥檛 be carried out by a cop over the next 60 minutes. The potential advantages of predictive tech are undermined by restricting it to something that cops use to catch supposed bad guys. 鈥淲hat you are left with is a perceived chess match between cops and robbers,鈥� said Isaac. 鈥淭hat鈥檚 a very simplistic version of what crime really is.鈥�

Parham offered a similar view. The worst thing Linden鈥檚 police department could do, he argued, was to believe that Goldstein really did have mystical math that would allow officers to drive to the specific locations of shootings just as they were about to occur. During the meeting with Goldstein, he told his department鈥檚 brass that it needed to avoid turning policing into a video game.

Parham recalled a training exercise he helped run. Officers were sent to a train station and told to interview people in the station, to treat people as more than potential perps. After 10 minutes, each officer would write down what he had learned. The winner was the person with the most useful information.

The officers who were logging the most arrests always performed the worst, Parham said. If law enforcement is a matter of receiving a target from a computer and then attacking that target, it doesn鈥檛 matter how precise the computer model is. Parham鈥檚 job is to produce cops who are better at the train station drill. 鈥淥ur officers, the more technologically savvy they get, the less human they become,鈥� said Parham. 鈥淚 don鈥檛 want that.鈥�

糖心TV