New York Just Set a ‘Dangerous Precedent’ on Algorithms, Experts Warn

It was supposed to be groundbreaking. When New York City’s task force to develop policy on algorithm technologies was introduced two years ago, it was praised as a beacon of transparent and equitable government. It was supposed to inform other policymakers grappling with how to address their own use of automated technologies that make decisions in place of humans.

But for all its good intentions, the effort was bogged down in a bureaucratic morass. The task force failed at even completing a first necessary step in its work: getting access to basic information about automated systems already in use, according to task force members and observers.

“The fact they were unable to even get information about what tools the city was using is very problematic,” said Deirdre Mulligan, associate professor at the UC Berkeley School of Information. Algorithmic tools “in and of themselves embed really significant policy choices.” A spokesman for New York City did not respond to an inquiry about why the task force did not provide a catalog of automated systems.

A shadow report published last week in response to the initiative has issued a stark warning cry about “the limitations of existing bureaucratic procedures” to tackle automated decisions.

Cities use algorithmic and automated technologies to make all sorts of decisions, from determining where kids are sent to school, to deciding whether someone should be released on bail before a criminal trial.

But thus far, there have been few protections to ensure these technologies are making fair decisions. And early experiences with AI tools have shown that they can be categorically unfair, baking into automated tools the biases that already existed in the humans and data that trained them. For example, algorithms used by law enforcement to assess and score criminal or child safety risks have been found to produce results that can negatively and disproportionately affect African Americans.  

In other words, automated systems make decisions that affect people’s lives in very real ways right now, said Meredith Whittaker, co-founder of the AI Now Institute at NYU and a member of New York City’s Automated Decision System task force. “This is not a theoretical concern,” she told CityLab.

It was this sort of concern that led New York City to create a task force as a first step toward setting policy around the use of algorithmic tools. The task force did convene and make some structural recommendations. Among the most significant: It called for creation of an “organizational structure” within city government to serve as a centralized hub for future policy guidance and automated systems management. After the report was unveiled, Mayor Bill de Blasio responded to this recommendation by announcing a new Algorithms Management and Policy Officer position within the Mayor’s Office of Operations.

“This Officer will be carrying the important work of the Task Force forward so that city agency use of algorithms will be fair, equitable, and transparent,” the New York City Hall spokesperson told CityLab.  In a press release touting the new position, members of the task force, some of whom serve in city government roles, praised this first step.

But Rashida Richardson, editor of the shadow report and director of policy research at the AI Now Institute at NYU, said the new algorithms officer role is “is just an effort for the Mayor and the city to seem like they did something.”

And one task force member who does not work for city government, Julia Stoyanovich, assistant professor of computer science and engineering at NYU, acknowledged in the release that automated decision system “management and oversight are difficult.” She continued, "I am particularly looking forward to seeing evidence of a strong commitment on the part of the City to a broad education effort.”

Part of the challenge for task force members was that city agencies, accustomed to keeping details of vendor technologies secret, were unwilling to provide even a basic list of automated systems already in use.

“There were delays, and obfuscations, and then, by the spring of 2019, outright denials. It wasn’t because the data didn’t exist,” wrote Albert Fox Cahn, executive director of civil rights and privacy group Surveillance Technology Oversight Project, who participated in some task force meetings, in FastCompany. “The City Council passed the task force into law to hold the administration accountable for its secretive use of algorithms, but that was the last thing the administration wanted.”

In response, a city spokesman said: “Mr. Cahn represented his former employer three times out of several dozen Task Force meetings, working sessions, public forums, and community roundtables. In his op-ed, Mr. Cahn has completely misrepresented the facts of the process, and his understanding of the Task Force's legal mandate is fundamentally flawed.”

If governments knew more about the systems they use, they could evaluate their potential impact through processes like algorithmic impact assessments. This is why civil rights and tech justice advocates want governments to reveal technical details about automated systems including source code, algorithms, and the data employed for training.  

But illuminating these details has been a struggle, especially as tech firms insist on protecting such information from disclosure. If this level of disclosure ever happens, it is very rare.

Indeed, the city’s own official task force report acknowledged the group “faced challenges” in balancing government transparency with considerations including “proprietary and other sensitive types of City information.”

The shadow report, endorsed by the ACLU, NAACP Legal Defense and Educational Fund, and more than 20 legal, human rights and tech justice organizations, argued that municipalities need to do a better job of ensuring control over contracts with tech providers. Its authors, technology experts and advocates for civil rights and algorithmic accountability, called for New York’s city council to pass a law requiring more transparent procurement of automated decision systems.

Richardson said deep-pocketed municipalities like New York could use their leverage when negotiating tech contracts to require more disclosure as a condition of doing business with the city. “New York City should use its purchasing power to only contract with firms willing to disclose the details of their technology, how they will be used, and who could potentially be harmed. Any firm not willing to be transparent about their technology should be left off the NYC payroll.”

Richardson, who was part of a coalition that sought more community engagement during the task force process, worried that the failure to make any meaningful progress in understanding how New York City is using algorithms will set a perverse example for other governments.

“The concern is around New York City setting dangerous precedent that the city doesn’t need to do anything,” she said.

In fact, New York City isn’t the only place that’s considered how to reform its use of algorithms. But even cities that have already mulled more aggressive regulation of algorithmic technologies haven’t disclosed much about their use of them. Portland, Oregon, is weighing a ban on algorithm-based facial recognition tech and recently passed a privacy resolution calling for evaluation of automated decision system impacts. But it has not conducted an inventory of automated decision systems, said Hector Dominguez, open data coordinator at the city’s data technology oversight body Smart City PDX.

“We are looking at what NYC, San Francisco, Oakland and Seattle and other sister cities are doing,” Dominguez wrote in an email.

San Francisco and Oakland have both banned facial recognition technology use by city government. Seattle has not, but the city updates a list of surveillance technologies every quarter and includes non-surveillance tech systems used by city agencies. The city and county of San Francisco unveiled a toolkit for assessing algorithmic impact in 2018.

A movement to update traditional government technology purchasing policies for these and other new technologies is already underway. Berkeley professors Mulligan and Kenneth Bamberger have called for “a move from a procurement mindset to policymaking mindset.”

In a paper expected to be published in the Berkeley Technology Law Journal next year, they write, “Through policy choices and design, we can build purposeful tools that are aligned with values chosen based on reason, expertise, transparency, and robust and ongoing deliberation and oversight.”