New York City Restricts AI Hiring Tools, Implementation Details Still Unclear

A New York City law restricting the use of AI tools in the hiring process will take effect early next year. The law is seen as a pioneer in protecting job seekers from algorithmic bias, but so far, little is known about how employers or vendors can comply, raising concerns about whether the law is the right path to address bias in hiring algorithms.

The law includes two main requirements: employers must audit any automated decision-making tool before using it for hiring or promotion, and must notify job seekers or employees at least 10 business days before use. First violations carry a fine of $500, with each additional violation fined $1,500.

Although Illinois has regulated AI analysis in video interviews since 2020, New York City's law is the first in the nation to apply to the entire hiring process.

The law aims to respond to concerns from the U.S. Equal Employment Opportunity Commission and the U.S. Department of Justice that "blind reliance" on AI tools in hiring could cause businesses to violate the Americans with Disabilities Act.

"New York City is taking a comprehensive look at how automated decision-making systems are changing hiring practices," Dr. Julia Stoyanovich, a computer science professor at New York University and a member of the city's Automated Decision Systems Working Group, told HR Dive. "This is about ensuring people have equal access to economic opportunity. What if they can't find a job and don't know why?"

Beyond the "model population"

AI hiring tools are designed to support HR teams throughout the hiring process, from posting ads on job boards, to screening candidate resumes, to determining appropriate compensation packages. The goal, of course, is to help businesses find people with the right backgrounds and skills.

Unfortunately, every step of this process can introduce bias, especially when an employer's "model population" of potential candidates is compared against the existing workforce roster. Notably, Amazon once had to scrap a recruiting tool—trained on resumes submitted over a decade—because the algorithm, after self-learning, penalized resumes containing the word "women's."

"You're trying to identify someone you predict will be successful. You use the past as a prologue for the present," said David J. Walton, a partner at Fisher & Phillips law firm. "When you look back and use data, if the model population is predominantly white, male, and under 40, then the algorithm will naturally seek those characteristics. How do you redesign the model population so the output is unbiased?"

AI tools used to evaluate candidates in interviews or tests can also pose problems. Measuring speech patterns in video interviews could screen out candidates with speech impairments, while tracking keyboard input could exclude those with arthritis or other conditions limiting dexterity.

"Many workers with disabilities would be disadvantaged by how these tools assess them," said Matt Scherer, senior policy advisor for worker privacy at the Center for Democracy & Technology. "Many tools operate by making assumptions about people."

Walton said these tools are similar to the "pull-up tests" often given to firefighter candidates: "It's not discriminatory on its face, but it may have a disparate impact on protected classes under the ADA."

There is also a category of AI tools designed to help identify candidates with suitable personalities. Stoyanovich said these tools also have problems, and she recently published an audit of two commonly used tools. The issues are both technical—the tools produced different scores for the same resume submitted in plain text versus PDF format—and philosophical.

"What is a 'team player'?" she said. "AI is not magic. If you don't tell it what to look for and don't validate it with scientific methods, then the predictions are little better than random guesses."

Legislation or stronger regulation?

New York City's law is part of a larger trend at the state and federal levels. The federal American Data Privacy and Protection Act, proposed earlier this year, contains similar provisions, while the Algorithmic Accountability Act would require "impact assessments" of automated decision systems across various use cases, including employment. Additionally, California is planning to bring the use of AI hiring tools under the scope of the state's anti-discrimination laws.

However, some worry that legislation is not the right way to address AI hiring issues. "New York City's law doesn't impose anything new," Scherer said. "The disclosure requirements are minimal, and the audit requirement is just a narrow subset of what federal law already requires."

Given the limited guidance issued by New York City officials before the law takes effect on January 1, 2023, it remains unclear what a technical audit looks like or how it should be completed. Walton said employers may need to work with people who have expertise in data and business analysis.

On a higher level, Stoyanovich said AI hiring tools would benefit from standards-based audit processes. She said standards should be publicly discussed, and certification should be conducted by independent bodies—whether nonprofit organizations, government agencies, or other entities that do not profit from them.

Given these needs, Scherer said he believes regulatory action is preferable to legislation. The challenge for those working to strengthen regulation of such tools is getting policymakers to drive the conversation.

"The tools already exist, and policy hasn't kept pace with technological change," Scherer said. "We're working to ensure policymakers realize there need to be real audit requirements for these tools, and meaningful disclosure and accountability when tools lead to discrimination. We still have a long way to go."