Business industry groups have lined up against legislation requiring automated systems be tested for bias, companies alert people of use
A years-in-the-making California bill that would take a step toward regulating artificial intelligence and other automated systems used in hiring, work benefits and insurance claims is drawing fierce opposition from the business community and pushback from within the state government.
The trade group representing some of San Francisco’s biggest technology companies has made killing the bill a top priority this legislative session, while the California Chamber of Commerce has come out against it, too. Meanwhile, the state Senate Appropriations Committee on Monday unanimously voted to move the bill to a file that is often the first step in tabling legislation without a public vote. “It’s wild to have it be so controversial,” said Samantha Gordon, chief advocacy officer at TechEquity, a citizen advocacy group that’s sponsoring the legislation. Assembly Bill 1018 would regulate automated systems that make or “facilitate” “consequential decisions,” such as those involving employment, housing, education and health care. The legislation, introduced by Democratic Assemblymember Rebecca Bauer-Kahan of Orinda in February, would require developers of such systems to test them for discriminatory effects. It would also mandate that companies, government agencies and organizations deploying those systems notify customers, job seekers and others who might be affected by them of their use before they are used. If companies, agencies or organizations use such a system to make a consequential decision, the bill would require them to alert people affected by that decision of how the system came to that determination and the types of personal information it relied on. Under the legislation, those affected by the decision would have a chance to correct any inaccurate information that was used in it and to appeal such decisions.Gordon pointed to health-insurance claims as an example. She said the bill is essentially saying that if an insurance provider uses an automated decision to determine whether to pay for a procedure a patient’s doctor has ordered, that system should be tested to make sure it’s working as it’s supposed to and doesn’t discriminate in making those decisions. It’s also saying that if the system didn’t use correct information about the patient in making that decision, the patient has an opportunity to address that, she said.California residents broadly support policies to regulate such systems, particularly to prevent discrimination, according to a new poll of 1,000 residents that Gordon’s group released Tuesday. When asked whether strong laws were needed to ensure that companies design AI systems so they are safe and secure or whether government should stay out of the way, because companies can responsibly innovate, 70% of polled Californians favored strong laws, according to the survey. Some 73% of residents support policies that would ensure that AI systems uphold civil rights and that would protect people from AI-powered discrimination. The poll had a margin of error of 3.1 percentage points. Nevertheless, business organizations are fiercely opposed to the bill. The Business Software Alliance, a trade group whose members include San Francisco’s OpenAI, Salesforce, Asana, Dropbox and Docusign, has made defeating AB 1018 its top priority of this legislative session. The California Chamber of Commerce, speaking on behalf of a collection of business and tech groups, has dubbed it a “cost driver.” The BSA agrees that developers of automated systems should be examining and managing the risks represented by such systems, said Craig Albright, the group’s senior vice president for U.S. government relations. But the focus should be on systems being used in high-risk situations, such as determining whether a citizen or consumer receives an important service or benefit, he said. AB 1018 would go well beyond that and could apply to seemingly innocuous systems, such as ones that help physicians schedule appointments, Albright said. The legislation also fails to take into account the software development process, where there are not always clear lines between developers and deployers, he said. Although these kinds of systems can be modified or customized at several points along the way between the initial developer and the organization using them, AB 1018 would put much of the burden of testing them for problems on the original developer, he said. Additionally, starting in 2030, the legislation would require developers to have third-party auditors assess whether they are complying with the evaluation procedures outlined in the bill. But there are no standards for auditing such procedures, Albright said.For its part, the CalChamber argues that the bill would be overly burdensome on small businesses and would require costly and unworkable individualized notifications. Under the bill, a company that decided not to hire someone it found through a LinkedIn search might be required to notify that person why it didn’t hire them, even if the person never actually applied for the job, Ronak Daylami, its policy advocate, argued in a letter last week to the Senate Appropriations Committee as it then considered the legislation. “In the entire history of employment, no employer has ever been required to provide personalized post-decision-making notices and explanations to the vast majority of applicants who are not selected for a position,” Daylami said.Ex // Top Stories The City’s official flower is the star of this Golden Gate Park show Dahlia Society of California’s annual display returns to San Francisco County Fair Building this weekend Amateur pizza nerds go all out in North Beach’s annual contest Third annual Pizza, Bagel and Beer Festival featured slate of returning competitors, reigning champion as judge Claude ‘is pretty iconic,’ and so are his birthday celebrations The albino alligator turns 30 next month, and the California Academy of Sciences is pulling out all the stops The bill puts off the auditing requirement until 2030 specifically to allow time for standards to develop and auditors to get up to speed on them, she said. She has separately authored legislation — AB 1405 — that would set up a process by which the state would register AI auditors and require them to conduct examinations according to “widely recognized industry standards.” Additionally, AB 1018 explicitly would not apply to databases, calculators, spam filters, access management software and other low-risk systems. It also would shift the burden of testing systems for bias to deployers in particular instances, such as if they use such a system for a purpose other than what the developer has approved or they’ve substantially modified it and made the modified version available to other companies or organizations. Bauer-Kahan first started working on a bill that would require developers of automated decision systems to test them for biases two years ago. She was motivated by being a woman in a male-dominated field and by seeing how such systems were already being used to disadvantage people, particularly women and people of color, she told The Examiner. She said she and her staff met with business community representatives early on in the development of the bill and have met with them numerous times since. “I honestly thought when I started this bill there would be more of a partnership with business,” she said. Michigan provides an example of how such a system can go awry. Between 2013 and 2015, an automated system used by the state’s unemployment insurance agency wrongly accused 40,000 residents of fraud and blocked them from getting benefits. Some lost their properties, some went into bankruptcy and many continued to fail credit checks for years afterward. It took years for the state to address and correct the issue; Michigan didn’t reach a final settlement with some of the victims, totalling $20 million, until last year. AB 1018 is intended to ensure that such outcomes never happen, Bauer-Kahan said. She pointed to a separate example from the same time period of how things ought to work Starting in 2014, Amazon reportedly started developing an automated system to review résumés to find the top candidates. By the next year, the company realized that the system was biased in favor of male applicants. Because the tech industry is dominated by men, many of the résumés that had been submitted to Amazon from the past had been from males, and the system, which was trained on that data, essentially taught itself to prefer them.That experience “tells me,” Bauer-Kahan said, that asking companies to scrutinize automated systems for bias “is totally doable.” Even so, her first two attempts to legislate such testing stalled out in the last two legislative sessions. It’s still to be determined whether this year’s bill will as well. The State Assembly overwhelmingly passed AB 1018 in June, as did the Senate Judiciary Committee last month. But on Monday, the Senate Appropriations Committee unanimously voted to put the legislation on its suspense file, which it does with bills that could cost the state at least $50,000. As CalMatters has reported, that move is frequently the first step in killing a bill without a public vote. The Appropriations Committee’s vote came in the wake of a bill analysis by its staff that reported that state agencies collectively said the legislation would cost them potentially hundreds of millions of dollars to implement. It’s unclear how the agencies came up with such estimates, Gordon said. She noted that according to a CalMatters report in May, the state reported that as of last August, no state agency was using an automated system to make decisions related to housing, employment, credit, education, health care or criminal justice. Coming out of that report, TechEquity figured that “since none of them are using , there should not be high-cost burden” for public agencies to comply with AB 1018, she said. “Seeing those estimates come in, honestly, my reaction is confusion.” Regardless, such costs are well worth it, she said, because such costs are likely a fraction of the costs of buying and implementing automated decision systems. And the costs when such systems go wrong can be much higher than testing them. Michigan, for example, is spending $78 million more to replace the unemployment insurance system that prompted the $20 million settlement. Policymakers shouldn’t just focus on the costs of testing these systems or offering notifications about their use, Gordon said. Instead, she said, they should keep in mind that “the costs are to all of us” when the systems get things wrong. If you have a tip about tech, startups or the venture industry, contact Troy Wolverton at twolverton@sfexaminer.com or via text or Signal at 415.515.5594.
Artificial Intelligence California Legislature Discrimination
United States Latest News, United States Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
– Rep. Marlin Stutzman’s MAHA Bill Requires Food Label for Bill Gates-Funded ProductSource of breaking news and analysis, insightful commentary and original reporting, curated and written specifically for the new generation of independent and conservative thinkers.
Read more »
Best Political Tweets 7-25-2025'Oh no I really hope Bill Clinton isn’t in the Epstein files, I’ll have to get rid of my Bill Clinton hat and my Bill Clinton sneakers and my Bill Clinton cologne and my Bill Clinton watch and take down my Bill Clinton flag and I’ll have to delete my Bill Clinton nft trading cards.
Read more »
Starting A Business Isn’t Enough – You Need A Strategy To Make MoneyStarting a business is not the same as building a profitable business. Passion is powerful, but it’s not a business model.
Read more »
Chamber of Commerce finds Americans favor pro-business policies in 'big, beautiful bill'The U.S. Chamber of Commerce found in its polling that American voters support crucial pro-business tax provisions in OBBBA despite opposition to the overall legislation.
Read more »
Bill Maher Drags ‘The View’ Hosts Following Recent Controversial CommentsBill Maher poses for a promotional photo for &39;Real Time With Bill Maher&39;
Read more »
Phoenix horseman expands business to new Gilbert ranch during Black Business MonthCameron Polom is an award-winning multimedia journalist who joined ABC15 in July 2017.
Read more »