California advances landmark legislation to regulate large AI models

A California landmark legislation to establish first-in-the-nation safety measures for the largest artificial intelligence systems cleared an important vote Wednesday that could pave the way for U.S. regulations on the technology evolving at warp speed.

The proposal, aiming to reduce potential risks created by AI, would require companies to test their models and publicly disclose their safety protocols to prevent the models from being manipulated to, for example, wipe out the state’s electric grid or help build chemical weapons — scenarios experts say could be possible in the future with such rapid advancements in the industry.

The bill is among hundreds lawmakers are voting on during its final week of session. Gov. Gavin Newsom then has until the end of September to decide whether to sign them into law, veto them or allow them to become law without his signature.

The measure squeaked by in the Assembly Wednesday and requires a final Senate vote before reaching the governor's desk.

Supporters said it would set some of the first much-needed safety ground rules for large-scale AI models in the United States. The bill targets systems that require more than $100 million in data to train. No current AI models have hit that threshold.

“It’s time that Big Tech plays by some kind of a rule, not a lot, but something,” Republican Assemblymember Devon Mathis said in support of the bill Wednesday. “The last thing we need is for a power grid to go out, for water systems to go out.”

The proposal, authored by Democratic Sen. Scott Wiener, faced fierce opposition from venture capital firms and tech companies, including OpenAI, Google and Meta, the parent company of Facebook and Instagram. They say safety regulations should be established by the federal government and that the California legislation takes aim at developers instead of targeting those who use and exploit the AI systems for harm.

A group of several California House members also opposed the bill, with Former House Speaker Nancy Pelosi calling it “ well-intentioned but ill informed.”

Chamber of Progress, a left-leaning Silicon Valley-funded industry group, said the law is “based on science fiction fantasies of what AI could look like.”

“This bill has more in common with Blade Runner or The Terminator than the real world," Senior Tech Policy Director Todd O’Boyle said in a statement after the Wednesday vote. “We shouldn’t hamstring California’s leading economic sector over a theoretical scenario.”

The legislation is supported by Anthropic, an AI startup backed by Amazon and Google, after Wiener adjusted the bill earlier this month to include some of the company's suggestions. The current bill removed the penalty of perjury provision, limited the state attorney general's power to sue violators and narrowed the responsibilities of a new AI regulatory agency. Social media platform X owner Elon Musk also threw his support behind the proposal this week.

Anthropic said in a letter to Newsom that the bill is crucial to prevent catastrophic misuse of powerful AI systems and that “its benefits likely outweigh its costs.”

Wiener said his legislation took a “light touch” approach.

“Innovation and safety can go hand in hand—and California is leading the way,” Weiner said in a statement after the vote.

He also slammed critics earlier this week for dismissing potential catastrophic risks from powerful AI models as unrealistic: “If they really think the risks are fake, then the bill should present no issue whatsoever.”

Wiener's proposal is among dozens of AI bills California lawmakers proposed this year to build public trust, fight algorithmic discrimination and outlaw deepfakes that involve elections or pornography. With AI increasingly affecting the daily lives of Americans, state legislators have tried to strike a balance of reigning in the technology and its potential risks without stifling the booming homegrown industry.

California, home of 35 of the world’s top 50 AI companies, has been an early adopter of AI technologies and could soon deploy generative AI tools to address highway congestion and road safety, among other things.

Newsom, who declined to weigh in on the measure earlier this summer, had warned against AI overregulation.