The bill, SB 1047, was introduced seven months ago by Sen. Scott Weiner and passed the state assembly on Wednesday.
It’s now on Gov. Gavin Newsom’s desk, and he has until September 30 to decide whether to sign it into law or veto it. He has not publicly signaled his position.
The bill aims to force safety measure development at companies that spend $100 million or more training AI models so their tech cannot be used to harm society, such as creating dangerous weapons or undertaking cyberattacks.
The measures include forcing companies operating in California to report any safety incidents to the government, protecting whistleblowers, and allowing third parties to test their models for safety. If necessary, they would also push companies to enact a full shutdown.
Picking teams
The bill has drawn a line through Silicon Valley, and some of tech’s biggest figures have taken sides.
Jason Kwon, OpenAI’s chief strategy officer, warned in a letter to Sen. Wiener on Wednesday that the bill could stifle progress and drive companies out of California.
The ChatGPT maker joined heavyweight Meta in its lobbying against the bill. Meta said that the bill could discourage the open-source movement by exposing developers to significant legal liabilities.
Former OpenAI employees said that the company’s opposition to the bill was disappointing — but in line with its recent path.
“We joined OpenAI because we wanted to ensure the safety of the incredibly powerful AI systems the company is developing,” former OpenAI researchers, William Saunders and Daniel Kokotajlo, wrote in the letter. “But we resigned from OpenAI because we lost trust that it would safely, honestly, and responsibly develop its AI systems.”
Elon Musk also supported the bill.
In a social media post on Monday, Musk, who founded AI company xAI last year and has a long-standing rivalry with OpenAI’s Altman, said that although it was “a tough call and will make some people upset,” he thinks “California should probably pass the SB 1047 AI safety bill.”
“For over 20 years, I have been an advocate for AI regulation, just as we regulate any product/technology that is a potential risk to the public,” Musk wrote on X.
Amazon-backed Anthropic seemed to have switched sides in the middle of the debate, after the bill was modified.
In a letter sent to California Gov. Gavin Newsom last week, Anthropic’s CEO, Dario Amodei, said the bill’s “benefits likely outweigh the costs.” However, he added that “we are not certain of this, and there are still some aspects of the bill which seem concerning or ambiguous to us.”
Amodei said the bill now “appears to us to be halfway between our suggested version and the original bill.”
