Despite claiming to support AI safety, powerful tech interests are trying to kill SB1047.
Meta CEO Mark Zuckerberg arrives for an interview on The Circuit with Emily Chang at Meta headquarters in Menlo Park, California, on Thursday, July 18, 2024.
(Jason Henry / Bloomberg via Getty Images)
Artificial intelligence could help us solve humanity’s greatest challenges. But, left unchecked, it could cause catastrophic harm. Well-designed regulation will allow us to harness AI’s potential, while securing us from its potential to do harm—not through bureaucrats’ specifying technical procedures, but through rules that ensure that companies embrace safe procedures.
California is on the brink of passing regulation to start to do just that. Yet, despite universal recognition among leading AI executives of the risks their work poses, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act (SB1047) has become the target of an extraordinary lobbying effort. The bill is said to be certain to stifle technical innovation in Silicon Valley and almost purposely designed to end “open-source” AI development.
Malarkey. This fight is less about “corporate capture” or the California legislature’s desire to kill its golden-egg-laying goose than it is about the same-as-it-ever-was power of money in American politics. If the bill fails—and next month will determine whether it does—it will signal yet again the loss of America’s capacity to address even the most significant threats.
Who’s Afraid of AI Safety?
At its core, SB1047 does one small but incredibly important thing: It requires that those developing the most advanced AI models adopt and follow safety protocols—including shutdown protocols—to reduce any risk that their models are stolen or deployed in a way that causes “critical harm.”
Which models? Initially, simplifying, models that cost $100,000,000 or more to train or models fine-tuned at a cost of $10,000,000 or more.
Current Issue
“Critical harm”? The law covers models that lead to the “creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties” or that lead to cyberattacks on critical infrastructure costing more than $500,000,000 or that, acting with limited human oversight, result in mass casualties or damage greater than $500,000,000.
Thus, the law covers an incredibly small number of model builders so as to avoid potentially huge harm. And the mandate that it deploys to avoid “critical harm” simply requires that companies adopt robust protocols to increase model safety. The law does not specify what those protocols must be. It simply requires the company, considering the state of the industry and entities advising the industry, to adopt rules to ensure that a small slice of their products are safe.
In some sense, every company developing models of this size would say it has already adopted such safety protocols. So then, why the opposition?
The problem for tech companies is that the law builds in mechanisms to ensure that the protocols are sufficiently robust and actually enforced. The law would eventually require outside auditors to review the protocols, and from the start, it would protect whistleblowers within firms who come forward to show that protocols are not being followed. The law thus makes real what the companies say they are already doing.
But if they’re already creating these safety protocols, why do we need a law to mandate it? First, because, as some within the industry assert directly, existing guidelines are often inadequate, and second, as whistleblowers have already revealed, some companies are not following the protocols that they have adopted. Opposition to SB1047 is thus designed to ensure that safety is optional—something they can promise but that they have no effective obligation to deliver.
That companies would want to avoid regulation is not surprising. What is surprising is how awful the arguments against the bill have been—especially by people who should know better.
To start with, members of Congress have written to the bill’s sponsor, State Senator Scott Wiener, telling him, “The bill requires firms to adhere to voluntary guidance issued by industry and the National Institute of Standards and Technology, which does not yet exist.” That is simply not true. The bill simply requires developers to “consider industry best practices and applicable guidance” from organizations like NIST, which NIST has already begun to supply.
These representatives continue by saying that they object that the bill “is skewed toward addressing extreme misuse scenarios and hypothetical existential risks while largely ignoring demonstrable AI risks like misinformation, discrimination, nonconsensual deepfakes, environmental impacts, and workforce displacement.” It is true, of course, that the bill is not focused on lots of other AI risks. California—and Congress!—ought to address those risks, too. Indeed, California alone introduced over 50 AI bills this year, many of which are targeted at those risks. Yet how that is an argument against addressing the risks the law is addressing is not clear. It’s like saying that a bill addressing wildfire risks should be rejected because it doesn’t address flooding risks.
But consider the term “hypothetical existential risks”: Many in the field of AI have spoken of the “existential risks” that advanced AI may present—“existential” in the sense that if they are realized, humanity is over. Those risks are not SB1047’s direct concern. Its focus is on more practical harms, such as cyberattacks on critical infrastructure or economic harm of $500M or more. Every AI company that is likely to train the models that this bill would regulate—including companies such as OpenAI, Google, and Meta that oppose it—believes, or says it believes, that its most powerful AI models might pose these sorts of risks in the not-too-distant future.
Regardless, how should we think about these severe risks more generally? Some believe such risks are unavoidable. They reject the term “hypothetical.” Some believe such risks don’t exist: Like time travel, they can be imagined, but they cannot be realized. Yet most speak of these risks in probabilities, for example, “a 10 percent chance in 10 years.”
It’s not clear which of these three possibilities these members of Congress mean. They write, “There is little scientific evidence of harm of ‘mass casualties or harmful weapons created’ from advanced models.”
However, no one is claiming that we have seen “mass casualties or harmful weapons created” so far. The point of the bill is to avoid such harm, especially as models become so enormously powerful. (There is also little scientific evidence of harm of mass casualties or harmful weapons created from bioengineering; is that a reason not to regulate bioengineering?) Sure, if you’re certain such risks could not be realized then there’s no reason for this bill. But when did members of Congress become experts in AI?
However, the representatives continue further by echoing shibboleths about open-source AI. “Currently, some advanced models are released as open source and made widely available,” they write. “This openness allows smaller, lesser-resourced companies and organizations, including universities, to develop on top of them, stimulating innovation and having large economic impact.”
That’s true enough. But then, as the members’ letter continues, truth begins to fade. They say, “This bill would reduce this practice [of open-source development] by holding the original developer of a model liable for a party misusing their technology downstream.”
Popular
“swipe left below to view more authors”Swipe →
Wrong. The bill creates no liability simply because someone “misus[ed]” a “technology downstream.” No doubt, it imposes upon “developers” of “covered models” the obligation to take “reasonable care to implement…appropriate measures to prevent covered models and covered model derivatives from posing unreasonable risks of causing or materially enabling critical harms”—again, “mass casualties” or economic loss exceeding $500M. Who is against that? What industry in America today, without explicit legislative exemption, is entitled to deploy, without regulation or liability, a product that creates “unreasonable risks of causing…critical harms”?
Open-source software developers have long used licenses to avoid economic liability for harms that flow from their software. For ordinary economic harm, that may well make sense. But the law of tort—independent of, but codified in important ways by, SB1047—is not bound by software licenses. In the face of “critical harms,” let alone existential risks, there is no good reason to exempt developers of software from the ordinary duty that everyone else bears: to take “reasonable care” to implement “appropriate measures” to avoid “unreasonable risk.”
What’s more, targeting the shutdown protocols that the law requires the companies to develop, the representatives write that “kill switches” “would decimate the ecosystems that spring up around [open-source] AI models.” No entrepreneur would want to build a product around an AI system if the developer could pull the plug at any time.
Again, this is wrong. First, the law does not require anyone to build a “kill switch.” It merely requires developers to have the ability to shut down their own software. Second, the law does not require anyone to enact that shutdown at any time. It only requires that the companies develop the capability and describe the protocols governing when they would be used. The rule is like a regulation requiring companies building electrical grids to include circuit breakers in their design. Do circuit breakers “decimate the ecosystems” of companies developing electrical products?
Third, the law requires “full shutdown” capability for models “controlled by” a developer. Once code is adopted and deployed by others, so long as those others are not “developers” of “covered models,” the obligation does not reach them. But fourth, and most bizarrely, imagine an open-source model did have a kill switch, and imagine the developer flipped it because a runaway model began to cause “critical harm”—again, “mass casualties” or economic harm of $500M or more. Are these members arguing that the developer should not flip the switch? Or that the entrepreneur using the model would rather cause “critical harm” than have its model stopped?
Indeed, the argument goes the other way around. Having circuit breakers built into the system makes it more likely that companies will develop products based on open-source technologies if only to avoid the ordinary tort liability that would follow any harm that those products would produce. A company building its product on top of an unreasonably dangerous product could itself face tort liability. The capability to stop runaway critical harm could make the underlying product more valuable to follow-on developers, not less.
Finally, the members of Congress write that a recent NIST report recommended that the “government should not restrict access to open-source models with widely available model weights at this time.” True, it shouldn’t, but nothing in SB1047 would. The bill does nothing to “restrict access” to models; it only requires that “developers” of “covered models” (again, those spending $100M or more) or covered “fine-tuned” models (again, those spending $10M or more), develop protocols to advance the safety of those models, at least to the extent reasonable, given the state of knowledge in the field.
A Simple First Step
SB1047 is a protocol bill. It mandates that a handful of companies take meaningful steps to adopt procedures that the leaders of every one of these companies agree such models could conceivably create.
The bill isn’t perfect. There are plenty of ways in which it could be improved. I agree with much in Anthropic’s balanced and insightful analysis of the bill—an analysis by one of the companies that would be regulated that yet concludes that the benefits of the bill outweigh the costs.
But every bill is imperfect. And this one has one more important argument in its favor: If Donald Trump is elected, he has promised to immediately remove even the minimal protections that the Biden administration has imposed. That would be catastrophic. SB1047 is not a substitute for those protections, but it is a backstop and a critical first step.
Can we count on you?
In the coming election, the fate of our democracy and fundamental civil rights are on the ballot. The conservative architects of Project 2025 are scheming to institutionalize Donald Trump’s authoritarian vision across all levels of government if he should win.
We’ve already seen events that fill us with both dread and cautious optimism—throughout it all, The Nation has been a bulwark against misinformation and an advocate for bold, principled perspectives. Our dedicated writers have sat down with Kamala Harris and Bernie Sanders for interviews, unpacked the shallow right-wing populist appeals of J.D. Vance, and debated the pathway for a Democratic victory in November.
Stories like these and the one you just read are vital at this critical juncture in our country’s history. Now more than ever, we need clear-eyed and deeply reported independent journalism to make sense of the headlines and sort fact from fiction. Donate today and join our 160-year legacy of speaking truth to power and uplifting the voices of grassroots advocates.
Throughout 2024 and what is likely the defining election of our lifetimes, we need your support to continue publishing the insightful journalism you rely on.
Thank you,
The Editors of The Nation
More from The Nation
Extreme heat has long been a concern for incarcerated pregnant women and those behind bars with underlying health conditions.
![On Feb. 27, 2013, just a few hours after Solicitor General Donald Verrilli endeavored to protect the Voting Rights Act before the U.S. Supreme Court, he dropped by his boss’s office and told US Attorney General Eric Holder that arguments in the case, Shelby County v. Holder, had gone poorly. Verrilli warned Holder things might be worse than they feared: The Court intended to go after the entirety of preclearance, the VRA’s most crucial enforcement mechanism. Holder scarcely believed that could be possible. “The Voting Rights Act? Come on,” Holder said. But to the veteran solicitor general, recovering from the most brutal experience he’d ever had before the Court, the writing was on the wall. “I walked out of that courtroom certain that’s what was going to happen,” Verrilli told me. “I was never optimistic at all.” Verrilli had underestimated how meticulously John Roberts had planned for this moment. He never imagined that he’d be hit with such mendacious numbers and arguments in a sanctum he revered. And though his pessimism turned out to be abundantly justified, the dishonest reasoning behind Roberts’s decision – that the chief justice could just conjure a doctrine of his own creation and use it to eviscerate the most important civil rights legislation in the nation’s history – haunts the solicitor general to this day. It has remade American democracy as well. [dropcap]J[/dropcap]ohn Roberts had schemed for decades prior to this moment. Three years earlier, a test case known as Northwest Austin allowed Roberts to carefully plant the seeds for the challenge that foes of voting rights law mounted in Shelby County. A tiny municipal water board in a new Texas development posed a large constitutional question. The VRA’s preclearance regime required every locality in the state to -approve in advance any changes to voting procedures through the Department of Justice. Preclearance worked to right historical wrongs: The VRA mandated it in the handful of states and localities with the worst records of racial discrimination in elections; when Congress reauthorized the Voting Rights Act by near-unanimous margins in 2006, it relied on a record of ongoing modern chicanery stretching toward 14,000 pages, bearing eloquent testimony to the ongoing need for preclearance. Two lower federal courts had agreed that the utility district did not qualify for exemption from the VRA, and that preclearance itself remained a proportionate response by Congress. “The racial disparities revealed in the 2006 legislative record differ little from what Congress found in 1975,” wrote federal appeals Judge David Tatel. “In view of this extensive legislative record and the deference we owe Congress, we see no constitutional basis for rejecting Congress’s considered judgment.” Before Tatel wrote his decision, he read every page of the 2006 congressional report, and tracked the stories of canceled elections and last-second precinct switches across Mississippi, Louisiana and other covered states. He did so, he told me, because “I had no confidence that the Supreme Court would ever look at the record.” His fears were justified. On April 29, 2006, during the oral arguments over Northwest Austin, Roberts expressed impatience, and sounded as if he simply didn’t believe these challenges continued. “Well, that’s like the old elephant whistle. You know, I have this whistle to keep away the elephants,” he said, dismissively, as the defense pointed out the ongoing need for preclearance in the case. “There are no elephants, so it must work. “Obviously no one doubts the history here,” he added. “But at what point does that history stop justifying action with respect to some jurisdictions but not with respect to others . . . . When do they have to stop?” Neal Kaytal, then the principal deputy solicitor general, responded that since Congress had reauthorized the VRA for another 25 years, that date would be 2031. Roberts was unimpressed. “I mean, at some point it begins to look like the idea is that this is going to go on forever.” The court’s five conservatives wanted to move on, and several appeared ready to address the larger constitutional issues that would trigger a challenge the VRA, but didn’t have much beyond vibes to go on. So Roberts brokered a deal, and wrote an apparently unifying decision for a court that appeared deeply divided during oral arguments. Everybody won, sort of. The water district would be allowed to bail out of preclearance requirements. Other small entities were invited to apply for a reprieve. At the same time section 5—laying out the VRA’s preclearance regime— survived. The liberal justices bought time for Congress to potentially address the court’s impatience with the preclearance formula once more. “It wasn’t exactly a principled constitutional decision,” says Tatel, who had scoured the law to see if he could deliver a similar ruling that sanctioned a bailout for the water district, in part to keep the VRA away from the high court. But the law clearly didn’t allow it. (“It doesn’t work with the law. It’s not right,” he told me, “but that didn’t bother the court.”) “They made a deal,” says Edward Blum, who helped bring the challenge to the VRA, and would also mastermind the clutch of cases that led the court to end affirmative action in college admissions. Roberts seemed to have done the impossible, and he won praise from the media and court watchers for his measured and far-seeing “judicial statesmanship.” Liberals even claimed victory, crowing to The New Republic that they prevented the VRA from being struck down 5-4 by threatening some thunderous dissent that either led Justice Kennedy to get cold feet or the chief justice to back down. If the liberals wanted to celebrate an imaginary win, Roberts had no problem with that. The chief justice was busy digging a trench and setting a trap. Indeed, the liberal justices scarcely seemed to notice the actual language of the opinion that they signed onto. “Things have changed in the South,” Roberts declared, writing for the full court. The VRA, he wrote, “imposes current burdens and must be justified by current needs.” Roberts had conned the liberals into signing onto a broad indictment of the reasoning behind preclearance, aimed at a future case and a future decision. “The statute’s coverage formula is based on data that is now more than 35 years old,” Roberts argued, as though civil-right legislation had a sell-by date. “And there is considerable evidence that it fails to account for current political conditions.” Then Roberts made one additional stealth play that helped assure that the next challenge to the VRA would arise quickly and would be aided by the plaintiffs’ success in Northwest Austin. He made an observation known as dicta—a comment that might not be necessary to resolve a case or even be legally binding in the future, but that can be cited as a “persuasive authority.” This is where Roberts gave birth to the fiction of a “fundamental principle of equal sovereignty” among states. The trouble with the principle is that it does not exist. Roberts created it with an ellipsis and what can only be understood as deliberate misapplication of the law. Indeed, the Supreme Court had rejected this precise reading in a 1966 voting rights case that upheld the Act’s constitutionality, in the very sentence that Roberts later claimed said the opposite. How did he get away with turning up into down? He cut the clauses he didn’t like and called it law. Here is the actual decision from the case in question, Katzenbach v South Carolina: In acceptable legislative fashion, Congress chose to limit its attention to the geographic areas where immediate action seemed necessary . . . . The doctrine of the equality of States, invoked by South Carolina, does not bar this approach, for that doctrine applies only to the terms upon which States are admitted to the Union, and not to the remedies for local evils which have subsequently appeared. And here is what Roberts wrote: The Act also differentiates between the States, despite our historic tradition that all the States enjoy “equal sovereignty.” Distinctions can be justified in some cases. “The doctrine of the equality of States … does not bar … remedies for local evils which have subsequently appeared.” But a departure from the fundamental principle of equal sovereignty requires a showing that a statute’s disparate geographic coverage is sufficiently related to the problem that it targets. Before Roberts wrote this, there was no such principle—let alone a fundamental one. The cases that Roberts cites as authorities for the idea of equality among states actually concern the “equal footing doctrine,” which secures equality among newly admitted states. No fundamental principle of equality among states governs the Fifteenth Amendment, which explicitly hands Congress the power to enact appropriate legislation to ensure equal treatment of all voters within states. Liberal justices either didn’t notice the dicta or did not think that Roberts would be so brazen as to write it into law citing his own invented precedent the next time a preclearance case came before the court. They underestimated both his chutzpah and hubris. It would not be the first, or the last time. [dropcap]I[/dropcap]f Roberts weakened the VRA’s foundations in Northwest Austin, four years later, in Shelby County, he came with the bulldozer. The preclearance formula had become outdated and no longer considered “current conditions,” Roberts wrote, brushing aside the lengthy congressional record filled with modern-day examples. Singling out states for disparate treatment, he held, failed to accord each state its “equal sovereignty.” The decision appeared modest and suggested that the Court had little choice but to act. But the feigned modesty was pure misdirection. The Shelby County decision is a deeply radical one. It usurps powers the Constitution specifically awards Congress. It uses the fictional principle of equal sovereignty, cooked up by Roberts in 2009, as its basis. And it cites statistics that are factually wrong and misstate the U.S. census. In the wake of other draconian hard-right decisions, such Dobbs v. Jackson Women’s Health (2022), reversing the right to abortion, and Loper Bright Enterprises v. Raimondo (2024), striking down “chevron deference” and effectively undermining the authority of federal regulatory agencies, public confidence in the high court has plummeted. Large majorities of Americans see the justices as partisan proxies, viewing the law through whatever lens might create victories for their side. But even this disillusioned American majority may not suspect that the justices might just be making the law up as they go along—that their opinions carry footnotes and the force of law but stand on air. Nor do most Americans fully appreciate that five, and now six, of them can get away with this because they form a majority ideological bloc, accountable to no one. That’s the real story behind John Roberts’s opinion in Shelby County. The chief justice obliterated the most successful civil rights legislation this nation has ever seen not just on flimsy criteria but on none at all. “It’s made up,” the conservative judge and law professor Michael McConnell, a George W. Bush appointee, told NPR. “This is a principle of constitutional law of which I had never heard,” the conservative judge and legal scholar Richard Posner observed, “for the excellent reason that … there is no such principle.” “Yes, that’s right,” says Leah Litman, a law professor at the University of Michigan. “There are passing references to the idea of equal sovereignty. But if you pause and think about them for more than a second, it’s clear that [Roberts] made the doctrine into something that it just wasn’t.” Litman is the national authority on equal sovereignty. In 2016, she wrote a complete 67-page history of what Roberts called a “fundamental principle” and “historic tradition.” Her conclusion? Roberts manufactured it for his own purposes. It is, she writes, an “invented tradition”---invented by John Roberts, and then cited by John Roberts. What Roberts didn’t make up, he got wrong. Roberts based his reasoning that things had changed in the South on voter registration statistics that, to him, showed that Blacks and whites had reached something close to parity—and that in some states, Blacks had even surpassed whites. His opinion even included a chart ostensibly documenting this. In reality, though, Roberts had the statistics backward. They did not show what he said they did. In many cases, they showed the opposite. Roberts used the numbers from the Senate Judiciary Report—the one that Republicans generated after the VRA’s passage to plant a record for its judicial demise. And intentionally or otherwise, the GOP report got it wrong. Roberts and the committee overstated white registration numbers. The Roberts chart counted Hispanics as whites—even those who were not U.S. citizens and therefore ineligible to vote. That basic error threw off all the demographic comparisons. In Georgia, for example, Roberts claimed that Black registration had risen to 64.2 percent and white registration had fallen behind at 63.5. But without the Hispanic numbers, white registration grew to 68 percent. It’s an improvement from 1965. But it’s not an example of Black registration outpacing whites, as Roberts claimed. In Virginia, meanwhile, Roberts argued that the gap between whites and Blacks had narrowed to just 10 percent—when in reality, it was more than 14 percent. Roberts simply didn’t understand how the census reported race. The Bureau treats race and ethnicity differently. Hispanics are counted as an ethnicity, then usually included under white. The chief justice should have used the data for white-non-Hispanic. But that would not have given Roberts the result he wanted. When reporters asked the court to explain how he could have gotten something so basic so wrong, the chief justice declined to answer questions. The court “does not comment on its opinions,” said a spokesperson, “which speak for themselves.” [dropcap]“T[/dropcap]his was Congress’s decision to make,” solicitor general Verrilli told me—a power awarded explicitly by the Reconstruction amendments to the Constitution. Debo Adegbile, who defended the VRA before the court during both Shelby County and Northwest Austin, sees the long throughline of the Court’s resistance to the full sweep of those amendments. The court’s impatience with the past, its eagerness to declare the job complete and the nation whole, reminded him of the 1870s Cruikshank and Civil Rights Cases decisions that choked off Reconstruction and insisted Blacks must “cease to be the special favorites of the law” even as freed slaves carried scars of their bondage. “It’s just the continued resistance to the commitment to make the country whole and to be an inclusive democracy,” he told me. “And it’s being dressed up in sophisticated legal arguments. It’s not that we’re actually past anything. It’s that we are now at a point where we have the power to decide that we’re going to vary from the mission, create a situation where voters are exposed and . . . advantage the manipulations of state actors and local actors to impose barriers.” Holder still stammers in disbelief. “Okay, Mr. Chief Justice, you say that America has changed. OK. And what’s your basis for saying that, as opposed to Congress holding hearings, thousands of pages of testimony, hundreds of exhibits that say America has changed some, but not enough? You’re saying, ‘No, Congress, essentially you’re wrong.’ . . . OK. Then where were your researchers?” The former attorney general winces in horror when the case is referred to by its full name: Shelby County v Holder. He cites two days as the worst of his tenure: The day he accompanied President Barack Obama to console parents of children slain during the Sandy Hook massacre, and “the other one was to hear from the Supreme Court that the Voting Rights Act of 1965 was, in substantial ways, murdered.” “Nothing had changed in the South,” he told me. “The only thing that changed was the personnel on the U.S. Supreme Court.” Verrilli also replays this crushing defeat in his head, wondering if there was anything he could have done to guard the Voting Rights Act against implacable foes. “I wish I could tell you, David, that I have stopped doing that, but I have not. It haunts me to this day.” He slows and wipes his eyes. It’s clear he is fighting back tears, unsuccessfully. “I think all the time about what I might have done differently, because it was a devastating defeat and it had huge consequences. I take solace in the thought that I don’t think there’s anything I could have done differently. But that only makes it marginally less powerful.” Adapted from Antidemoccratic: Inside the Far Right’s 50-Year Plot to Control American Elections by David Daley, Mariner Books, 2024. All rights reserved](https://www.thenation.com/wp-content/uploads/2024/08/RobertCrop.jpg)
On February 27, 2013, just a few hours after Solicitor General Donald Verrilli endeavored to protect the Voting Rights Act before the US Supreme Court, he dropped by his boss’s off…
These are some concrete steps a President Harris could take to undo the damage of her current boss.
As always, the DNC was an endurance grind. But my serendipitous encounter with a woman who embodies Harris’s reproductive justice agenda was the high point for me.
From district courts all the way up to the Supreme Court, judges are refusing to retire, holding on to power well past their prime.
