The AI Safety Crisis Is Growing: Why Researchers Are Speaking Out

Insiders are sounding a major A.I. alarm and as Anthropic and OpenAI rack up mass evaluations, concerns about A.I. safety are reaching a boiling point, with researchers speaking out and even leaving their jobs over fears that the technology could outpace human control. Joining us now to break down what these growing concerns could mean for the future of A.I. is Christopher Zara, News Director at Fast Company. Christopher, thank you so much for joining us. Yeah, thanks for having me. Now, this week there has been a lot of attention, of course, around an A.I. researcher who recently left Anthropic after raising concerns about A.I. potentially getting out of control. How significant are those concerns and how significant was that development? Yeah, I think to the second question, the development was extremely significant. The concerns have been around for a while, but I think what's changed this week is people are really paying attention to them because of, as you said, this young researcher, Jacob Coxon, who resigned publicly and put out a tweet that kind of went viral. The other thing that happened with this tweet is when he expressed these concerns, his concerns were endorsed, essentially, by someone who's still working at open A.I. at a very high level, not open A.I., I'm sorry, Anthropic at a very high level. So you have this kind of snowball effect where all of a sudden all these people were talking about it online and it just became a major news story. And here we are talking about it again on Friday. So I think the turning point aspect of this is very significant now that it's in the public conversation. The concerns about the dangers of A.I. and the threat that it poses, I think, have been around for a while. What's changed now is people are really paying attention. And as you said, we're seeing some more A.I. workers publicly speaking out about those safety concerns. Why do you think we're seeing more now? Is it because this one researcher went viral and it's been a headline? Or are people starting to see more of those concerns? You know, it's hard to say why something goes viral like this at any given moment. I think a few things happened this year. The one big thing that happened earlier this year was a swarm of A.I. agents from open A.I. attacked a website called Hugging Face. This was something they sort of did on their own volition. They weren't told to do it. They kind of operated in secret. And then it was like discovered later. This was not a catastrophic hacking of any sort. But the fact that it happened at all really kind of changed a lot of people's views on what could potentially happen when A.I. gets a lot smarter. So if they can do this at this level, where we are now, think about what they can do four or five years from now or even sooner. And I think that's what Jacob Coxon was essentially saying in his tweet and then in the later endorsement of the tweet was that we're really worried about what happens when we get to that next level and A.I. becomes, you know, quote unquote, super intelligent. And I think you have that plus the idea that it can act on its own and do its own thing. And it becomes pretty concerning. So yeah, that Hugging Face incident that happened earlier this year, I think really sort of changed the conversation. And now, you know, this tweet that went viral is kind of continuing that conversation. Right. By making that move of leaving a company like Anthropic, how does that actually help move the conversation around A.I. safety forward, especially in a case like this that it is getting a lot of attention? Well, I mean, one way is potential regulation that could come out of it. I think that you've already seen Bernie Sanders, for example, has a proposal that would both slow down the pace of A.I. research as it's happening and also sort of ban this idea of super intelligent A.I. altogether. Now, how realistic that regulation is, of course, is probably a big debate. I would say that a lot of people might say that's not very likely to happen given the power that these companies have. But they do want some kind of regulation and they do want cooperation from governments and from other A.I. companies to sort of try to get on the same page here before it's too late. So one way I hope that it could advance the conversation is by having a bit more coordination and communication between governments and the companies. As these companies continue to push forward, how much tension is there between that push to build more powerful A.I. systems, but also the push to make sure that they're safe? Well, I mean, they seem to be plowing ahead regardless. And I don't think you've heard anyone really say we're going to stop because I think part of the reason that they're not stopping is because they figure their competitors are going to keep going. And that's not just competitors within the United States, but competitors within countries like China. So you have this sort of global arms race, to put it bluntly, of A.I. technologies that it feels like each player is just a little bit too afraid to step back and say we're not going to do this because they think their competitors are going to do it. So it's a tough position to be in. Definitely. And with both Anthropic and OpenAI, they're both seeing enormous investor interest, which I'm sure is another major factor. How does that kind of money create more pressure for both of these companies to feel like they need to be moving as quickly as possible? Yeah, there's enormous pressure. I mean, both OpenAI and Anthropic are planning to go public. Anthropic, I believe, is planning to go public a little bit sooner. The latest reporting shows that they may publicly disclose their documents as early as next month. It would be the largest IPO in history if the reporting is accurate. We just had the largest IPO in history with SpaceX, which has a large A.I. component to it. But this would even beat SpaceX at like a two trillion valuation, which is just insane to think about. The stakes are pretty high from a monetary sense. Unfortunately, the stakes are really high from a humanity sense as well. As business journalists, we don't often get to talk about the extinction of the entire human species. We write stories with high stakes, but not usually this high. You have to have this debate and really confront it before we can move forward, or we're in a lot of trouble. And I think that's what this researcher who quit, I think that's what he was trying to say. We have to get a handle on this before it's too late. So whether or not that will happen is, of course, an open question. But I think that's where the conversation is now. Right. Well, with these existential risks that are getting a lot of attention, we're also seeing others come out and say that those fears are overblown. So where do you feel the expert community actually stands? Yeah. So, I mean, you do hear a lot of people... Actually, you heard a lot more people saying that the fears were overblown, I think, earlier in the year. I still think there are a lot of people who think... It's really hard to separate like what's the marketing spin part of this. And I think that my personal default position, I think probably for most of this year, has been to say, well, these companies have a huge stake in sounding very important. So, of course, they're going to tout the abilities of their technology and in ways that make it sound very important. But I do think what changed again was what happened with Hugging Face earlier this year. And there was another incident where there was AI swarms acting on their own. I think when you start to couple that with what these researchers have basically been saying, you can't ignore the possibility that there's a real concern here. And at the end of the day, we don't need to rush this technology as fast as we are rushing it. The human race has survived a long time without all this AI technology. So I think that when you look at the potential stakes, even if there's a little bit of marketing spin to it, you do have to take it seriously. Yeah, it's impossible to predict the future, of course. But is it possible that we could see a growing wave of AI employees quitting or speaking out? And could that force governments to take AI regulation a little more seriously? Yeah, I hope it does. I mean, unfortunately, I would say Congress is not really known for being bipartisan and getting stuff done with urgency. And I don't mean that as... I don't mean that that's not a revelation. People know that's just how our politics works now. But I do hope this moves the needle a little bit because there does need to be some conversation and governments have to play a role in it. And not just our government, but any government where there's major activity, major AI research happening within the country. They need to be aware of what's going on and aware of the potential risks. I hope it moves the needle. Of course. And finally, what should the public be watching for as these AI companies get bigger, more valuable and more powerful? Well, as consumers, we're using the technology a lot. And that's a really good question. I'm glad you asked this question because one thing that is kind of getting lost in this whole debate about whether or not humanity is going to go extinct, are the real impacts of AI that are happening right now. There's a real disruption happening to education, happening to workplaces. The way we work and the way we learn is being disrupted in ways that it never has before. If you're a parent of a young kid, I assume you're really concerned about how your kid is using AI technology to skirt through school at this point. The kids are not learning to read because they don't have to. They're not learning to write because they don't have to in ways that they used to. And so I think that like, that's what consumers should be focused on. I think that's a big part of what we should be focused on now. Like, how are we solving the problems that are hitting us right now, not five years from now? Definitely. Thank you so much. Lots ahead still around this topic. Thank you so much, Christopher Zara, News Director at Bass Company. We really appreciate your time today. Yes, thank you.