"Do You Think AI Is Gonna Kill Us?" | This Week In AI

Abhi: So funny enough, I went on a date last week and when we talked about my job, first question she asked me was like, Do you think AI is gonna kill us?

Shane: The models that they are seeing must be so much better than the models we have if they're worried about it, because I can't get my model to do a simple task sometimes. Welcome to Agents Hour. It is September 14th. We are on site in San Diego and we are going the to do the news like we do every week. And it was an eventful week. There was kind of one big topic. Let's not beat around the bush. Let's jump right in. Pacing the frontier. Should we pace the frontier? What the heck does that mean? This is the topic. So it all started with this post from someone unnamed person, no one really knew who this person was. They didn't have it hardly any followers. They had never posted on X before. Just to be clear this post got 171 million views that is insane for someone who's never posted on X. It's honestly like algorithm breaking in a way. So we can talk about why that is, but it says. I resigned from Anthropic today, I spent the last three years doing pre training research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self improving superintelligence and gambling with our lives. More thoughts below. And then he goes on to talk about some of the reasons. He doesn't get into a lot of details, but basically talks about how people are not taking this seriously enough. Even people inside Anthropic think that this is going to happen or it's possible and no one's doing anything about it. So that was that's how it all started. And then Evan. Hubinger. I'm trying to know exact title. Alignment science lead is Evan's title. Evan says, Jacob is correct here. We really do earnestly believe AI could kill all humans. Exclamation point. Kinda like he's excited about it. I personally think it is greater than ten percent within the next decade. I believe Anthropic is trying its best. But we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. So this was the thing. I think it w. The original post was one thing, but this was the thing that got people extremely kind of bent out of shape for good reason. Because why? Like where did you come up with 10%? That's kind of important. Like what is the percentage chance that you think that we actually have we're all gonna die in humanity?

Abhi: The way they die, but It's like extinction level ramifications. So like Jacob went on CNN, talked to Anderson. Cooper. One thing I'll just say I just noticed. Very much from him is it's all high level, no details. The threat to our existence isn't here today, but if we continue down south. Recursive self improvement, then AIs will have ways to improve models themselves and then destroy the whole world. Which, you know, if you go on CNN, like I feel that's just like such a doomer thing because the people watching CNN are not in this industry deeply. And I guess it you know, it makes sense to keep things general, but it's a lot of FUD. You know, at least that's my opinion. I do not think a twenty-seven year old researcher who pro who is like two years old during nine eleven should have any like any you know, point of view here. But maybe he knows something that we don't, you know?

Shane: Yeah. We shouldn't and we'll get into this a little bit. We are not detectives. We did our own research. You should do yours. But there's some kind of interesting sketchy things about how this became so super viral and how many people jumped on it so quickly. For this to even happen, a a lot of things had to go perfectly right with the algorithm. And so it It was a little like skeptical. I think in the first fifteen minutes there was like three major news article reposts. There was a Wall Street Journal article that actually came out a few minutes before the post itself. So this was all like coordinated in some way. Like but obviously the other thing is it clearly resonated with folks too. And I think one of the reasons is just like human psychology. Everyone. Like fear spreads faster than anything. So if you can spread fear and panic, people will spread that faster than any good news. Like bad news spreads faster than good news, right? And so I think there was probably this bent up. Like everyone's on edge, everyone hates data centers. You know, if you like talk about the just general perception, especially across like the US. So I think w this is like there there's a lot of pent-up demand for people want looking for reasons to hate AI. And this was like fuel for the fire, so to speak. And the other thing about this. Jacob Coxon is he did work at OpenAI for I think three years. But Anthropic, I think he was only there for like six between six weeks and three months. Like it wasn't that he wasn't there that long. So it is interesting that he so quickly left and he is treated like he was that Anthropic for a long time, but he actually like couldn't have had much impact in r. I would question how much you can even understand about a company in just six weeks. And maybe it was a little longer. I think it's still to be determined. So then this person, Parker, comes out and this also. I'm only sharing posts that got a lot of yeah, a lot of views because there there was so many posts around this topic. And it says this post looks like a start of a very sophisticated and well-funded PR operation. So this person with minimal followers and no previous activity goes to the Wall Street Journal. Publishes an exclusive within hours it has tens of thousands of reposts and then according to grok some of the repost accounts were like a lot of head of AI policy head of AI futures projects. A lot of people that are pretty big into spreading AI. Doom narratives across, you know, the industry. And then a lot of the if you track down where the funding comes from, it's a lot of like people that are funding these. AI. Doom. NGO type organizations and this person doesn't really have that much of a resume. So there's just a lot of if you r if you kinda dig into it It's interesting. It makes you at least think that maybe this wasn't completely organic. There was maybe someone like putting their finger on the scale, so to speak. Yeah, Bernie Sanders. Had an AI bill wanting to stop all AI. No AI progress at all. And one of the things in this is you get 20 years in prison if you try to advance AI at all. That's interesting part of the thing. And of course, if you do want all AI to stop, I guess maybe this is reasonable, but also seems a little wild of a claim. My thought is they put out this really like crazy stop everything and then maybe there's a middle ground and I think maybe that's what they're going through. Like here's the extreme scenario. Now we get some real regulation that finds some kind of middle ground. I think that's the idea. It's like you try to tip the scale by going to the extreme. And then this was the big response. So Dario from Anthropic puts out an essay that says we must pace the frontier. I've written a new essay on why the AI industry should slow down with a three part plan for doing so. Anthropic is unilaterally committing to the first of these steps. You wanna talk, Abhi, a little bit about what were those three steps and you know, like what is Anthropic committing to at least start.

Abhi: Well before that I'd like to just talk about like the history of Dario 's views here. To go back in time here, so Dario was at OpenAI. You know, one of the first interviews I was researching was him on Lex Fridman's podcast. And that podcast was really cool. Like it was very much optim a lot of optimism and a lot of technical deep dives into AI in general. And there was mention of having governance then. And then from founding Anthropic in like 2021, you know, Anthropic is a PBC, a public benefit corporation. It still is. You know, Anthropic 's mission is to create AI w in a safe environment. And then you know. Fast forward to 2022, you know, most researchers were doing RLHF and you know Anthropic was one of the first research teams to be like we should take humans out of this loop and have the models critique themselves. But you need to have that in a very ethical way. So they were like making sure that you know this research is happening both at a fast pace but in a safe way. In twenty three he had his first judicial committee hearing where? He did start talking about the misuses of AI. So there are biological threats, cybersecurity threats, and he was advocating for External evaluations. And he maintains that today. We're gonna get into the things that he wants. But even back then, so it's not like some new thing. So 2023, he set all the problems that could exist and he his his first thing is hey we should have external evaluation on all frontier systems prior to it deploying in a mass quantity. Later in twenty twenty three, this is where things kinda differ, right? Like he starts. Thinking about how. AI can get poisoned by commercial incentive. And you know, then they should have this rubric of AI safety levels, ASL, similar to a biosafety level, BSL 1 to a BSL four. That same type of thing there. Now we're in 2024. That ASL framework has evolved and it's something that is part of Anthropic’s culture. Safety of AI. No dooming yet. Everything up until this point I do respect a lot because it's saying there are potential issues. Here are some guardrails and things we can do. But then, you know, this last year, this year, we have obviously gone in exponential rates of growth. And I think that it's moving too fast for the ASL to contain. He went on CBS to talk about what's let's say the first Doomer take where he went on CBS Sunday Morning, talked about that, you know, the growth rate is so fast that you know the safety protocols. Are not catching up in time and the raw scaling will just outpace the actual safety that can come with it. And that leads us to today, you know, we must. Pace the frontier, the latest thing. So a couple things that were established. One, there needs to be global coordination. And that's a big contrarian view for a lot of Americans, which is like US needs to own the AI boom, but Dario's advocating for global coordination so across many governments should own both the governance and the progression of this. And so if you're a democratic nation. Maybe all major labs need to have safety baselines. Second thing is this embedded evaluator concept where there should be third-party audits and government government regulation on any AI deployed. And then lastly, if there should be like, you know, there's not only compute constraints, there's these scaling constraints. People hate data centers. All this stuff. And that is an advocation of pausing development or slowing down. Now, when I go through the timeline like that, I'm not hating on Dario. Like it is a natural progression of how this thing evolved. I think a lot of people saying that he's a doomer and he sucks and stuff is just because you're s hearing what he's saying today, but if you do the research on his how his opinions have changed? It makes sense. So like I'm not against him or anything. But the problem is, I think in America, we're very just worried about the Chinese labs. And so any pausing is like telling builders to stop moving and making moves, which I think is what everyone's really tripping about.

Shane: I think if you were just to look at what Dario has said in your Completely new to AI, you have not been paying attention other than you know it exists and maybe you've heard about it, seen some ads. This is gonna really scare the shit out of you. But if you combine this with the previous topics, right? And of course Jacob's going on all these shows, Dario's now going on CNN. I mean everyone is kind of sounding the alarm. And so yeah, I think pe a lot of people are gonna be scared. I think Dario's actually coming from a good place. I think he generally does care about safety. I think the challenge that I have with Dario is I think he thinks that, you know, it's almost like main character syndrome in a way where it's like he's the one or they're the ones that can do it and it should be in the hands of It's like the power should be in the hands of a few, which is always a scary thing if you think about just governments and control in general. It's just if you want to control it, well then How do we know that you are not corruptible? Typically, spreading control is a better way to prevent that kind of thing. It's not perfect. These none of these systems are perfect, but centralizing all the power behind one like government, especially governments, multiple governments, centralized power, does seem challenging to pull off and potentially scarier than the alternative where everyone has superintelligence, right? So I think that's the challenge that I see is I think Dario’s coming from the right place, but I think s he thinks that he's the guy and only he can do it.

Abhi: He does have the receipts though. Like We're gonna get into the other people who agree with him, but at least he has the receipts from the beginning and we'll see what the other other dudes are saying.

Shane: Elon says. Dario was right. And the funny thing is before this, some people. That are really pro-Elon flipped sides at this point. They were so anti-whatever Dario was saying. Then Elon said it and they're like, well, maybe he has a point. It's so funny. You see the people that are like, you know, flipped their position when Elon chimed in. But Elon also has a history of, you know, this is why he funded Open. AI at the beginning, right? He was worried that you know dominance by I think Google was a risk. If Google controlled all the AI, that's a risk that having one company control everything. And so opa. OpenAI was supposed to be a nonprofit, supposed to be open. Elon has some of the receipts as well, at least as far as like how he thought about it. I think he's always been worried that AI could potentially be negative, right? And we've all seen the movies, right? Everyone knows what it could potentially do. Clement from Hugging Face said, it's now clear that alignment is critical and won't be solved behind the closed door of a handful of Frontier Labs. So today we're launching the open alignment initiative led by Tom Wolf hugging face and asking to be part of the embedded evaluators program that Dario just committed to. So let's make AI safer by making it more transparent. So more people with credibility coming in. Sam Altman even came in and said, I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We will have more to share soon. So all you can kind of have the big three, I would say. The big three are all in agreement that this needs to happen. And then this came out. I thought this was kind of funny. You can't spell Dario. Amodei without AI Doomer, which is just funny that works out that way if you rearrange the letters. It's kind of funny. Even if th this is a very serious topic, it's it's a little humorous. We should we can all we should be able to laugh. Even in times of grave danger, potentially. We should be able to laugh. And then so David Sacks came out with a post and said, Dario has written that we need to pace the frontier, and Sam has agreed. People may be surprised by my response. Go ahead. You guys are the frontier. By any reasonable metric, market share, revenue growth, model capability, the two of you have a duopoly on frontier intelligence. You've also claimed the lead is widening because of recursive self-improvement. I don't see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. On top of this, he also says, most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want. It is also just giving customers what they want. There's a bit more talks about Bernie Sanders' take, and so basically says you should go ahead and do it. And if you do, you'll buy goodwill for the next conversation. If you don't, we'll know this was just another bid for regulatory capture or an election season psyop. So that of course got a ton of views as well. A lot of people resonated with that. Meanwhile, this came out breaking. China's Z.ai has raised $5 billion. The interesting part is where the money is going. They're saying 60% of the net proceeds will fund next generation GLM models in what Z.ai calls. It's fully self-training system. Sounds like precursor of self-improvement to me. So the question is, if China's already investing to do it and they're investing a lot, can we actually pace the frontier? Maybe, but I think that's why a lot of people are worried because you can't actually control what everyone does. We can only control what we do. Gavin Baker came out and said, while 24 hours and a lot of different proposals have been made, the only tangible new fact is OpenAI and Anthropic are going to have embedded third-party evaluators. We talked about that. Dario floating METR as a possibility for one of those third party evalu evaluators. Yann LeCun jumps in and says, pessimists arc is right. Dario was already claiming that GPT-2 was too dangerous to open source back in 2019. I made fun of them then. Everyone should make fun of them now. Somewhat related to pacing the frontier, you know, Thibault from OpenAI said that they're pausing. Sign ups to their 200 pro plan. Mostly I think this is compute related, but it the question is like are they just trying to improve or make sure that there's enough compute to go around? Are they running into scaling problems? Is this one of the reasons they want to like slow down the frontier a bit? Is there is it connected or not? I guess that's you know a bigger question. I'm not sure. What do you think?

Abhi: There is definitely some strain on capacity. But also like I mean this came out before all the Doomer stuff or at the same time. So it's kinda it's kinda weird. Like I guess we should have all bought a couple more subscriptions because now we can't. But there's definitely something going on with capacity. Fable had the same issue, right? There was so much demand that they had to knock it off the subscription. This is unfortunate. I think there's a lot of backlash in the developer community. But what do you expect? You know, we've been subsidized for so long.

Shane: Mark Zuckerberg chimes in, says. AI development needs to move faster. He decided he's taking the opposite. Take of the other the big three, so to speak. If there's a Mount Rushmore of AI, like frontier model lab figures, you know, Mark Zuckerberg is like not quite on the monument, but he's trying to get on it, I guess. And so he just says, No, we need to keep moving faster. So I thought that was pretty funny. How do we want to summarize this? What is your overall takes in should we be pacing the frontier? You know, for people like us, we're builders, we're out here using the things, does it affect us? Well how should we respond when my mom or my grandma says. I saw this CNN article is AI gonna kill us all? Like what do you think? I'll t I'll give my take. I will caveat this by saying, we're not in the labs, right? We don't know what they've seen. We only know what we see in the outside world.

Abhi: So funny enough, I went on a date last week and when we talked about my job, first question she asked me was like Do you think the AI do you think AI is gonna kill us? And she's not in tech at all. So like that is the general sentiment because the news comes from They use, you know, the CNNs and everything. So from that perspective, it it's kind of like we're not at the point of this extinction level thing and a lot of people are worried about it. So I was like just feel bad because a lot of people get anxiety from these news articles and things and it's just not an it's not an anxious point today. For builders though, you know, if the models stopped be getting better. Our life doesn't really change that much 'cause we are building with the models at every part of its like growth. So if it stopped today, you know, we may not get anything better, but we're already building harnesses and the tooling around it to try to make it good enough. So I don't really think it affects us as much, you know?

Shane: Yeah, I do agree with that. If everyone's decided, hey, we're gonna define what the frontier is and we're never gonna pass it, we basically haven't figured out all the things we can even do with it today. There's a lot that could still be done. Now the challenge is I don't think it's actually gonna happen. I don't think people are gonna really slow down. I would say. You know, we've seen the hugging face incident, and if you really dig in, there's some interesting things there, but also I kind of exp would have expected that outcome given what the models have been trained on, what they would do. I think if we saw like a dozen hugging face level incidents, I would start to get more worried. I think only seeing, you know, a few of these that have come out, I do think that the gap from going from there to like full scale AI takeover controlling everything, all these external systems. And of course there's no like perfect off switch to this, but it's all compute and data centers, right? I don't think that we're at this point where this whole thing is going to be taken over in an uncontrollable way in the next couple years. Now maybe in the next decade could it happen? I'm not an expert. The models that they are seeing must be so much better than the models we have if they're worried about it because I can't give my model to do a simple task sometimes. You know, you're just like missing obvious things, but yet. I think this model's gonna full on take over the world and kill all humanity. That seems skeptical to me. But again, maybe we don't see all the models that they have cooking in the labs.

Abhi: So I guess our tip to everyone is invest in a bunker and then you'll be fine either way.

Shane: Yeah, I will say I am not investing in a bunker. I'm not overly concerned yet. I think we should be careful. Like this stuff is dangerous. We should have some safety. Guardrails. I think I am worried about too much over regulation in that like one controlling body controls all progress and everything has to go through that because I think if we slow down too much, you become like some of these other overly regulated countries who can't really be competitive. And then I think if you're in the US, you'd be concerned that, well, is that actually gonna slow down.

Abhi: I'm selling pre-sale tickets twenty thirty six to Abhi’s bunker. You can Venmo me, Cash App, you know. But if you're really worried about it I got you. Dude, did you subscribe?

Shane: Dude, I host the show. Did you subscribe?

Abhi: Did you subscribe?

Shane: Subscribe to Agent. Sour every Monday, noon Pacific. Let's get off the doom topic, even though this next one's kind of related. Safety in practice. Anthropic published a detailed threat intelligence report, covers how people misuse Claude for cyberattacks, influence operations, surveillance, biology, and building weapons, and how they found an stopped them. Zihad, I think is how you pronounce that, had a pretty good take on at least a summary of all the things that were in the report. If you don't want to read the full report. Here's some TLDR from it. There's a suspected Russian state linked group using Claude across phishing intrusion, data theft, and mad malware development. Including rebuilding malware after security products detected it. There's ShinyHunters linked hackers used AI agents to scan 1.8 million Android apps for exposed secrets and help run breaches across multiple companies. A China-based group built an automated exploit setup that could research vulnerabilities, build offensive tools, and keep working against target with little to no supervision. One consultant used it to build Lakana 360 for Mali 's intelligence service, a system designed to Monitor roughly twenty-five million SIM cards across the country, including calls, messages, voice interception, watch lists, and automated intelligence dossiers. A whole bunch of things. There's a Yemen group, another. Russia attack. I think they talk about distillation from Kimi, from Deep. Seek, Z.ai. They talk about all the different how Moonshot was basically distilled from a lot of these like Chinese models were distilled from Frontier. And that is one of their one of Dario's takes and one of Anthropic's takes is if we're moving the intelligence forward and then we're getting distilled by these. Chinese companies were actually helping them move faster. So which I can agree with. Like that 's the one.

Abhi: I think the wildest thing was Anthropic claims that Moonshot DeepSeek were routing requests to Claude. Instead of their own model for some users without telling them. Which I think people have said that's un that's not true or there is true, whatever. But that is quite interesting. And then if you see all these like weap you know, weapons six weapons programs are using Anthropic, et cetera, like all these AI personas on dating apps, people trying to do these griffs with old people. Yeah, dude, safety is important, you know? And I think Anthropic’s looking at this list and saying, we don't have an ASL to protect us from this right now. So we probably should slow down because we have no guardrails for this.

Shane: OpenAI on September 10th said GPT-Live-1 is now available in the API. Bring ChatGPT's natural back and forth to your app with voice agents that listen while they speak and work with the models and harness you choose. So now you can get real-time voice right directly in your app. And I think this is something that a lot of people have wanted for a while because they Interacted with Chat. GPT and the voice was pretty good comparably to a lot of things you could build yourself. And so I think a lot of developers will like that they can maybe just use their voice mode. Directly, Cursor introduced Projects. This is from September 10th. It's a new way of working in Cursor. Rather than creating a chat for every task, you work with a coordinator agent in a single persistent thread. Like Bot, your agent is always on, proactively manages work with subagents and improves over time. On September 10th, we also had DeepSeek-V4.1-Flash. Smarter, faster, more efficient. So it's the smallest model in their new architecture family. And it's good. Yeah, it's pretty good if you look.

Abhi: If you don't get rate limited, it's great.

Shane: Yeah, for and for how small it is too, right? It's like impressive. It's impressive that you can get such good levels of intelligence on these smaller models.

Abhi: But you don't know if it's hitting Claude anymore.

Shane: Yeah, that's true. Maybe it's just c. You know, it's it's like the Scooby-Doo thing, you pull it off, you pull off the hood, and it's like, oh, it's just actually Claude. Maybe that's why. Cognition released Devin. Voice. Your favorite AI software engineer just got a landline, so you say it, Devin. Shipson. So it's powered by GPT-Live and our new SWE-2 model. So there you go. This is tangibly AI related. If you remember back, Tailwind had a lot of they had some layoffs because no one was going to the Tailwind docs anymore because AI was just reading the docs for you, right? This was probably 18 months ago or so, a long time ago at this point. And so it's kind of full circle. I feels like Tailwind is kind of winding down, so to speak, and becoming part of Shopify because a lot of people use Tailwind and don't even know it these days. Which is, you know, unfortunate for the Tailwind team because they built something that everyone uses but no one pays for it.

Abhi: At least they have a home now to keep continue working on it.

Shane: Yeah, so congrats to them and I think that it is good because Again, so many people are building web apps these these days. So many agents are building web apps these days, and a lot of them are using Tailwind under the hood. All right, this last one. What is this, Abhi? You wanna tell us about it?

Abhi: The death of agent plugins maybe? In the latest MCP spec. We talked about MCP's. V2. This is another feature which is super cool because you can define extensions that load agent skills alongside the MCP server. So if you already have an MCP server. You can then bundle your skills within the server and then you can grab skill metadata. You can then use it as like you know how skills work is agents have to load skills into context to then use them. Now you can do that through MCP. So why ship an agent plugin when you're already shipping MCP servers? Which also makes me think why Anthropic was not involved with agent plugins. And I would agree that this is a way better delivery mechanism, in my opinion.

Shane: I mean we will see now we have like two standards trying to do some of the same things. We'll see who who ends up.

Abhi: MCP's having revitalization right now. And this is a perfect like add some more fuel to that fire.

Shane: A lot of people kind of lost sight of MCP, even though enterprises didn't. It's still very well, very widely used across the enterprises that I that I talk to. But I think people it you know, it was very trendy and then it kind of diminished and people. You know, we're saying it's not actually that good of a spec. But you know, with some of the new stuff they've released and the two two point zero spec plus some of this, it's having a bit of a rev revitalization. I think, you know, MCP is back, at least as far as like hype terms as we kind of alluded to earlier. There's all these hype terms and I think MCP is it's still it's still got a little hype behind it. But that is it. That is the show today. Thank you for tuning in to Agents Hour. We do this every week. We talk about the news. We give our takes. Sometimes we're right, sometimes we're wrong. You be the judge. We'll let history decide. But you follow us @mastra if you want to see what Mastra’s talking about, all the things we're shipping, we ship a lot. Check us out on YouTube so you don't miss any episodes. Please give us that subscribe, that like. That helps other people find the show. We appreciate that. I'm Shane Thomas, sm. Thomas3 on X. That's Abi. Iyer @abhiaiyer on X as well. And With that, I guess we will see you next week on Agents Hour. Peace. See ya.