Concerns are rising inside AI labs that competition is pushing tech companies to race toward self-improving models that risk spiraling out of human control
An Anthropic researcher is quitting the artificial-intelligence industry over fears that the lab and its competitors are racing to build systems they won’t be able to control, a sign of mounting safety concerns within top AI companies.
“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon said, adding that safety trade-offs are inevitable when companies are competing against one another and Chinese upstarts.
After Coxon announced his departure, a top scientist at the company, Evan Hubinger, wrote on X that he thought Coxon was correct about the risks of AI development.
“We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” Hubinger wrote. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.” Others also amplified the view.
What do stackers think? Are we really at >10% risk of ai destroying all humans?
Or are these people just part of a deluded echo chamber of like minded fools?
They might be smart engineers, but history suggests that intelligence and wisdom are two different things.
I'm with you on the echo chamber for 80%. The other 20% says "their lack of faith in humanity is disturbing"
Alarmists are usually wrong because they underestimate humanity's ability to adapt
Yes. So for the 10% to be true, all that needs to happen is a US-China war. Then Claude can assign US' targets and Kimi CN's targets. So it's not that far away, if humans are going to do that war. I don't know how much more or less loss of life there would be without AI though. Humans fuck up too.
But yes, localized, AI is already "responsible" (in quotes because you cannot hold AI responsible) for deaths. In every hot war the past 5 years, there was AI involved. So, AI is a killing machine. Except it's the humans that do it. So again, not sure if it would have been less without AI. People kill themselves over AI slop. And others too.
I think bottom line we should ask ourselves what we could but would prefer to not use AI for. For me, in my direct day-to-day those are 2 very contrarian things: writing code, and discovering content. Which is the opposite of what the rest of the world is doing. But I wonder if people are thinking it through. And are really doing assessments whether it is better. Or just fomoing into some hype. I fear the latter, and with that, I fear that in the end humans will make the right choice, but not before fucking up royally.
I'm surprised that you prefer not to use AI for writing code and discovering content. I find that those are two of the most direct and least controversial use-cases. Versus artistic creation, for example.
On your assessment point, I totally agree. Assessment of AI output is hugely trailing AI's ability to produce it.
I just see a lot of code, and in the past 4-5 months, I see an overwhelming majority of slop code. And I see the difference. Because the code doesn't carry accountability anymore, because the thing that wrote it isn't sentient and has nothing at stake, and the person it wrote it for can just blame anything on the AI, because even labs get away with that. ZERO accountability. Code with no soul may as well just have been a prompt that got shared. At least the prompt had a bit of soul.
As for content discovery (and then I don't mean research, but the SN kind), what I have really come to appreciate, and moreso since about a year, is the human interest that triggers a post. You see, I'm not interest in any content that exists, I'm interested in content that triggered you, or other stackers. And if I run into something that triggers me, I'll share it too.
I see the lack of accountability and slop avalanche as a product of our evaluation bottleneck. Truly think that's the problem that needs to be solved: our ability to actually process the flood of unlimited AI generation we're facing.
I see what you mean about content discovery.
I agree with that. I think that "AI will take your job" made a lot of people overconfident. Also "I don't read, I ship" in January was a terrible example. But I can see a lot of devs liking that because now you don't have to review deeply, or think. Which is the only thing you should be doing. And it's the least fun part!
To some degree, we can fight fire with fire. But you cannot let an LLM be a judge. You cannot ask it: "is this ok?" What you can ask is: "find all the problems." And then you have to read it to make sure that those really are problems. Currently, hit rate is... 90% or so, so I don't use it for critical things because a 10% error rate compounds rapidly, also when these are false positives.
I think I even see the difference in the bugs:
This is clearly an LLM bug, right? I don't think a human would write code that could cause a bug like this. It's a funny one, but I don't think it was intentional. It makes me wonder about the rest of the code, though.
/cc @k00b
Nah, here's an LLM bug: https://github.com/postcss/postcss/pull/2119/changes/22cb61b7e893a4ffe762e62d830808b6bddd956e - see it?
I spotted it in the diff when I was doing dep upgrades:
#3189but I misclassified the impact and I didn't see the problem during testing because I didn't test multi-paragraph posts: So after merge, @SimpleStacker found it in the live env: #1556331So that's on me being not precise enough in testing and misinterpreting the sass integration, double-whammy opti-fuckup.
In general, SN 1st party code is carefully engineered, more than what I can dry-eyed say about what I'm seeing in the npm ecosystem in general lately, including some of the SN deps. I spend a lot of time validating if there is impact to new bugs introduced in bugfixes. Luckily, most of the time, like for example some of the shit npmsec puked out last night that then contains new bugs in a dotdot release, is pre-empted by measures taken in the code / config to disable some nextjs features that no one should ever use to begin with, and thus safe.
But, sometimes it's pretty complex, and fuckups, like mine, can happen.
It's intentional. I thought it was cute.
https://github.com/stackernews/stacker.news/blob/d4aaf0ac28e3a1cd2b2b6ed0fbd3647c0e1a5422/wallets/client/components/balance/text.js#L26
lol
It's def cute!
deleted by author
I doubt it's that integrated into the economy yet.
Could there be an enormous catastrophe from fucking up critical logistics or infrastructure, maybe starting a nuclear war? Sure, but I don't see it being an extinction-level situation.
I often wonder if there's a connection between deep intelligence in a particular field, and alarmism arising about it. Like climate scientists who become increasingly shrill about doomsday climate scenarios, or AI researchers becoming increasingly shrill about doomsday AI scenarios. Or, more adjacent to bitcoiners, monetary doomers who get increasingly shrill about the imminent collapse of our financial system.
There's probably something related to the ego and the need to inflate the importance of ones' own expertise
Could be. My experience is that hysterias arise from people adjacent to the fields of deep knowledge.
Most climate scientists aren't hysterics and actually have fairly measured views, but they also don't push back on the hysteria (one can guess at the reasons). As @denlillaapan loves to point out, bitcoiners generally aren't monetary experts and monetary experts generally aren't hysterics about the system collapsing.
That makes sense. And the guy who resigned seems like a relatively junior AI researcher, so it tracks.
man is that an eloquent transformation of hundreds of my ranty SN posts. Undisc, ever the poetic synthesizer
eeeeh, that's demoralizing but would explain a lot. Reality check, calm-the-fuck-down sort of. v neat.
My dad's explanation for the psychological attractiveness of doomerism is exactly that desire to feel important.
hopelessness, too. A sense of meaning and ego. If you, being infinitely wise and credentialed and educated, devote your life to fixing this big problem, then by definition it must be an important/worsening problem, right?
If I think the thing is more important than it objectively is, makes it easier for me to keep going/do a good job
If AI is put in charge of simple things that affect public health, like clean water supply, then it becomes very plausible. Poisoned water has long been a major risk factor in emergency planning dating back decades. It just depends on what AI is saddled with.
Even that sort of thing isn't an extinction threat.
Drinking water is managed in lots of different ways, so it would have to fatally poison municipal water supplies, every bottled water distribution system, and all home wells, in addition to overcoming all commercial filtration systems.
That's so unlikely to happen by accident that it would have to be an intentional initiative of the AI. The odds of the AI even attempting to poison every single water supply are below 10%, before getting into the odds of being able to pull it off.
...and then there is Flint, MI, proving all those assumptions wrong. And that was managed by people without an AI involved.
How does Flint prove anything I said wrong?
Did I miss every single person there dying or is almost everyone still alive?
Oh I see, you actually have to have a body count to acknowledge the matter in the discussion.
"... an estimated 140 000 individuals were exposed to lead and other contaminants in drinking water."
It clearly was a comparison that if humans can muck up the drinking water supply without AI, it opens up possibilities of much more with AI involved. But that's okay, keep looking for bodies while everyone is poisoned. It's a bit of an ostrich head in the sand position but it works well with denial of risk.
https://pmc.ncbi.nlm.nih.gov/articles/PMC6309965/
Again, you're drawing strange conclusions and projecting very odd motivations onto me.
If there were more than 140,000 people in the Flint area (there were), then what you've shown is that even poisoning the municipal water supply will leave many unexposed. Since that was exactly my point, I'm not sure what it is you think you're demonstrating here?
The biggest concern is that we're moving so fast with AI that we may not understand the risks until it's too late.
You got an archive link for the article?
deleted by author
deleted by author