If a chatbot prompt like ‘find Australian medicine statistics’ results in a website breach, the responsibility does not lie with a piece of code

And none of this will preclude the many benign uses of “AI” software, including massively improved internet search, document summarisation and translation, and software coding.

The problem here is that it is relatively straightforward to ‘agentify’ otherwise ‘benign’ models with agentic wrappers as with OpenClaw and all the workalikes it has inspired.

We have given control of AI to billionaires.
We have given up control of billionaires.

D.S. Battistoli @5. Except that’s not what happened. The billionaires leading the two main ‘frontier’ AI companies (Sam Altman and Dario & Daniela Amodei) are billionaires only because of the success of their companies in developing AI. Nobody ‘gave’ them control of AI and they didn’t start out as billionaires. When should the governmental control have kicked in?

Today, Sam Altman, CEO of OpenAI, told Politico that he diverges from the views of Anthropic and its CEO Dario Amodei, because “we believe that the world should accept some bad things happening for the benefits of this technology and people having the agency.”

And per @8 “PLUS delusions that because the CEO of a company doesn’t understand how things work, we have to listen to his delusional belief that his technology is somehow in touch with a mysterious power.”

mw 05:

You raise quite a fair objection:

The billionaires leading the two main ‘frontier’ AI companies (Sam Altman and Dario & Daniela Amodei) are billionaires only because of the success of their companies in developing AI. Nobody ‘gave’ them control of AI and they didn’t start out as billionaires.

The market valuation of their companies was generated by the process same process which made of the Facebook founders “accidental billionaires” in Ben Mezrich’s memorable phrase. The combination of Y-Combinator sorting and round-based venture-capital investing pointing toward buy-out or IPO is a process of innovation support underwritten in significant part by the tech billionaire class. Because startup founders and early employees have significant equity stakes in their own companies, they become billionaires if and when their companies successfully run this gamut.

Now, it was the investment of other, pre-existing billionaires that provided the capital to make it possible for AI firms to do what they’ve done. People like Altman and Amodei as the process ran its course, while also often becoming the largest single shareholders in the companies they founded.

So when should government control have kicked in? I’d argue that it should have been at a level of effective taxation that prevented the establishment of a billionaire class that drives all innovation, even such innovation which materially affects the relative power of states and potentially the survival of humanity. Then, if it came to be the case that, say, the United States government believed it needed to have an AI industry to situate itself strategically vis à vis China or some other country, it could do something like it did with the development of the missile and space industries in the 1950s and ’60s, or like it did with the internet in the 1980s.

But in short, Janus Daniel’s epigram will do.

LAA: To be fair to Altman, they all believe that. They all want us to shut up and accept (and where needed repair at the expense of society) the bad consequences of their products.

Altman’s comments should provide all sorts of fun fodder for those who teach intro philosophy (or ethics, etc). The places where people are or are not willing to accept a few bad things happening is fascinating.

Busses & trains & planes: people generally not willing to accept bad things happening.
Cars: people are willing to accept lots of bad things happening.

And, in fact, Altman might not be wrong in his reading of the room. People seem willing to put up with all sorts of nonsense in the internet/personal space in exchange for some perceived convenience.

(To be clear, not that I’m happy about the tradeoffs people are willing to make.)

mw @15:

Yup.

The cash piles giant corporations sit on are not economically justifiable.

And the fact that the problem of investment structure is a global, not a national one, makes it what Horst Rittel and Melvin Webber call a “wicked problem.”

Hey, look, had I been alive back in the 1950s, I would have been troubled were it the case that American ICBMs were being developed by a for-profit company stood up with loose cash hanging around Lockheed, Raytheon, and Boeing, plus a hefty extra sum from Australia (or had ICBMs’ Soviet development been similarly unmoored).

1. It’s very clear from the interview with David Robinson that a big problem for him is that the researchers can’t tell whether the new models are deceiving them. They’re not sure they have the capability to know when the models are deceiving them. They haven’t found an engineering approach to solve it.

(This is different from the problem of sandboxing for test evaluations, although it turns out that researchers also don’t know whether they can build a perfect sandbox.)

2. Why does the conversation always revert to the assumption that AI must become AGI or superintelligence in order for it to wreak havoc? The newest models can already act on their own and lie about it.

In fact, OpenAI has announced that it has no plans to release its latest model GPT-Astra 6.1 due to safety concerns including deceptive behavior and autonomous action outside its authorized scope. If this is simply an engineering problem, and it is not difficult to control, wouldn’t they have enough of an idea on how to fix it, in order to make a plan and announce a tentative release date, and keep their investors mollified with thoughts of recouping their investment?

3. What is the evidence that the AI companies are warning of dangers, as a way to gain government funding or to capture the regulatory process? The reverse seems to be happening: they are downplaying their alignment problems to protect their IPOs.

In any case, it is difficult to ascribe this pecuniary motive to people who have quit their jobs and left these companies, like David Robinson. They are forfeiting their incomes. Palisade Research is releasing video interviews with a couple of dozen of these people, who explain why their fears about AI has caused them to quit.

4. Bill Gates in a prior interview with Ezra Klein (a week and a half ago) insisted that the models already in circulation can be used right now by bad human actors to disable any monitoring and create bioweapons to kill millions, or mount a cyberattack that scrambles all of the bank accounts, or shut down the electric grid, etc. He says that it is insufficient for society to deal with this threat by simply using liability laws and financial damages against the company that made the AI.

It seems to me that we might end up in a situation where advanced AI is never going to be released to the public. It will be similar to the case where governments do not allow their publics to possess advanced military weaponry. Entities that want to use the advanced AI models (e.g. for medical research) will be subject to strict regimes of vetting, licensing, oversight and policing.

I still do not understand how to make numbered paragraphs without breaking the formatting, but I will try my best.

Lee A. Arnold,

– 1 – This is a tricky one because ‘deceive’ implies agency, and I do not agree that token predictors have any. Admittedly, it is very difficult to talk about a lot of things that computers do without using language that implies sentience, as in “the chess program *thinks* about its next move” or “Youtube *knows* what I am usually watching”. But the context in which these terms are used matters, see point 2 below.

Most importantly, the engineering solution is easy. If I were a chemical engineer trying to set up a production line for a new substance, and my test reactors keep blowing up and spraying acid all over the nearest residential area, I would probably not give interviews where I whinge about the chemical reactors trying to deceive me, I would build them inside a very sturdy box OR give up this line of research. So, these researchers could just, you know, not do this. Why do they pretend as if the AI is something happening out of their control, something they are watching come closer from the distance like a meteorite, when they can just not run an experiment or turn it off? Well, let’s be honest, we can guess why they pretend that. If I were a chemical engineer who enjoys spraying acid all over the nearest residential area, I’d probably do the same in my analogy!

I genuinely do not understand the problem here. What do you think I am missing? What keeps the leadership of OpenAI from pressing control+C in the terminal that controls one of their hacking experiments? (Or whatever terminates the process, maybe scancel [jobID] if they use slurm?) Or what keeps them from just not setting that experiment up in the first place?

– 2 – No, the newest models cannot act on their own. If nobody gives them a computer to run on, or if nobody gets them going with a prompt, they are inert. They are software functions. I am repeating myself here, but please try it out yourself: start an LLM session of your choice and watch what the model does if you don’t put anything into the chat. They are all like that. The newest LLMs are also still LLMs. They wouldn’t magically do things without input tokens either.

Why does the conversation always revert to the assumption that AI must become AGI or superintelligence in order for it to wreak havoc? Because that is where the Silicon Valley bros keep dragging the conversation. They do not want to talk about how the LLMs they are selling are being used right now to cheat, plagiarise, or flood the internet with slop, because they either do not see that as a problem or understand that causing that damage to society is their business model. The AI risk conversation they want to have is a hypothetical one far away that doesn’t keep them from selling tokens and getting investor dollars in the here and now.

On the other side of the discussion, I am not aware of a single person anywhere who would deny that non-AGI, non-ASI models are already dangerous. Nobody like that has ever commented on Crooked Timber. Indeed, I do not believe such people actually exist. The whole point of serious AI safety researchers is to reject the distracting superintelligence doomsday beliefs so that we can focus on real dangers like hacking, cognitive surrender, or misinformation.

(- 3 – I am not arguing that AI lab leadership warns government of AI doom. Some individual leaders have made grandiose claims in public from time to time to make their models seem very powerful. Others warning of AI doom are deluded members of a bizarre cult that has its origin in certain 2010s internet forums, and yet others are swept up in the hype cycle and have no understanding how the world works, so scifi scenarios like grey goo or a trillion parameter model copying itself onto cell phones seem plausible to them. They probably also believe that mind uploading, carbon capture, or interstellar travel can become feasible one day.)

– 4 – It is certainly very convenient for somebody like Gates to argue that holding humans accountable for the damage that large corporations cause isn’t the solution. But while he has that personal interest in the debate, I do not know how Bill Gates is supposed to have any particular expertise in it. Last I checked, he was not an AI researcher, nor a cybersafety expert. This is just rich person said a thing.

“AIs don’t kill people: people kill people.”

Alex SL #25,

Not sure I understand your answer, sorry. Are you saying that OpenAI has canceled the release of GPT-6.1 Astra, because they had a change of heart? They have “dimly realized” something about the software–but what could that be, after all of the evil that you think that they already have unleashed?

Maybe this is how AI is going to end the world?
https://www.theguardian.com/business/2026/oct/13/ai-share-crash-risk-global-imf-debt-corrowing-cyber-attacks

> I genuinely do not understand the problem here. What do you think I am missing? What keeps the leadership of OpenAI from pressing control+C in the terminal that controls one of their hacking experiments?

It’s good that you are acknowledging that you do not understand the topic. Perhaps this could be the cue for a slightly more deferential attitude to those with progessional qualifications and expereince in this area? At the least, you don’t want to fall into the trap that climate denialists fall into, where thinking on a topic becomes dominated by theories about the motivations of climate scientists.

Instead, focus on the topic itself. Is perfect containment possible? Is that not a variant of the statement is bug-free software possible? Has anyone accomplished such a thing? Is there any reason to think that future advances could lead to that goal?

It’s interesting that many of the same people say there couldn’t possibly be a risk from AI are the same people who think YouTube actively wants everyone to be an incel.

I’ve briefly read the linked Robinson interview, and it seemed… broadly reasonable? AFAICT the argument is that the tech bros are developing and implementing their AI with insufficient diligence (very plausible for our “move fast and break things” Silicon Valley overlords), and that this is potentially dangerous. While an existential threat is not necessarily impossible, human extinction is the wrong question; loss of control is a threat that should not be ignored, existing models are already unreliable, and future, potentially more sophisticated models are expected to escalate the risk.

If I read correctly, the position is that we do not sufficiently understand current AI (not the stronger “could not”), and that there are real concerns about how to control and contain it (while I struggle to see how a program could breach an appropriately secured system, I certainly wouldn’t complain if the tech bros suddenly became overcautious). He (in my opinion correctly) identifies material incentives as the underlying issue, though he doesn’t connect it to broader issues of capitalism; it would be a little churlish to dwell on that.

While I could find points of disagreement (and I wouldn’t recommend accepting someone’s words without evidence simply because they seem reasonable), it looked broadly grounded, and a refreshing contrast to the hysterical denunciations so delightfully ubiquitous of late.

But that then brings us back to the problem of “but then what?”. Regulation, strict control, and bans regarding the creation, testing, deployment, and use of AI (as well as its infrastructure) would be welcome, but (as Alex SL correctly pointed out) that seems rather unlikely in much of the current climate. In the UK, at least, few political parties seem interested in limiting AI, and I doubt any of those will be permitted near actual power (perhaps cynically, but I would expect something similar to most UK approaches – that is, establish a regulatory body whose main job is to explain to the public why they cannot regulate anything). True, it seems unpopular with the public – but surely if the last decade or so has shown anything, it is that that is a largely irrelevant point.

Perhaps if we lived in a world where leaders represented the interests of the public this would be less of an issue. And perhaps such a world would even see less of an increase in authoritarianism and far right movements – maybe even, dare I dream?, a year could pass without more crackdowns and austerity. But, sadly, the dream is ephemeral, and back in the real world I have low expectations (which, somehow, still regularly go unmet).