The story of last week, which will certainly return, is the threat to the survival of humanity (and maybe all life on Earth) from artificial intelligence. I don’t know how real that threat is and find it hard to think about, but I am worried that even if humanity survives, AI will be a means of endless tyranny, with all hopes of human freedom extinguished.
Much of political philosophy over the past three centuries or so has consisted in elaborating a picture of what it means for a society to be one of free and equal human beings, and the implicit self-understanding of most of our societies has been to have that as our imagined collective destination, perhaps delayed by practical concerns and by deep and unresolved questions about what it would mean, about who is included in the scope of freedom and equality (and who not). When John Rawls wrote about an overlapping consensus or when Joseph Carens discussed shared democratic values or even when Habermas outlined the conditions of an ideal speech situation, they were pointing to possibilities unrealized in our actual societies but implicitly acknowledged as the promise of the future. And radical critique of our societies, such as articulated by Marxists or anarchists, did not so much renounce the aspiration to freedom and equality but argued that the desire for it that capitalism and modernity called forth would inevitably be frustrated within an actual society that reproduced domination and exploitation: that liberal society failed to keep its own promises, which would instead be realised by a socialist alternative.
But now we find ourselves on the edge, or perhaps over the edge, of a descent into a world where such visions of freedom and equality are unimaginable and where, to borrow a cliché, resistance is simply futile. The herald of this predicament was Edward Snowden, who revealed how comprehensive a monitoring of individual thought and action was possible in the world of the internet. Anyway it has turned out that, for most of the population, privacy is something they are perfectly happy to sacrifice, just so long as the algorithms end up being better tuned to the satisfaction of our wants and desires. And progress is no long measured in terms of our liberation from various forms of oppressions but rather in the ever increased abilities we have to acquire and consume, even if that comes at the expense of the natural and physical basis of our very survival.
Benjamin Constant thought that a private sphere was one of the constituent parts of the “liberty of the moderns”, and if he was right then Snowden’s revelations and, subsequently, the panopticon of the tech oligarchs have put an end to freedom already. But it is one thing, perhaps, to have all that information about us and to put it to effective use beyond the satisfaction and shaping of our consumer choice. Well, a first step in politics is to tailor our information environments so as to manipulate us, or enough of us, to cast our votes in favour of the preferred candidates of crypto billionaires, and we are already beyond that stage thanks to the likes of Palantir. It turns out that enough of us can be scared by the threat of “illegal immigrants”, “fighting-age males”, “grooming gangs” or simply brown people standing up for their rights, to make it likely that far-right populists will take power in parts of Europe as they already have in the United States. And when they are in power they will benefit from the fact that our conventional and sensible politicians have already weakened or removed the guardrails that limit state power, undermining human rights and criminalising those who have been active in protesting the climate emergence, the Gaza genocide or the tens of thousands of migrant deaths.
When Le Pen, or Farage or the AfD are in power, as Meloni is in power in Italy, Trump in the US, Erdogan in Turkey, Modi in India, and, indeed Xi in China, what makes us think that it will ever be possible to remove them? Well perhaps that thought is too soon, for now, because there are still a few institutional obstacles in the paths of some of them, and ageing and death will at least deal with some individuals eventually.
But this only puts off the day when states, armed with vast amounts of information about every single on of us, are able to interrogate that data using artificial intelligence and able to thwart or anticipate any organized political resistance. States are already criminalizing people, such as Just Stop Oil protesters in the UK, for discussing online how to resist. For people to resist they need to co-ordinate with one another, to plan, to argue, to locate their fellow thinkers. Good luck with that if all that planning, co-ordination and chat is analysed in real time by AI, which can dispatch law enforcement to arrest the putative conspirators forthwith.
More than that, we can we even hope that if political resistance is impossible we, or at least a few brave individuals, might maintain a faint light by acts such as harbouring the regime’s victims, as people did in Nazi Germany? For such acts would likely be quickly detected by AIs questioning the vast pool of data generated by our moving around, shopping, adjusting our central heating and all the mundane actions that interact with “smart” devices of various kinds. The “illegal immigrants” hiding in your basement will, almost inevitably, generate changes to the digital readings from your building that will distinguish it from the other buildings in the neighbourhood. Again, the police, of ICE, can be sent to investigate on the basis of the data.
It is tempting, in the face of such thoughts, to advocate that people who wish to live as free and equal individuals should simply find somewhere to live off-grid, perhaps practising some form of subsistence agriculture to meet basic needs. But such an option is unattractive for those of us who have been negligent enought to allow our wants to be irremediably shaped by living in a consumer society and we would, of course, also be depriving ourselves of many of the benefits of modern science and in particular of medical technology without which many of us could not survive. Moreover, there is no land in which we could live beyond the boundaries of states, as once there was, so we cannot escape to form Maroon communities or their latter-day equivalents. Insofar as any such experiment were successful, it would likely be suppressed as a threat by the incumbents of state power who would perhaps dress up their intervention as the humanitarian rescue of, for example, children, deprived by their parents of the benefits of modern civilization, including artificial intelligence. Again the “terrorists” or the “separatists” can be led away.
Of course, I’d like to believe that my fears are unfounded, that approximations to freedom and equality can still be attempted and that resistance to tyranny can still be viable, perhaps using the Master’s tools. But for now I’m gripped by a deep pessimism, sometimes relieved by the thought that I will probably not see the worst, a thought that is immediately undermined by reflection on what my children and my children’s children will have to live under.
{ 78 comments… read them below or add one }
Chris Armstrong 09.21.26 at 2:11 pm
It’s hard not to be exactly this pessimistic, Chris. It makes me think political theorists were sleeping on the job a bit – we all should have been reading more science fiction (they’ve been onto this malarkey since the 70s at least).
Syd 09.21.26 at 3:06 pm
In Stroessner’s totalitarian Paraguay, resistance groups circulated an underground SciFi comic entirely written in ‘alien’ language, impenetrable to the State and its agencies. Our one hope is the likes of Anonymous and its successors will create similar devices and also systems of obstruction to render the AI surveillance engines useless. For me, as well as being potentially, existentially catastrophic, AI is likely to be perilously fragile and useless. We can break this stuff easily if we put our energy into it, or even better, get it to destroy itself. Perhaps when enough of us have not enough water to drink because it’s all gone into cooling data warehouses, the house of cards will tumble rapidly.
Catriona McKinnon 09.21.26 at 3:18 pm
Share all these sentiments – and recognise the final thought that at least I’ll be dead before the worst of it hits (or maybe not? And what about my kids??). Shameless self publicity alert … I have a book coming out next year arguing for ‘postericide’ – a new international crime committed by those among us who put humanity in danger of extinction. Political theorists may be out of business on the freedom and equality front but we can still say what we see. fwiw.
MisterMr 09.21.26 at 3:47 pm
A bit tangential to this story, but since Meloni was cited:
Meloni started with very far right rethoric but once she was elected she governed more like a normal right leaning politician.
Currently a new political right wing “star” emerged in Italy, a certain Vannacci, who is challenging Meloni from the rightso that she is doing more (obviously cosmetic) far right stuff because she fears V is going to steal too many votes from her (he already polls around 8%, but they might ally to each other in the next elections, who knows).
This is to say that right leaning political parties are going to have the same problems left leaning ones have, in particular once they get in government and it becomes obvious that many of their slogans are unworkable.
So, I’m not THAT negative about politics in the EU, as of now; even Le Pen and AfD might turn out to be much more normal parties than they currently look.
NomadUK 09.21.26 at 4:05 pm
Not to worry. Following the devastation of World War III, humanity will rebuild and begin expansion to the stars, and those starship commanders will be well-versed in the skills required to defeat supposedly intelligent computers with logical paradoxes and self-contradiction. Kirk shows how it’s done.
oldster 09.21.26 at 4:10 pm
“The herald of this predicament was Edward Snowden, who revealed how comprensive a monitoring of individual thought and action was possible in the world of the internet.”
That is not how I would describe the man who gave Putin more power than he had ever had before. Snowden is not a champion of freedom: Snowden is the best friend that the worst authoritarian has ever had. He enabled exactly the nightmares that you are claiming to fear.
Inter alia, we know that he turned over his entire treasure trove of repression-tools to Putin in 2013, and Putin used them to invade Ukraine in 2014. Of course, Snowden is now a Russian citizen, so he may well support Putin’s attempts to reconstitute the USSR.
If the history of internet repression is ever written, Snowden will be known as one of the people who made everything worse.
J, not that one 09.21.26 at 6:21 pm
Is there anything to disagree with here (other than maybe the surveillance paranoia)?
Conservative elites are afraid of brown and black populists. Liberal and socialist elites are afraid of white populists. Radical and left elites think everyone should spend their time criticizing the flaws of the System and pondering how tragic other people’s lives are. “Rationalist” kooks think we just need to submit to AI, which will have all the answers. Others offer single weird tricks to solve everything, whether Marx or Jesus. The rest of us are wondering if we’ll have a job, health insurance, heat this winter, and wading through phone trees to try to persuade someone to help us with all that. I’m fascinated to see what we get next.
Jeremija Krstic 09.21.26 at 7:06 pm
Yes, and also these computer programs will be able to check if you’ve produced more CO2 that allowed this month, or made a joke someone may find offensive, or voted for (or “liked”) that horrible far-right populist Meloni. Oh well.
Chris Armstrong 09.21.26 at 7:48 pm
@6 – I don’t think anything you say there is incompatible with what the OP says about Snowden (which was hardly praise).
oldster 09.21.26 at 8:35 pm
Chris Armstrong @9 —
I would be delighted to find myself in agreement with Professor Bertram.
Tm 09.22.26 at 7:06 am
Some of the tendencies described are real, and have been long known and debated, but the fact remains that we still have more freedom – vastly more freedom -, by any measure you can think of, than any generation of humans before us, at least for the last couple thousand years. There is an uptick in state repression against movements like climate justice and Palestine solidarity, which is very worrying, but the methods and intensity of this repression aren’t much different from what happened for examples in the 1960s. And resistance both political and practical against ICE in USA is definitely happening, there is growing resistance against Palantir, against surveillance (https://boltsmag.org/jefferson-county-florida-rural-sheriff-dismantles-flock-camera-network/), against AI and data centers in general.
What concerns me more than AI doomsaying is the chokehold that the oligarchs, the hyper-rich have on almost all the mass media (including social media of course). We can express our opinions freely online, denounce evil all we want and call for resistance. But what does it matter? Voters vote for the likes of Modi, Erdogan, Netanyahu, Trump, and assorted European fascists not because they are coerced by an oppressive big brother, they do so of their free will. Yes they are free. But they are inundated with propaganda controlled by a handful of oligarchs who have literally invested billions to strengthen fascist movements in the apparent belief that this is their best bet for keeping their ill-gotten wealth and power. And there is no counterweight whatseoever. There are three kinds of mass media now: the social media directly controlled by fascist oligarchs (Musk, Zuckerberg etc.), the traditional media directly controlled by fascist oligarchs (Springer, Murdoch, Bolloré, Bezos), and the traditional media controlled by more or less centrist publishers who hate the left more than the fascists (NYT etc.). So when a socialist with a funny name is elected mayor of New York City, or a left party wins an election in a German city state, not just the fascist but also the non-fascist media are in complete meltdown like it’s 1917, just beyond imagination frankly. Under these circumstances it’s amazing that the left still does sometimes win elections. Propaganda works, it doesn’t work always and not on everybody but it is effective most of the time, and we have no idea what to do about it. The wealth concentration that gave rise to this insane level of media concentration is simply incompatible with the survival of democracy, and probably with the survival of humanity.
lurker 09.22.26 at 8:09 am
‘or made a joke someone may find offensive’ Jeremija Krstic , 8
We do not need AI for that, good old-fashioned human informers will do the job, for free:
https://en.wikipedia.org/wiki/Reprisals_against_commentators_on_the_Charlie_Kirk_assassination
D. S. Battistoli 09.22.26 at 10:06 am
I’ve recently written about another angle of the AI challenge: what happens if the tech broligarchy creates moral persons (say, in the Lockean sense of having the capacity for reflexive cognition)? I don’t address the increasing tendency to treat non-human, non-corporative entities, like the Irquis River in Ecuador, the Marañón River in Peru, and the Mar Menor lagoon in Spain, as having legal personhood.
While I respect and learn from Chris’ reasoning here, I am inclined to separate from him on one key step, above:
Now, I’ve lived, for years, in the United States, one of the most resource-intensive economies on the planet. And in a Maroon community (and in a city in the country where the Maroons are one of the largest ethnic groups, a country that is carbon-neutral). I currently live in the Global South, and have neither central heating nor central air (though three smartphones, a tablet, and two laptops at home might count as a lot of “smart devices”). I say this not to become one of Doug Muir’s “guy[s]—it’s always a guy—who wants you to know that he was with British Forces Arabian Penninsula at Aden back in the day, and nobody had ever heard of this aircon nonsense” but just to indicate that I am not speaking merely theoretically. I can still definitely be wrong, of course.
Thinkers as diverse as the authors of the Vedas, Karl Marx, and Henri Bergson have been advising material asceticism for millennia. I would not go as far as saying, as George Orwell famously did in 1939:
And of course, the British standard of living has only risen since 1939 (though so has that of most societies that look “ascetic” when viewed on Google Earth while sitting atop the cliffs of Dover).
Yet to paraphrase a Bob Dylan line, “there are a lot of perfectly good ways to be ascetic.” I’d be fascinated in a real give-and-take about what desires and tastes are truly irremediable among otherwise morally upright folk.
Richard Melvin 09.22.26 at 1:53 pm
Worries about surveilance seem a bit of a distraction. Neither Hitler or Stalin needed any such tech to become unoverthrowable by any internal movement.
Total defeat in war aside, dictatorships generally end when the army refuses orders to shoot protestors. That can’t happen if there are no protestors, or if your miltary accepts those orders. So the two keys areas are information control and physical enforcement.
For physiucaly enforcement, drones capable of autonomously shooting peaceful protestors could likely be assembled, from off the shelf 2025 tech, by even the weakest of governments. Which leaves media control as the key battleground. albeit one mostly fought between cnetrist billionaires and fascist billionaires.
RM 09.22.26 at 5:39 pm
There was surely good reason for Snowden to flee the extraordinarily punitive USA (esp. when it comes to whistleblowers). But given the American reach there were surely very few places he could flee to. Besides its a bit anachronistic to judge him for what Putin and Russia may have become. I can recall other times and other evaluations.
John Q 09.23.26 at 5:19 am
Some more optimism on developments in the EU.
https://substack.com/@jquiggin/p-216825620
But gloom about the US https://johnquigginblog.substack.com/p/us-republicans-are-worse-than-any
Thomas P 09.23.26 at 6:46 am
Richard, I wouldn’t say that Hitler was “unoverthrowable”, he was just extremely lucky to survive all the assassination attempts on him.
John Q, when it comes to Sweden the good news is that the far right Sweden Democrats lost votes, but the depressing fact is that this happened to a large extent because other parties have adopted the same anti-immigrant policies.
J, not that one 09.23.26 at 12:15 pm
John Q,
The free AI can’t seem to create a correlated list matching income with the 2024 vote for Suffolk County. Of course the NYC region does not house all the nation’s rich people, and if like Germany we had a party like the Christian Democrats, mild mannered conservatives who like law and order (and think the Democrats are Communists) might have had a different option. Regardless, obviously we’re seeing a move to the right, and I suspect those “respectable” right wingers who insist they’re not motivated by money, but only by the high value they put on everything and everyone being in their place, will continue voting for that, one way or another. Unluckily some places will fail to find a “respectable” way of doing that.
LFC 09.23.26 at 8:07 pm
D.S. Battistoli @13
Thinkers as diverse as the authors of the Vedas, Karl Marx, and Henri Bergson have been advising material asceticism for millennia.
I’m surprised to find Marx being cited as an advocate of asceticism. (His own standard of living was not particularly high and he relied often on help from Engels, but that was a result of circumstances probably more than anything else.)
Alex SL 09.23.26 at 10:55 pm
AI being used as part of a toolbox of oppression is a plausible scenario. However, even this post suffers from the vagueness of the concept of AI and the lack of any specifics regarding the threat. There are three aspects to this:
First, the examples of what AI could do all appear to be things that simple algorithms from ten years ago can do. If Facebook algorithms can be in the news years ago for supposedly knowing that a woman is pregnant before she does, as demonstrated by the advertisements she gets served, then an equivalent algorithm could presumably tell that refugees are hiding in your attic. What we are currently calling AI – word prediction models – are less useful for this than much simpler technologies, and a post with the same general argument could have been written in 2018.
Second, neither the word prediction models nor the good old algorithms seem to be working very reliably. The Youtube ad algorithm currently thinks I really, really need to know about two different brands of credit cards for overseas travel despite not having any travel plans and despite being viscerally repulsed by the tone of those ads, and embedding ads in large language model services seems to be a commercial failure, at least to date. And this is where money is directly on the line! I somehow doubt that a corrupt neonazi government that hates the concept of expertise would be able to deploy more competent systems.
The realistic risk model here is not a competent AI identifying the house where a dissident is hiding, it is bumbling goons hitting their deportation quotas by grabbing innocent citizens out of their cars; it is not a competent AI planning military strikes with previously impossible precision to ensure that the Nazis win WW3, it is bumbling soldiers blowing up a girls’ school and then pointing at AI as the alleged decision maker to avoid getting shouted at by their superiors (which is the worst repercussion for war crimes they would at any rate have to expect, unfortunately).
The only way out of this is again to postulate a hypothetical future AI that works perfectly. Don’t mind me, then, while I stand over here postulating a flying unicorn that also lays golden eggs and worry about how it could influence the price of gold.
Third, the way a dictatorship establishes is by having a plurality of citizens be okay with having a dictatorship, and the way a dictatorship ends is if a a vast majority of citizens want it gone. The former never needed AI to happen, as demonstrated by anything from Czarist Russia to the Third Reich being able to oppress to their heart’s desire without it. The latter usually happens when bread or, these days, petrol get too expensive, and in the future, it will likely be when water runs out (climate and/or overuse) or electricity gets too expensive. If millions of people get seriously desperate to see change, the secret police quickly turn into a smear on the ground, and the military would have to shoot their own sisters and mothers to maintain the regime. A model that statistically predicts what words were associated with other words in its training data will not be of much consequence in that scenario.
Therefore, it seems odd to discuss our current predicament primarily in terms of AI. We are facing grave risks, we have grave problems, and I agree that democracy may be on its way out, followed within decades by complex technological civilisation. We are likely to go the way of the Mycenaeans, of the Indus Valley Civilisation, of the Olmecs, and of the Roman Empire. But the framing around AI appears more based on the current large language model hype than on an objective assessment of the key factors at play.
When I think about what is causing our predicament, I contemplate unsustainable resource use, global warming, social media, inequality, and generations of politicians, journalists, and voters who have grown up without ever being materially affected by any real problem and who therefore see elections and decision making as “a kind of entertainment & mode of self-expression”. This last statement is a quote from a recent Bluesky post about a conservative Republican who, after spending decades sheltered and privileged, described a nearby data center as “the first time in my adult life where there’s been an issue that’s come up that I feel like is going to directly impact me”. Cuts to food stamps, ground water levels dropping, global warming, ICE brutalising innocent people, why should I care? Doesn’t affect me! But the data center does. This is what we are dealing with, I am afraid. I hate the IP theft that large language models are based on, but they aren’t going to cause or secure a dictatorship. Dave McCowboyHat and Karen Humveedriver, however, will cause and secure a dictatorship, because they think students having pronouns or walkable cities or the taxi driver wearing a turban are such outrageous impositions that a dictatorship is a justified response. They didn’t get there through Claude Sonnet, they got there through privilege.
Jimmy 09.24.26 at 3:51 am
The post is strongest where it separates extinction fear from the more ordinary possibility of endless, institutionally useful control. Even if one is uncertain about the largest claims, the governance problem is immediate: systems with opaque incentives become hard to resist once they are embedded into everyday decisions. In my small corner of AI coding tools, I track Codex reset windows at codexreset.today for a much narrower reason, but the underlying habit is related: make the operational constraints visible before people have to coordinate around them under pressure.
Tm 09.24.26 at 6:47 am
John 16: You are of course course correct that Trump is far more extreme, in rhetoric and policies and actions, than any European right wing püarty (yet). His extremism and its normalization by the media (on both sides of the Atlantic) has actually given a boost of radicalization to his European counterparts, and has helped radicalize some of the supposedly moderate right wing parties, like the CDU, which maintains close connections to the Republican networks.
It’s therefore always weird when observers find it remarkable that Trump supports European fascists. What is more remarkable is that European fascists are supporting Trump, who’s as popular in Europe as the Cholera (https://www.pewresearch.org/global/2026/06/23/european-views-of-trump-and-the-u-s-are-especially-negative/), even among the right wing crowd. And that supporting Trump doesn’t hurt them electorally.
Tm 09.24.26 at 8:41 am
Some commenters may be interested to hear that Sarah Wagenknecht is negotiating with the fascist AFD for a sort of power sharing in Sachsen-Anhalt. Its shape is yet unclear. Of course Wagenknecht isn’t elected there or anywhere else but everybody assumes that she’s going to decide. Her person cult party BSW failed in Berlin and Meck-Pomm and is disintegrating in Sachsen und Brandenburg so Sachsen-Anhalt is her last lifeline.
We mentioned her a few times on CT arguing about whether her culture war bullshit should count as economic populism (e. g. here https://crookedtimber.org/2025/02/23/the-german-elections-a-view-from-below/ but the thread I most remember I couldn’t find (why don’t we have a search function?))
So it may be of interest what she identified as common ground with the AFD: Russia (of course), energy (more fossil less renewable), public broadcasting and education. Much affordability, very economic populism!
“Trotz der deutlichen Abgrenzung sieht Wagenknecht aber auch Möglichkeiten der Zusammenarbeit mit der AfD im neu gewählten Landtag. Insbesondere beim Verhältnis zu Russland und bei Waffenlieferungen an die Ukraine gibt es Überschneidungen zwischen den Parteien. Wagenknecht nennt zudem eine andere Energiepolitik, eine Reform des öffentlich-rechtlichen Rundfunks sowie Veränderungen im Bildungssystem als mögliche Themen für gemeinsame Mehrheiten.”
https://www.t-online.de/nachrichten/deutschland/innenpolitik/id_101437664/nach-sachsen-anhalt-wahl-sahra-wagenknecht-findet-deutliche-worte-zur-afd.html
Tm 09.24.26 at 12:30 pm
Alex 20: every word spot on. Would like to see Chris Bertram enage with these and similar arguments.
engels 09.24.26 at 12:39 pm
It makes me think political theorists were sleeping on the job a bit
Don Quixote long ago paid the penalty for wrongly imagining that knight errantry was compatible with all economic forms of society.
somebody who remembered life before the internet 09.24.26 at 1:47 pm
Don’t worry so much as the backlash is working beautifully and more and more people are going back to the basics and they might still use their phone occasionally but the internet has overdone it and when it becomes mainstream to ignore AI and embrace baking croissants in the waitingroom what the banana on the wall says will prevail –
LFC 09.24.26 at 2:10 pm
Alex SL @20
The recent flurry of “existential” concern re AI seems to have been triggered by ‘agents’, if that’s the right word, doing (undesirable or potentially dangerous) things entirely on their own. On the political oppression point: for sure, AI is not required, but it’s conceivable, at least to a non-tech-expert (such as myself), that it could make oppression/surveillance etc. of certain sorts easier and/or more “efficient.”
J, not that one 09.24.26 at 9:41 pm
There’s reasonable surveillance-AI paranoia and unreasonable. I think the unreasonable version goes something like: AI is super-intelligence; super-intelligence is omniscient and infallible and capable of integrating every existing fact into its decision making; AI will logically necessarily devote itself to looking for every possible trace indicator of terrorism, criminality, etc., and taking decisive action against it. “Agents” have little to do with this. The decision would be made when the AI characterizes something as undesirable, as a matter (according to its training) fact.
There are things to worry about wrt surveillance, obviously, but focusing on abstract arguments about technology instead of the actual details, how things work, how experts in the technology think about the issues (this was a key trait of discussion of the Snowden data dumb too), not to mention power and politics, is an excellent way to make sure nothing can be done. That’s clearly the way too many people like it to remain.
Of course if the hyper-paranoid approach seems appealing, there’s no issue. And the idea of coming to an agreement on who the experts are seems out of reach. Or we can assume this has already been done, which I assume is appealing to the same people who find the idea of a omniscient AI plausible.
Alex SL 09.24.26 at 10:39 pm
Tm,
Thanks for the kind words.
LFC,
As with the existential threat, so with the oppression, I would find this all a lot less frustrating if the discussion were more specific and less speculative with regard to technology and mechanism of action. At the root of the current hype and drama are LLMs and their equivalents in image generation. Just to be perfectly clear, they are astonishing technology, and I really wish we could go back to a few years ago when harmless fun could be had playing around with the first iterations of these models. They are great (if wasteful) for stuff like rewrite this Shakespeare sonnet in the way Trump speaks or take this photo of me and turn me into a mermaid. LLMs are much less useful for anything that requires accuracy, and OpenAI’s own researchers have published a paper admitting that ‘hallucinations’ are intrinsic to these models. Every AI pusher who claims the unreliability is a residual bug that is currently being ironed out is deluded or a fraud, and I am thinking here of some very specific people I have met, not merely of distant CEOs.
How, then, does it work for anything useful at all? In the case of coding, there is a harness around the LLM that has an enormous number of ancillary instructions for the LLM call to keep it from making known mistakes and that tests whatever the LLM spits out to ensure it works as specified. Already here, it helps to spell it out like this instead of leaving it at “AI is great for coding”. A lot of the magic comes from the harness around it and iterative token waste as the harness rejects what didn’t work. In other words, a lot of the magic comes from non-AI.
Agents are a bit of a rubbery word, but to the best of my understanding it is generally a non-AI traditional piece of software that (a) has rights to do something like search the web, send an email, or communicate with a customer and (b) asks an LLM what to do next at every step. Again, I find this framing much more useful than “AI agents are scary”. The AI is still an LLM that tries to guess what words are associated with other words and understands nothing and has no agency and is utterly inert when not prompted. The scary part is not the AI, it is that somebody was stupid and reckless enough to build software that can do something with real consequences while acting on instructions from a model that is known to be very unreliable, and that nobody seems to fire or arrest them for doing something so reckless. Just this morning I read about a story where a customer service AI agent of a fast food chain was used by a pretend-customer to cheat on math homework. They are just word prediction models. They are not actually intelligent.
To the best of my understanding based on reading and listening to actual computer scientists, the scary AI hacks of recent weeks are done by non-AI software that is given permission to do things online, unsupervised, while running in an endless loop constantly asking an LLM “I now got this error message: xyz. What should I try next”? And after who knows how many iterations of failure, it sometimes succeeded in hacking. A serious AI researcher would have reported the number of failures in their paper. In a serious society, the CEOs would have been arrested for felony hacking. But in investment hype, different rules apply. More to the point, the power of this approach is in the endless loop funded by a bottomless token budget and the speed at which computers run, not much different from a DoS attack twenty years ago. What is happening here is closer to somebody putting a brick onto the gas pedal of a combine harvester and letting it run off into the nearby school yard than it is to SkyNet.
Again to be clear, computers can be used for bad purposes, be it in hacking or in oppression. But they were potential tools of oppression in the 1990s too. The very specific ‘AI’ we are talking about today is unbelievably inefficient in terms of cost, and unreliable. The Twitter algorithm pushing people towards right-wing ideas is much more insidious and orders of magnitude cheaper. I really wish we could be specific and precise about what exactly is happening, because only then can we do something about it.
J-D 09.25.26 at 12:14 am
I’m not sure whether your information is more recent than what I see in Wikipedia:
https://en.wikipedia.org/wiki/2026_Saxony-Anhalt_state_election#Government_formation
Lee A. Arnold 09.25.26 at 11:42 am
Chris Bertram, I share your fears that our societies were already headed into police states and that AI can be used to aid and accelerate the systems of surveillance and control.
I also think that the reverse is possible: that AI could help us to a social democratic alternative. I am a little surprised that the left has not developed arguments regarding this good possibility for AI.
1. AI could be used right now by workers to design and operate new cooperatives that could grow to challenge some of the existing big businesses in the marketplace.
2. Similarly, AI might be used right now to design and operate insurance NGOs that pay for equitable access to necessary sectors such as healthcare.
3. Why not use AI right now to write the software system for a new form of newspaper that tracks developments in security and economics, with an eye toward the future, and repackages it for continuous, easily readable updates?
4. And a little further down the road, we are not far off from the day when AI might design our entire political economy. Not in the deterministic sense of the old Soviet dream, but in a new, emergent sense that continuously revamps and equitably provides the benefits to all. Then AI would be the final winner of the socialist calculation debate.
The usual objection to this is that market capitalism is the only system which allows individuals to utilize their unique knowledge and engage in the discovery process of competition, creativity and innovation.
The answers to that objection are several, and are mostly connected to the notion that, someday soon, no one will really care. For example, in no particular order: In an era of surplus, and without the need to pay an ROI to the private financial system, we might not be so worried about efficiency… There are already many basic goods and services for which consumers really don’t care about product differentiation… Improvements in goods and services can be made rather instantly available to all…
On the question of whether inventors deserve the profits from their inventions, we might observe that, using AI, almost anybody can invent anything–and so, why should I claim personal priority for my own endless emolument? Intellectual property would become less important. An individual in search of Trumpian lucre instead of working on expanding his or her own consciousness would become a social embarrassment, an atavistic throwback to the Hayekian days of yore.
This last point is connected to a point made by Joseph Schumpeter in Capitalism, Socialism and Democracy (1942) that research and development will become so automatic, so impersonalized that it will erode the social status of the bourgeoisie. This is a valuable, psychologically penetrating book which will repay study by anyone who is interested in the psychological premises that might develop and lead us naturally to the end of the capitalist system.
LFC 09.25.26 at 4:34 pm
Alex SL @29
Thanks. A few things. First, I have no technical expertise so I can’t contest anything you’ve said at that level. Second, pretty much my only contact with AI is when I occasionally put a question into a search engine and the first answer that comes up is, in the case of Google, an ‘AI overview’ or, in the case of, say, DuckDuckGo, roughly a similar thing though it might be labeled a bit differently. Third, if your view is right on the degree of inefficiency and unreliability, it raises the question why such huge sums of money are being spent to build AI data centers, why China and the U.S. are competing in this field, etc. (Presumably the tech titans want to make money and investing heavily in very unreliable, inefficient technology would not seem an optimal way to do it.) Also, why all the chatter about potential uses in medicine and medical/pharmaceutical possible breakthroughs, etc.
Now it’s well known that AIs/LLMs can “hallucinate” fake citations and sources, and if I were an academic or someone else who routinely publishes in scholarly journals (which I’m not) I wouldn’t use AI at all for writing or editing (I am that immodest about my own writing ability, sorry). I don’t know whether/how I’d use it in research, but clearly everything an LLM says and every source it comes up with should be double-checked. But AI’s/LLMs’ limited usefulness in these respects doesn’t necessarily mean it can’t be useful in some other ways. Correspondingly, it may also pose new dangers but I don’t claim to know the specifics. Anyway, I think I’ve exhausted what I have to say here.
Tm 09.25.26 at 7:20 pm
JD 30: the situation in Sachsen-Anhalt is still in flux. The BSW has so far ruled out a formal coalition with AFD but they have clearly stated a willingness to cooperate informally. The quote I gave above paraphrases Wagenknecht’s own account of where she would like to cooperate with AFD.
They have further agreed to elect an AFD parliamentary president. Negotiations are ongoing.
https://www.mdr.de/nachrichten/sachsen-anhalt/landtagswahl/landtagspraesident-vizepraesident-verstaendigung-104,afd-bsw-gespraeche-102.html
Tm 09.25.26 at 7:35 pm
Regarding the discourse of AI doomerism, there is a curious paradox. The doomerism isn’t primarily promoted by critics but by boosters, by the tech oligarchs themselves, and the effect of this discourse isn’t to stop or limit or regulate AI development or make it safer in any way or (as Alex keeps pointing out) to hold anybody responsible for the dangerous and illegal activity of AI companies. Almost nobody is even seriously talking about any of these! To the contrary, the only effect of AI doomerism seems to be to funnel more billions into the AI bubble and make the oligarchs even richer.
Seems remarkable to me.
“AI could be a risk to humanity as a whole” (the CEO of Anthropic recently)
https://bsky.app/profile/piglet42.bsky.social/post/3mwayxontjs2e
Alex SL 09.25.26 at 9:24 pm
Lee A. Arnold,
I would be more pessimistic on that, because the most powerful models require enormous infrastructure to run, so they will always be in the hands of large corporations unless governments decide to nationalise, which is not plausible to me for the foreseeable future. In my experiments with these models for work, I found that compressed open weights models that can run on my laptop are barely usable compared to API calls to large, commercial models running on data centers that pollute poor communities. To me, it seems a bit like hoping that nuclear power plants would empower individual workers and small cooperatives.
LFC,
I am by no means an expert in finance, but to my understanding, such huge sums of money are being spent in AI for the same reason that huge sums were invested in securitised subprime mortgages, that people bought NFTs, and that Zuckerberg sunk tens of billions into the Metaverse. Mania and hype, chasing the next get rich quick opportunity. LLMs have the advantage that in contrast to NFTs or the metaverse, they do have a variety of uses, especially in spam generation, washing IP theft, replacing some writers and artists (which is arguably another form of washing IP theft), and coding (ditto), so that makes the mania more understandable this time.
Still, corporations have invested well over a trillion USD by now and appear to be making revenues (not profit) of tens of billions USD per year, so those uses do not appear to be profitable enough. What is more, leaving aside Grok’s more disgusting aspects, high-end LLMs are all more or less the same, and it is much more difficult to lock customers in and then raise prices than it is for usual corporate software subscriptions like SAP or Office; users can relatively effortlessly move to a cheaper model, causing enormous competitive pressure on token and subscription prices. It is very difficult to see how corporations can extract the trillion dollars and growing plus profit from people like you and me before the current GPUs burn out, which is sometimes pessimistically estimated as three years, and more optimistically as five, at which point the next one or two trillion dollars of investment are needed. A financial crisis and global recession seem unavoidable.
Apart from an investment bubble, there are potentially three other forces at play. One is defrauding investors; companies like OpenAI and Anthropic may know that they are unsustainable, but as long as they can convince venture capital and private credit to keep funding them, life is good. If they can dump overvalued stocks on retail investors at the IPO, life is even better. And sunk cost fallacy, and maybe something will turn up (see next point). Second, many at those two companies may genuinely believe that they are creating an AI god and the singularity within the next five years. Being unsustainable is irrelevant, because soon the world will be transformed, and debt and jobs are things of the past. Third, and this is merely me repeating what I have seen an economist argue who knows more about this than I do, large IT corporations need to pump their stock prices up by maintaining a plausible growth story. They feed money to AI labs that then flows back to them as compute use and invest in more compute because if they do not, the stock markets realise that they are mature, static ad and software service businesses instead of exciting, innovative hypergrowth stocks, stock prices are adjusted down, and any C-suite members and staff rewarded with stocks or stock options lose large sums of money. I cannot necessarily judge that, so my money is mostly on hype bubble and defrauding investors.
As a researcher, I am shocked how many colleagues use or propose to use LLMs in research, especially for tasks that have had cheaper and more reliable methods available for decades. It really is a hype, and there is seemingly great value in having a paper to the effect of “we used AI to do a thing that didn’t need AI” on one’s CV.
engels 09.25.26 at 11:40 pm
I just can’t get my head around the argument that we should ignore warnings about AI because some of the people warning about it are the people who actually built it.
agcooper 09.26.26 at 3:01 am
engels 36: if you haven’t come across it, this 1993 essay by Vernor Vinge is worth reading. It’s of ‘scriptural’ importance to a lot of people working in AI and may help in trying to understand their…I guess ‘psychology’ is the right word:
https://ntrs.nasa.gov/api/citations/19940022856/downloads/19940022856.pdf
Also Prof Farrell wrote an excellent piece which covered some of the different strains of thought/[cults]/ideologies which programmers, researchers etc., mainly located in Silicon Valley adhere to:
https://www.programmablemutter.com/p/vicos-singularity
I’m not sure if either of these will let you “get [your] head around…” but they may provide some helpful context.
Finally, I’m straying a bit here but if you want an example of the level of catastrophe that can occur when these ways of thinking overcome sometimes pre-existingly fragile minds, here you go:
https://www.wired.com/story/delirious-violent-impossible-true-story-zizians/
Alex SL 09.26.26 at 5:30 am
engels,
I think the argument is not to disregard warnings of doom because they are made by the people working on AI, but (leaving aside the much more important question of plausibility), because the people who warn about AI while working on it do not behave as if they seriously believe their own warnings.
If somebody builds a machine, says there is a 10% chance that the machine is going to end humanity, yet then continues to raise investment to keep building it, and then does another interview saying that it may end humanity a few months later, gets told by the EU “okay, let’s regulate you, then”, says “that’s not what I meant, I meant my dear buddies at METR should write another paper about alignment while we don’t change what we do”, and then raises even more money and adds more parts to the doomsday machine, would you not at some point have to conclude that the main options here are this person is (a) lying or (b) insane*? Now, admittedly, liars or the raving mad can cause a lot of damage. But all else being equal, we generally do not think that they are better forecasters of future developments than the honest and the sound of mind.
More to the point of the present thread, the focus of these very loud people from California is on doom to humanity, whereas worries about the impact of AI on data safety, democracy, public education, job quality, job security, intellectual property, cyber security, etc. are the focus of a small and currently rather side-lined group of serious academic AI safety researchers. It would be much easier to take the former seriously if they visibly cared about that second set of less sensationalist but more plausible issues, which are at the core of the OP.
*) Here to be understood as having cooked their minds in Rationalist and Longtermist blogs and seminars until they seriously came to believe that sufficiently intelligent software can become a god.
MisterMr 09.26.26 at 7:09 am
IMO we should be wary of AI the same way we should be wary of nuclear or social media (technology with potential very high both upsides and downsides).
What IMHO we should not do is to assume that AI is the singularity that will change human history more than the previous two (or stuff like electricity, running water at home or the washing machine) did.
A lot of people push for this “singularity” view, but some of them are AI company shills who use this kind of fearmongery as a form of well disguised advertising, as TM above implies.
For example, Trump is against regulating AI because he assumes AI is the fundamental technology of the future and he fears the USA will fall behind China in the AI race; this kind of stance is not against AI fearmongery but a consequence of it.
Richard Melvin 09.26.26 at 8:18 am
__War is a Racket_ was writen by Smedley D. Butler, a US general.
From this we can apparently conclude that US miliary and foreign policy is perfect in every way.
engels 09.26.26 at 10:31 am
*Monty Python voice* Alright, but apart from data safety, democracy, public education, job quality, job security, intellectual property, cyber security, etc how could AI EVER doom humanity?
Tm 09.26.26 at 12:08 pm
Engels 36, I don’t advise you to ignore the warnings. But I’d be curious to know what it means, to you, to not ignore the warnings? What are you gonna do specifically as a result of the warnings? Invest in the Anthropic IPO? Write blog comments about the need to abolish capitalism? Or burn down a data center? If the latter, rest assured that opposing data centers doesn’t require buying into the AI hype /doomerism.
Alex is right of course. If the Anthropic CEO seriously believed that his own product is a threat to humanity, you’d think that maybe such a belief should show up in his actual actions. Because he’s the one person in the position to actually do something about the danger posed by his own product – contrary to you and me.
Thomas P 09.26.26 at 1:51 pm
Tm, they are also stuck in “If we don’t do it, someone with even worse security thinking will get there first.”
For us who aren’t directly finiancially dependent on AI, the best course of action seems to be to act for banning building more AI-datacenters until we know more about the risks. It will crash Anthropic and a bunch of other companies in the business, but that´s mainly their problem, and they did call for it…
I’m actually surprised there hasn’t been attacks on data centers given how many people hate them nowadays. For air pollution, higher prices on water, electricity and computer components as well as for the risks inherent in the technology. A trial where the defendant could bring up testimony from Altman that his technology might wipe out humanity would be interesting and a jury might decide to nullify the process.
J, not that one 09.26.26 at 2:07 pm
Tm, engels
We’re dealing with people who think “printf(“I am sad.”);” is probably sentient. Expecting them to understand anything is probably hopeless.
(Note I’m not making a philosophical statement. I mean they literally can be taken in by a performance that isn’t even good.)
engels 09.26.26 at 2:10 pm
If the Anthropic CEO seriously believed that his own product is a threat to humanity, you’d think that maybe such a belief should show up in his actual action
I understands he holds it has 10% chance of killing all of us (including him) and a 90% chance of making him a gazillionaire, and he’s decided to roll the dice. While that may not be admirable it’s hardly inscrutable.
engels 09.26.26 at 2:41 pm
I’m not buying into any hype or assuming anything about superintelligence, the singularity, etc, just computing technology we more-or-less have now, whether or not may that even deserves to be called AI (to take a very simple example, the paperclip problem doesn’t assume superintelligence—but this was all explained ad nauseum on the previous thread).
engels 09.26.26 at 2:57 pm
What are you gonna do
This is a start:
https://www.sanders.senate.gov/press-releases/news-sanders-casar-introduce-legislation-to-ban-artificial-superintelligence-and-temporarily-pause-advanced-ai-development/
Lee A. Arnold 09.26.26 at 7:20 pm
Alex SL #35: “the most powerful models require enormous infrastructure to run” etc.
Running a business can be complicated, but it isn’t brain surgery. Do you really need the most powerful models to run a business? There are already off-the-shelf business AI packages available which are optimized to run on smaller systems without being connected to the cloud. Meanwhile, on the hardware end, laptop capability will continue improve (up to some thermal limit which hasn’t been reached yet) to run bigger AIs.
Down the road a bit, if you’re looking to win the socialist calculation debate, those darn commies in China will be happy to help you with the weights to overthrow western capitalism.
Of course, a similar solution could help the Chinese people to overthrow the crooks who run the Party.
Alex SL 09.26.26 at 10:05 pm
engels at 41,
This comment does not make it feel as if you are arguing here in good faith. Perhaps you could directly answer with yes or no the question of whether you understand the difference between these two threat models in terms of both plausibility and amount of destruction if realised:
AI becomes sentient and super-powerful, decides that it should remove any threat of being turned off, and kills nearly all humans (somehow) with a designer virus, leaving only a few survivors on isolated islands, then later (somehow) sends killer robots to finish them off too.
AI is used by malicious human actors to flood the web and media with propaganda and disinformation. Billions of people who have no particular interest in participating in democracy continue living their lives and do not care and oh look, an AI-generated cat video, isn’t that hilarious?
Follow-up question: Do you understand how frustrating it is to have it repeatedly insinuated that I reject the second as even worthy of consideration because I reject the first as extremely implausible? Key words conflation, equivocation, motte and bailey.
Lee A. Arnold,
I can only speak from my personal experience, e.g., testing whether an LLM can take data from a text and convert them into a table format for me to use in downstream work. ChatGPT via API calls made errors in 5.8% of cases and ignored direct instructions such as “don’t skip any rows in the table” in ca. 0.2% of cases. Even that means that the results are unusable without very careful checking, which defeats the purpose of automation, but those are still decent numbers. As for open models run locally, however, allow me to quote at length from our paper:
When tested on one description, DeepSeek-R1 inexplicably retrieved most numerical values in µm (e.g., “500” instead of the “0.5 mm” found in the source text) despite mm being specified in the prompt. Mistral ignored direct instructions not to unite minimum and maximum dimensions of the same organ into one table row, even when we tried alternative prompts, including an empty example table and the instruction to use it as a template. OpenChat produced the best results for the single query and was therefore chosen for full replicate test. […] Despite the promising first test, OpenChat unpredictably skipped traits for 40 of the 109 descriptions, often, like Mistral, united minimum and maximum values for the same organ into one row, and produced formatting glitches in many of its markdown tables. Instead of consistently using “null” for missing data as instructed, OpenChat frequently used alternatives such as “Not specified”, “Not applicable”, “N/A”, “-“, “Not provided”, or “Not mentioned”. Because of this, its responses would be functionally useless for scripted processing of taxonomic descriptions at scale and required lengthy manual editing to enable the comparison of consistency with results obtained from ChatGPT. All data for one description were scored as wrong because the table produced by OpenChat was too incoherent to be worth correcting manually.
I later tried if DeepSeek would work better if I gave it a JSON template to populate with the results, and it produced ten pages of incoherent nonsense that read like the diary of somebody slowly going mad in R’lyeh. Maybe you are using models I never tried, and maybe everything will still get better in the future. But even then, I would not assume that the workers you want to empower can afford the most high-end computers, especially at current RAM and GPU prices. The risk of what an oppressive state can do with these therefore seems much higher than the promise of what working class people can do with them.
LFC 09.27.26 at 12:44 am
@ TM, Engels, et al.
Dario Amodei, Anthropic’s CEO, has written a post somewhere (Medium?) laying out his views on these matters. It might be worth reading; I frankly have no present intent to read it, but then I’m not making assertions about his actions or lack thereof.
ETB 09.27.26 at 8:33 am
Well, thank goodness that, apparently, it’s only AI that poses a threat to data safety, democracy, public education, job quality, job security, intellectual property, cyber security, etc. Presumably if we do manage to ban AI, the next step would be to go back to ignoring the marginalised – treating both them and their warnings with contempt while sneering about identity politics.
ETB 09.27.26 at 8:34 am
While handing over all data, wealth, power, and creative endeavour with no oversight or regulation is obviously A Bad Idea (let alone to our malicious, weird, and authoritarian tech-bros), that’s hardly some grand revelation about AI, is it? We can anticipate plenty of horrifying consequences, from the actively evil to the “merely” incompetent, but this was always the direction we were headed in – because the alternative was curtailing the power and privilege enjoyed by the elites, and that was never going to happen (you didn’t need to be some talented politics whisperer to see this – just to notice what was indulged with a tolerant chuckle, and what provoked unrelenting fury and hatred).
To be clear, I’m not saying “and therefore it’s perfectly fine to hand Louis Mosley the keys to the kingdom”, but I am saying our elites are edging each other on to ever greater acts of cruelty, our press is mobbing and monstering whoever they like, our nation is stripping out the copper wire to raise money for the wealthy, and our state is reacting to in/out groups with tacit support/violence as it sees fit – and it’s difficult to see how any of that would stop, even if AI vanished tomorrow. As the UK continues its descent into Children of Men, with protestors criminalised for the crime of talking to each other and the tools to spot the next ones quietly falling into place, I can’t help but wonder who was more correct – the blue-haired students warning about systems of oppression, or the very sensible people who have spent the last decade screaming at them.
So, by all means, ban data centres and AI development if you can – you’ll hear no objections from me. My point on the other thread was that I suspect the future will be less re-enacting Terminator, and more the sky being on fire and your neighbours dragged away in the night but still having to put on a tie and get to work by 8 sharp. But that’s just an opinion, of course.
engels 09.27.26 at 3:52 pm
I suspect the future will be… your neighbours dragged away in the night but still having to put on a tie and get to work by 8 sharp
I’m moving to Yorkshire?
engels 09.27.26 at 4:34 pm
Alex, yes I understand the difference, and I’ve never suggested anything resembling the dfirst (sorry, did I interrupt your lament about being mischaracterised?)
Thanks for the conversation.
engels 09.27.26 at 4:37 pm
Sorry: the artificial stupidity of the iPhone keyboard made that curter than intended.
Perhaps it might help to sharpen our disagreements if I say I am claiming AI (so-called) represents an existential threat that necessitates urgent legal action (much like climate climate) and will likely lead to a more unequal and authoritarian society even if this is contained, not vouching for its effectiveness in biology research nor speculating about superintelligence, sentience or naked body builders with limited English being teleported back in time to shoot people (sorry if that wasn’t clear).
And now I’ll let everybody get back to the real topic of the thread (and the most important issue of the day): why identity politics is great and Sahra Wagenknecht is evil.
ETB 09.27.26 at 7:38 pm
Comedy gold, that.
Tm 09.28.26 at 8:25 am
ETB: “Comedy gold” well I don’t really find it funny. engels’s rants are always so predictably incoherent why do we put up with this. So AI is a tool in the toolbox of the authoritarian oligarchy (a plausible claim that hardly anybody here disputes) and that’s a reason to worry less about authoritarian politics? I give up.
I’ll reiterate that the only perceptible outcomes of the AI doomerism discourse so far have been to
1 inflate the AI bubble even more, and
2 deflect attention from the climate crisis and other pressing issues, like, check notes, the consolidation of oligarchic fascism (which I argue is actually helped not hindered by the doomerist discourse).
If I’m missing anything, let me know.
engels 09.28.26 at 9:51 am
I didn’t say we shouldn’t care about authoritarian politics (or that anyone is an “Untermensch” or should shut up)?
engels 09.28.26 at 10:20 am
Chers ETB & TM, it seems you really want to have a fight about something that has nothing to do with the post and beat someone up for saying things no one here has said. Have fun!
Tm 09.28.26 at 12:58 pm
Surprised this hasn’t come up yet:
“The loudest voices warning about the end of the world come out of a very specific twenty-five-year-old internet subculture, one that understands itself as a community of people who think more clearly than everyone else. Outside Silicon Valley the subculture looks obscure, a fandom with unusually dorky literary preferences. Inside the technology industry it is anything but. Its writers make or break reputations. Its alumni staff the labs, the safety institutes, and the philanthropic organizations that decide what “AI safety” means. Its arguments, marriages, feuds, and sexual proclivities are heavily interwoven into the structure of the organizations now asking for the public’s trust about the future of the species.”
https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-the-apocalypse
Very long read, via Cheryl Rofer at https://www.lawyersgunsmoneyblog.com/2026/09/the-ai-sex-cults. To be clear, the author, Ian K. Duncan, isn’t sure how serious to take the apocalyptic warnings. But we really need to pay attention to these weird people anmd the cults they created, even if we’d prefer not to.
LFC 09.28.26 at 4:29 pm
I don’t know whom ETB @56 is quoting (ETB puts it in blockquotes) but afaict it is not anyone in this thread.
Tm 09.28.26 at 9:29 pm
Another take some might find interesting:
“ As soon as you realise the AI industry is packed with adherents to a cult of climate and fossil fuel dismissiveness, everything clicks into place.”
https://www.crikey.com.au/2026/09/22/anthropic-openai-artificial-intelligence-data-centres-climate-change-jacob-coxon/
Tm 09.28.26 at 9:34 pm
Another take some might find interesting:
“As soon as you realise the AI industry is packed with adherents to a cult of climate and fossil fuel dismissiveness, everything clicks into place.”
https://www.crikey.com.au/2026/09/22/anthropic-openai-artificial-intelligence-data-centres-climate-change-jacob-coxon/
And I almost missed:
“Pope Leo on Monday said concerns that artificial intelligence could destroy the world are not “fake news” and should be taken seriously, in an apparent rebuke of US President Donald Trump.”
https://www.reuters.com/world/europe/pope-leo-says-concerns-about-ai-doom-are-not-fake-news-2026-09-28/
Alex SL 09.28.26 at 10:01 pm
Tm,
I found that too and wondered whether I should link it here. Very good read, although I wonder if “LessWrong and its offshits” was a typo or deliberate.
engels 09.28.26 at 11:38 pm
Honestly don’t know who to believe on the dangers of AI: Geoffrey Hinton, Stuart Russell and a thousand researchers at top AI labs, or a an opinion columnist writing for “Crikey”.
ETB 09.29.26 at 6:18 am
Honestly don’t know who to believe on the dangers of climate change: thousands of researchers at top climate labs, or Jacob Coxon.
engels 09.29.26 at 9:06 am
Good work. Now go through all of the 1002 scientists I’ve referred to and find some wrongthink they’ve uttered so you can ignore them too.
Alex SL 09.29.26 at 9:18 am
This is perhaps becoming slightly off topic, but this last exchange shows how successful the followers of Yudkowski and MacAskill have been at convincing journalists that they are AI safety experts, when really what has happening is that a bunch of investors, start-up founders, managers, and, at best, coders, were at a critical time recruited into a Silicon Valley doomsday cult. None of them have expertise or qualifications *relevant to risk and safety* beyond reading a few thought experiments (“if we created an artificial god, what might it do?”) and being able to trial-and-error how to train chatbots.
One useful observation in this context is the same kind of doom-mongering does not come out of AI labs in China, nor from AI researchers in Australia or India, nor even very much from American AI researchers outside of California. Most people on the planet researching or using AI in their work are vaguely puzzled by this sensationalist discourse. What we have here is not the equivalent of all climate researchers agreeing that global warming is happening and a grave risk to humanity; it is the equivalent of hypothetically a small clique of Japanese climate researchers warning that the next ice age is coming by 2050 while the rest of the climate research community frowns and says, those guys are weird, not sure where they are getting that from. Also, all risk currently posed by ‘rogue AI’ (as opposed to indirect effects such as slop production) would disappear within a week if felony hacking laws were enforced against the leadership of OpenAI, Meta, and other related corporations. They could just sandbox their models properly while running experiments.
In fairness, Hinton is an actual AI expert, but even he doesn’t appear to understand how anything works outside of his very narrow AI expertise. He is quite famous for predicting ten years ago that by now all radiologists would have been replaced by computer vision models, which shows a fascinating array of nested misconceptions about how, well, everything works in real life, from workflows in hospitals across medical expertise and certification to liability. Engineer’s disease strikes again.
engels 09.29.26 at 9:26 am
Apologies for contributing to this nonsensical mud-flinging session. I should have just said I agreed with the post and left it at that.
Tm 09.29.26 at 9:38 am
Don’t let us keep you from believing, engels. But we still don’t know what you actually do believe.
engels 09.29.26 at 10:51 am
AI (so-called) represents an existential threat that necessitates urgent legal action (much like climate change) and will likely lead to a more unequal and authoritarian society even if this is contained
Tm 09.29.26 at 11:24 am
I really shouldn’t respond to this nonsense any more but I’m only human and calling climate change denialism “some wrongthink they have uttered” annoys me too much to let it slide.
ETB 09.29.26 at 11:48 am
I am suitably chastened by the grave accusation of both having a functioning memory and consulting it. Mea culpa, etc.
ETB 09.29.26 at 11:50 am
As part of the precariat underclass, I find there’s some gallows humour (seemingly the only kind available to me these days) in watching people, quite understandably, recoil at the prospect of living under something not entirely dissimilar to our everyday:
“Imagine how bad it would be if you were continually monitored with hostility and suspicion, your every word and action scrutinised, and you faced life under an adversarial system that seeks to crush you; where all your life is controlled by outside forces over which you have no control, where there is no appeal, and where all hope is but a distant memory?”
Well, yes. Quite. Imagine how bad that would be.
As I hope my continual repetition has made clear, I don’t disagree that AI has the potential to make things much worse (in ways both new and old). However, I can’t say that having your healthcare suspended, your life upended at another’s whim, or your rights stripped away feels significantly better coming from a human. But perhaps I’m just ungrateful for not appreciating the personal touch (“this isn’t just oppression, this is hand-picked, artisanal, luxury oppression…”).
As I half-joked in the other post, one pretty good way to avoid AI exterminating us with death lasers is not to build death lasers. I should perhaps have added that one pretty good way to stop AI being exploited by a dangerous, authoritarian government is not to have…well, you get the point. But we are already far too late for that. Even if everyone woke up to the danger today (which won’t happen), the elites are so entrenched, and so committed to burning everything to the ground and salting the ashes rather than accept any change, that reversing the damage would take decades. I don’t blame the majority of the UK – this march to oblivion has been deeply unpopular, despite the titanic efforts of the press and politicians, and has been carried out largely in defiance of the public (for all that was worth). But here we are, and here it is.
It gives me no pleasure to say all this. I live very much up to my neck in this bleakness, as do most of the people I know and care about, and I have no hope that the UK will pull out of its nosedive any time soon.
The family across the way aren’t there anymore.
Our community (to use an old-fashioned term) is still trying to find out what happened. Rumour has it they were moved on, but by who exactly and to where – well, who knows?
I still had to go in for my double shift.
So, Four Yorkshiremen thigh-slapping aside, that might be worth a degree of your contemplation. Not out of any concern for us (experience has made it very clear just how low we rank on anyone’s list of things to give a damn about), but because – with or without AI – we might very well be your future.
And on that cheery note: good luck!
Richard Melvin 09.29.26 at 1:27 pm
> One useful observation in this context is the same kind of doom-mongering does not come out of AI labs in China, nor from AI researchers in Australia or India, nor even very much from American AI researchers outside of California.
That would perhaps be a more persuasive argument if it were even slightly true. A cursory google finds numerous Chinese, Australian and European AI researchers who share concerns of what could happen if current trands in AI capability continue for another 2 to 20 years.
China:
One interesting thing about Chinese reseacrhers is that thety are in a position to have a discussion about these issues without focusing solely on the ‘AI stocks are a good/bad investment” that is 99% of the western conversation.
Meanwhile we have trustworthy American experts saying things like:
engels 09.29.26 at 7:35 pm
It’s like a jungle sometimes, it makes me wonder how I keep on reading Judith Butler.
Stuart Russell a couple of weeks ago (I also recommend his book, if you can get to a library without being followed):
https://www.theguardian.com/commentisfree/2026/sep/15/ai-safety-requirements
engels 09.29.26 at 8:22 pm
Another publicity stunt?
https://www.theguardian.com/technology/2026/sep/28/openai-new-model-astra-release-scrapped
Alex SL 09.29.26 at 9:08 pm
Richard Melvin,
Okay, I stand corrected. I had based my assessment on AI researchers who I know.