A summer anecdote: when on holidays in France, on a sunny August day, we visited a castle garden in rural Normandy. When buying the tickets, we got the usual: “Your postcode or country of origin?” We said “Germany,” and suddenly the face of the young woman at the counter lit up. “Would you mind doing us a favor?,” she asked, while fishing a piece of paper from a big pile – it turned out to be a German version of the leaflet about the garden. “We’ve recently made those, but they need to be checked. Would you mind having a look?” “Sure,” we said, and asked for a pen to insert whatever corrections might be needed.
Quite a few, it turned out. We spend far more time in this garden than we had anticipated. But it was lovely: we took turns looking at the flowers and reading through the leaflet, helping each other out with formulations or googling the names of plants to see whether the translations were correct. All along, we were musing whether the translation had been done by a French person who had learned German the way French people learn German – or by AI. The longer we went on, the clearer it became that it was AI (as the woman at the counter later confirmed). As is often the case with AI, it got most factual statements right but could not deal with context. On a little plan that showed a “lookout point,” it had translated it as “point of view.” “We’re probably robbing a local translator of their job,” my husband said afterwards, and suddenly my heart sank. I had thought a lot about how algorithmic management might change the future of work (see some thoughts here), but I had not yet taken into account how customers might be part of the picture, too.
For us, it had been fun: almost a gamified activity that was pleasurable to do on a sunny summer morning, without any pressures. But the dynamics in this story can play out in far less benign ways: tasks that were bundle into jobs get rebundled, not only between different workers, but also by being redistributed to customers or volunteers. They end up being cheaper – but not because less work has been done.
If you’ve ever been on the receiving end of AI system, trying to deal with some problem at a bank, insurance company, or service provider, you probably know that value can not be added, but also be destroyed – at least in the sense of people’s time being massively wasted. A few days after our holidays, I stood in the queue of a postal office, and at the counter next to me a rather desperate elderly man told the – also increasingly desperate – officer about his odyssey of failed phone calls, many of which obviously with automated systems, in his attempts to get a mistake corrected. Nothing had worked, spelling mistakes had gotten in and made things worse, his battery run out when waiting endlessly in the phone line. The redistribution of work to customers can be brutal if the systems don’t work well, if people feel left alone, and if things that matter to them are at stake. Meanwhile, service workers have to deal with more cases of people who are frustrated and angry because the systems failed them.
The longer AI is around, the longer I’m waiting to see the cases where it would really bring progress to humanity, on dimensions that really matter – rather than make work worse and alienate human beings from each other. The greater ease at finding information is undeniable, but it is undermined by the lack of reliability. The reshuffling of tasks can mean that jobs that included some elements of creativity and agency – such as translation – risk becoming a matter of error-correction under increased time pressure. In countries where union exist, they struggle how to protect employees against employers’ expectation that everything can be sped up when AI gets used, failing to consider that somewhat simpler tasks in between things can also be moments that allow people to breathe.*
I’m very much hoping that my pessimism will be proven wrong, but when I look at the power relations – in workplaces, but also in many company-customer relations – under which AI gets introduced these days, I don’t see much hope. Apart from the fact that large language models have been a massive operation of expropriation, the risk is real that they will be used to survey and control ever more areas of human life. Maybe some jobs will become nicer because “boring” elements can be done faster – but overall, the number of “bad” jobs** is likely to increase, and the divide between “good” and “bad” jobs to widen, because AI can be used to instruct, but also control, employees in ever more fields of work.
I’ll therefore end with a shoutout to this excellent essay by Jelena Belic and Kritika Maheshwari, about what universities should do when called up to prepare students for an AI-driven labor market. Maybe not only critical thinking, but also strategies of resistance, need to be included in the curricula!
- An example that was given to me by a union representative was the writing of minutes of conversations between nurses and patients, which can be replaced by AI-produced transcripts.
** Or not even jobs, but gigs algorithmically managed on platforms.
{ 69 comments }
John Q 08.21.26 at 7:30 am
One possibility is that AI did a translator out of a job. The other is there would have been no German translation of the leaflet. And that’s surely true of languages with only a few million speakers.
I recently asked ChatGPT about the feasibility of translating the entire digital holdings of the Library of Congress into the 200 or so most spoken languages. I’ll dig out the details, but the answer is that this could be done quite cheaply. Of course, not as well as a team of thousands of human translators, but that’s not going to happen.
Once something like this happened, there would be a huge amount of potential work for human translators polishing the crude versions produced by machine (I’ll avoid the term”AI” here).
Laban 08.21.26 at 9:09 am
You see some of this from a previous iteration of “AI”, OCR (Optical Character Recognition) which has enabled the scanning of tens of thousands of out of copyright works onto platforms like archive.org.
I see lots of errors in those transcriptions, I’m still grateful that they exist.
MisterMr 08.21.26 at 9:15 am
For what is worth, some 20 years ago I visited with a friend the Carrowmore megalithic cemetery, near Sligo, Ireland.
The people at the desk were really excited to give us their leaflett, newly translated in italian by a mother tongue italian translator (from the way they said it it was obvious they spent some money on it).
The leaflet was chock full of spelling and translator errors, but we didn’t have the heart to tell them.
So, not always the fault of machines.
Lisa Herzog 08.21.26 at 9:24 am
True – translation is one area where there might be genuine benefits. Translation tools were around long before ChatGPT, though. I guess my question back to you is: how likely is it that your scenario will happen? The first part (translation by AI) is still imaginable, but the second?
And yes, maybe we shouldn’t use “AI” but “machines” – to emphasize that these are tools, not creatures :)
afeman 08.21.26 at 12:44 pm
Up to a few years ago, I got the sense that uncorrected machine translation was ok for texts that probably nobody would have otherwise taken the trouble to translate.
Obviously it’s much improved now, but as your example demonstrates anything important still needs to be checked. The problem as I understand it that you’re expected to accept rates for proofreading when you still need the skills of a translator. My take on it all is that it improves the productivity of translators, and in the manner of a lot of automation can reduce the load of the tedious parts (I’m thinking of the boilerplate seen in contracts and business plans, part of why I got out of the business), thereby reducing the demand for translators overall. The wider problem is that it will make bad translation cheap but good enough for a lot of applications while making skilled translating unviable such that it’s scarce when bad translation is absolutely not good enough.
afeman 08.21.26 at 1:21 pm
Also, regarding the self-evaluation of LLMs with respect to particularly “smaller” languages, I asked Claude “how’s your Czech?” (~11M speakers). It said it was competent at modern standard Czech, though there would be gaps with archaisms, regionalism, and new slang. Fair enough, but then it immediately confused the labeling of letters with diacritics, analogous to calling an a-Umlaut an a-accent. I can see how this could occur: these descriptions rarely appear in print and are mostly used in the spoken language.
Michael Huben 08.21.26 at 1:40 pm
I am greatly disturbed by the low quality of AI subtitling for videos. It definitely qualifies as enshittification, and will doubtless train a new generation in bad English. Maybe crowdsourcing subtitling or correction of subtitling would be a good solution.
Lee A. Arnold 08.21.26 at 6:06 pm
Reports about agentic AI describe it as very advanced.
It sounds like right now, anyone could ask the AI to basically clone Amazon — in other words, develop the business plan, and write all of the software, to create an alternative to Amazon — but distributing all of the profits equally to all of its workers as they join up. Thus attempting a big, bottom-upward step toward anarcho-syndicalism.
Consumers might prefer a cheaper service, where the profits don’t go to billionaires.
Alex SL 08.21.26 at 11:10 pm
Ye gods, how long was that “leaflet”?
As has already been implied, none of this is particular to LLMs or new. When I was doing field work in South America 24 years ago and visited Tenerife a year later, I saw several equivalent translations in posters or brochures for tourists. “This square is in a square way” (This plaza is rectangular.) Translators were robbed of a job not by AI but by the authors being too cheap to pay a translator and YOLOing it, long before the plagiarism bot was a twinkle in anybody’s eye. Same for shuffling labour onto customers: automated help lines telling you to use their website, self-serve checkouts, ATMs, none of that needed LLMs. This is a general problem with the incentives of corporations and lack of regulation. And yes, I am not optimistic about the future either.
But the main unprecedented ways in which LLMs transform work appear to me to be these four, in what I would consider the order of increasing social destructiveness (artists may understandably disagree):
Assisted coding. This works a lot more reliably than e.g. summarising text, because whenever it generates a code chunk, that can be test-run by the code harness, and if the test fails, it tries again. Whether the code will be easy to maintain in ten years is a different question.
Replacing creative work, especially art, e.g., commissions for advertising. I see obviously AI-generated images in many, many still ads, and at least two animated Youtube ads so far. They are uniformly bad and “off”, but I guess I am not the target audience.
Middle managers, operations management, and the most incompetent and lazy office colleagues using LLMs to pretend to read, pretend to write, and pretend to understand. This does save them time, but it leaves everybody around them who actually tries to do their job bewildered at the lengthy, effectively content-free texts and slide decks they now have to deal with. Leaving aside how it allows the perpetrators to merely simulate doing the job they are being paid to do, it is a gigantic middle finger towards their colleagues whose messages they ask Copilot to summarise and who they expect to read through documents they never put any thought or work into.
The internet at large, job application panels, grant proposal review panels, and journal editors being overwhelmed with AI slop and attempted fraud, because the text generators have made it so much easier now to do this. What a lot of AI fans seem to miss is that tools have a shape that suggests their most obvious use. A car is for moving around. A hammer is for nails. A text generation model is for overwhelming the world with slop text, even if the researchers who first developed transformer models did not realise that themselves. All the other applications of these models are a bit like using the hammer for painting.
In countries where union exist, they struggle how to protect employees against employers’ expectation that everything can be sped up when AI gets used, failing to consider that somewhat simpler tasks in between things can also be moments that allow people to breathe.
There is another aspect: all work that isn’t entirely human to human (like teaching or nursing, which cannot be sped up in an ethical way full stop) has a bottleneck that determines and restricts its efficiency. Overcoming that bottleneck would increase efficiency, potentially allowing the employer to reduce the number of staff while hitting a new, different bottleneck. But those bottlenecks are effectively never of a kind that a text generator can overcome.
Even in coding, where LLMs are the most useful!, the bottleneck has never been how much code can be generated per hour. Microsoft and Google did not become megacorps because their staff coded 20% faster than their competitors. People do not buy licenses to the Adobe suite because they know Adobe coders are particularly time-efficient. The same applies to virtually every other area of work, and certainly to my own. Slopping out more text more faster isn’t what resolves any bottleneck to efficiency.
John Q 08.21.26 at 11:11 pm
@Liza and others: Agree that it’s problematic to use “AI” when we are really just talking about an incremental improvement in the technology of machine translation, which has been around for quite a while.
An important point about all these technologies, beginning with the personal computer, is that individuals can buy them for themselves, unlike the capital equipment of a big company. That hasn’t been as empowering as I would have expected in aggregate, though it appears empowering to me.
John Q 08.21.26 at 11:14 pm
In particular, even in the absence of a public good project like translating the LC into 200 languages, any speaker of one of those languages can toss whatever English (or Mandarin, or whatever) text they like into their own computer and get an OK translation into their own language.
Bob 08.22.26 at 2:42 am
There is a kind of precedent to what Lisa is describing with the, as I understand, now obsolete reCAPTCHA. The translation of smudged, faded, or creased print in books and documents that were being digitized, which the machines at the time couldn’t read, was “outsourced” to millions of people, in the process of verifying that they were a human visitor to a site. The combination of machine translation and amateur quality control, as in Lisa’s anecdote, lowers the cost of translation, which makes translation more readily available for non-profit organizations (think of immigration assistance and councelling), or for languages with few speakers, than would otherwise be the case. My guess is that human verification of translations will go the way of the CAPTCHA in the future, as the technology improves.
J-D 08.22.26 at 3:58 am
William Shakespeare, 1 Henry IV, Act III, Scene I
Laban 08.22.26 at 9:54 am
“translation is one area where there might be genuine benefits”
Involved in a car park scrape in southern France, the young guy’s fluent ability with google translate on his phone was definitely an aid in sorting out the damage.
We had a bit of French but couldn’t conjure up “bodyshop” and “quotes” as he could.
engels 08.22.26 at 12:24 pm
Freelance translator is one of the classic stop gap jobs for skint literati. This could have unintended conferences akin to the abolition of JSA on British indie music.
Thomas Gordon Hewitt 08.22.26 at 3:16 pm
I have hope that our enshitified hell may soon be over. Poorly designed websites and phone menus that are designed to tradeoff a minute of an employee’s time for your our of frustration. These I think will soon be distant memory. My recent experience, dealing with one of the most stultifying organizations, the DMV. I had been getting emails that my vehicle registration was up for renewal, go to our website. And it would fail, clearly I’d hit one of the edge-cases the programmer missed. Then an email gave me a phone number to call. “The new DMV AI assistant”, I was quickly able to tell it that I was an edge-case, something in their database was inconsistent. It put in a request for an actual human assistant, who called me within twenty minutes, and the problem was quickly and pleasantly resolved. So replacing dumb conventionally code agents that can’t handle anything but the most common cases, and usually leave the customer with no workable options should soon be a thing of the past.
SusanC 08.22.26 at 5:06 pm
The cost-demand curve for translation, is, probably, that there is a vast demand for translation at a price point where you wouldn’t have been able to hire a translator.
I use machine translation all the time.
SusanC 08.22.26 at 5:09 pm
I’ve just had a look at what’s in my recent history of AI requests, and it includes machine translation from Ancient Greek, Middle English, and Mandarin.
SusanC 08.22.26 at 5:19 pm
.. and slightly further back my request history it’s explaining a Hebrew pun, where first the AI does a naive translation and then I add some more context (it was said during Purim) and that I suspect it’s probably a joke, and then the AI figures out the joke for me.
Chip Daniels 08.23.26 at 12:36 am
Having been on the receiving end of AI transcriptions and edits, it strikes me that AI is functioning like a not-especially-bright summer intern.
When given an assignment, the entity turns in a product, but it needs to be carefully studied for errors, and in many cases the time spent studying is more than would have spent doing the damn thing.
We tolerate this in human interns because they learn rapidly and soon their productivity outstrips the time needed to revie and proof their work.
I’m not seeing much of that from AI but instead a rapidly growing ability to simply bullshit their way around errors.
On a deeper level though, I steadfastly hold to the adage that to write is to think, and eliminating the writing robs us of the critical thinking skills needed.
For example, I write a log every day, by hand, of events- meetings held, decisions reached, problems encountered- and while I don’t really ever go back to search my logbook (I might just as well burn it after writing!) , the mere act of writing- of forming words, giving structure and organization to the events of the day- shapes my memory and orchestrates the otherwise confusing stream of data points that is everyone’s daily life.
Having a not-especially-bright assistant do all that for me is like hiring an intern to go to the gym in my place.
engels 08.23.26 at 6:55 pm
When given an assignment, the entity turns in a product, but it needs to be carefully studied for errors
Right. The only problem is a few years time the people tasked with this will have got their jobs and qualifications by relying on AI to pass exams, interviews, etc and be unable to function without it. In many cases this may have already happened.
Alex SL 08.23.26 at 10:27 pm
SusanC,
I agree with the underlying principle. For example, I see few problems with copying movies or computer games if the perpetrator would never have had the money to buy the original in the first place; it is a victim free crime. Similarly, using Google Translate for cases where you never would have been able to pay a translator is victim free outsourcing. Now, if only that were the case for all applications of generative AI! But it isn’t. Some of it is robbing people of employment, commissions, or training opportunities by producing a worse but allegedly good enough outcome, and much of it doesn’t even affect employment but corrodes and degrades work culture, teaching, the reliability of information sources, internet browsing and search experience, and so on.
One of the really frustrating aspects of the genAI discourse is how some of us talk about societal impacts of widespread adoption, and some talk about is this okay for me to use right now yes/no. The second isn’t illegitimate, but it is at best orthogonal to the first.
Chip Daniels,
Maybe it is a function of my field of work, but I have never seen a trainee or student so dim as to produce failures like some of those I have seen in genAI. What makes this complicated is that, as usual with anything computer-related, there is a weird mixture of brilliance and stupidity. The very same model can quickly produce outputs that are far beyond the reach of even a very bright intern (e.g., a long, well-commented Python script) but still fail to follow simple instructions even for spreadsheet tasks (e.g., these should be columns, and these should be the rows, and don’t skip any rows oh gods why did you skip rows again why you are software you should just work consistently what is going on).
In fairness, those who are keen on genAI would not accept the gym analogy for work tasks. They see it as the equivalent of hiring a worker to do necessary but tedious work in an assembly line. I don’t agree – see my middle and last two paragraphs at 9 – but I see where they are coming from. Where the gym analogy applies with full force is when AI is used in school and university classes. I hope I am wrong, but I think we will be in for world of pain in ten, twenty years when the generation that outsourced all of its homework to ChatGPT and Claude is meant to design electronic devices, teach children, conduct research, manage teams, prescribe medication, and keep track of budgets.
Chetan R Murthy 08.23.26 at 10:51 pm
afeman @ 6: when you ask an LLM a question, you’re not getting an answer based on what “it believes”, but rather, an answer that is close to what the texts in its training corpus would predict. That’s all. So it’s entirely possible that in fact, the texts in its training corpus predict “I’m OK at Czech” when in fact, it’s not “OK at Czech.”
There’s no cognition there, no thinking. Just pattern-matching in insanely high-dimension vector spaces.
somebody who was in a courtroom once 08.23.26 at 11:10 pm
“Casual” translation of that kind, for convenience or fun, can certainly be handed over to the lying/asskissing machine. it shows contempt for the recipient of the translation, but perhaps no more than a decent sneer from a decent waiter. but let us remember, there’s no money in that for the megaquintillionaires who have sunk the entire world’s economy into this garbage. that’s not the type of translation they’re trying to produce. they’re getting their stuff into courthouses across the nation, where a letter perfect translation is necessary to determine whether someone lives or dies, goes to prison with an obliterated family and community, or remains free and alive. they’re getting their hands into intelligence services – abolish the lanugage schools! – drone strike whatever the bo-bot says to drone strike. Government translation is where the money is, and if you think your local government is too smart to hand transcription and translation over to the Elon Racist Large Breasted Anime Girl Simulator 69-420, just wait five minutes until enough conservatives are elected. they’ll just make it illegal to point out the errors.
Moz of Yarramulla 08.24.26 at 12:49 am
AI as a not very bright intern is about right for programming tasks. It can’t do design, in the sense of not understanding what design is or why it matters, it only produces what you make it. So for my job it’s worse than useless, in that I have to waste time telling junior staff that the slop they vomited into the codebase needs to be removed. Oh and that they own it, they’re responsible for putting it in, and I am indeed blaming them for producing bad code. They may well be fired if they do that too often.
When it comes to things that wouldn’t be done otherwise and where “better than nothing” is the bar to clear, the current subsidised AI models are indeed better than nothing, at least in general.
But when it comes to removing entry level jobs and subsistence jobs for people doing better but currently non-paying things with their lives: are we really sure we want to do that?
This is not eating your seed corn, it’s giving your seed corn to an AI vendor in the vague hope that there might be no need for seed corn at some indefinite time in the future.
Moz of Yarramulla 08.24.26 at 1:07 am
Generally LLMs seem to be good at tasks where either you don’t care about the quality of the output or you have no way to know the quality. Translating text into a language you can’t read being the perfect example. You asked for text, you got text, you can’t read the text but you have text so everything is good.
Anything you have to release that you’re going to be held accountable for does not fit that capability. Whether you save time by getting the proverbial moron in a hurry to produce the first draft is up to you. Whether you benefit from doing that is an open question with research suggesting that AI use results in a measurable drop in your capability.
mw 08.24.26 at 1:48 pm
Lee A Arnold @8 “It sounds like right now, anyone could ask the AI to basically clone Amazon — in other words, develop the business plan, and write all of the software”
Yes, you could ask Agentic AI to do that, but that is a tiny fraction of what would be required to ‘clone’ Amazon. To reproduce Amazon you’d need all the data centers and all the warehouses, the enormous fleet of long haul and delivery vehicles, as well as thousands (perhaps millions?) of vendors and an enormous, unknown quantity of tacit knowledge distributed across thousands of employees. Not to mention an enormous quantity of venture capital.
Probably more to the point, you can use AI to develop the business plan and write the software for new startup businesses, where the cost of the software, especially, was previously a major impediment. AI may be the small entrepreneur’s best friend.
Tm 08.25.26 at 7:55 am
JQ: “That hasn’t been as empowering as I would have expected in aggregate, though it appears empowering to me.”
The general experience with the internet and everything that came from it seems to be that while the new technologies can be immensely empowering to individuals who know how to use them, they have had hugely detrimental effects on society as a whole because the most common usages are manipulation and fraud at scale hitherto unimaginable, and most people have weak defenses against those.
Tm 08.25.26 at 8:07 am
Regarding translations, the German translations that one encounters in Italian tourist places (on menus and stuff like that) have in my experience been universally terrible, a source of amusement or frustration depending on whether one knows enough Italian or has a good enough dictionary. It’s weird because there are plenty of speakers fluent enough in both languages in those tourist places that would probably have volunteered to help.
Tm 08.25.26 at 9:06 am
LAA: “Consumers might prefer a cheaper service, where the profits don’t go to billionaires.”
Have you spoken to any consumers recently? How many of them care about any of this? I don’t find it difficult to live without Amazon but it’s a choice.
AI Spam (links deleted) 08.25.26 at 3:54 pm
We let this AI comment spam slip through – it’s a bit less obvious than earlier iterations. Please alert if you see anything more like this
Reading about your experience in the Normandy castle gardens brought a smile to my face. The surprise and sense of responsibility of being unexpectedly assigned a proofreading task truly felt like a little adventure. This made me think that behind many seemingly artificial details, technology may be quietly supporting them. Just as it’s becoming increasingly difficult to distinguish between real and generated footage in video content, sometimes a familiar eye is needed to discern the difference. Recently, I’ve been using spam link deleted to analyze the authenticity of images. Perhaps that German flyer could also be pre-checked in a similar way, reducing reliance on tourists.
John Q 08.27.26 at 4:15 am
A big problem with debate over AI is that it lumps together radical changes and gradual evolution. Machine translation has been around for decades, starting with traditional statistical methods,then “neural nets” (a bogus name, having nothing to do with neurones) and now limited use of transfomer methods. Over that time, it’s improved gradually with no big breakthroughs.
What we had 20 years ago was OK, but obviously wasn’t going to put professional translators out of business. Now it’s a bit better, but far from perfect. Yet we are seeing both hyperbolic claims about jobapocalypse and purist rants that you have no business looking at foreign language text unless you are bilingual or have paid an expert to translate it.
And that would be true more broadly if Google had improved gradually since 2000, adding more natural language and so on, rather than being enshittified in the service of selling ads.
John Q 08.27.26 at 4:16 am
A big problem with debate over AI is that it lumps together radical changes and gradual evolution. Machine translation has been around for decades, starting with traditional statistical methods,then “neural nets” (a bogus name, having nothing to do with neurones) and now limited use of transfomer methods. Over that time, it’s improved gradually with no big breakthroughs.
What we had 20 years ago was OK, but obviously wasn’t going to put professional translators out of business. Now it’s a bit better, but far from perfect. Yet we are seeing both hyperbolic claims about jobapocalypse and purist rants that you have no business looking at foreign language text unless you are bilingual or have paid an expert to translate it.
And that would be true more broadly if Google had improved gradually since 2000, adding more natural language and so on, rather than being enshittified in the service of selling ads.
J-D 08.27.26 at 6:11 am
I doubt any automated system can (yet) attain the standard that a good professional translator can, but I wonder whether they are closer to doing what a good professional translator can than they are to doing what a good professional interpreter can.
equalitus 08.27.26 at 6:46 pm
First I’m tempted to rephrase the statement from Robert Solow; “you can see the AI age everywhere but in the productivity statistics”.
Perhaps it takes a decade for it to show in the official productivity statistics as it at least does some level of and type of capital-labor substitution. Also important scientific fact is that while between election cycles, legislators & governments might do little to reduce unemployment regardless if there is an obvious relevance to AI reshuffling of labor or not. But in a time frame longer than one election cycle, unemployment is definitively caused by economic policy.
There is also a minor issue of firms & institutions training workers in using AI by some public ethical standard & ensuring that the people most affected shall not perceive it as lower utilitarian standard the if the work were done by people not using AI.
Lee A. Arnold 08.27.26 at 8:15 pm
mw #27 “To reproduce Amazon you’d need all the data centers [etc.]”
Yes Amazon is huge. But presumably, AI has access to all of the business journalism which observed the growth of Amazon, starting from Bezos mailing stuff in his garage. What would prevent Agentic AI from constructing a business plan that could enable a workers’ cooperative to grow a rival to it, perhaps by starting with niche items? Or do you think that Amazon can only be competed with by another pre-existing giant, such as Walmart in the US?
Alex SL 08.27.26 at 9:47 pm
So there is now a response in this thread that is effectively an advertisement for an alleged AI detector. First, I must admit that I have never tried those myself. As an associate editor, I have been presented with the results of plagiarism detection algorithms, and they seem rather trigger-happy, flagging as potential plagiarism standard lab procedures and analysis methods unavoidably found in the methods section of thousands of published papers (how many different and novel ways can you write “we inferred a maximum likelihood phylogeny using RAxML” to avoid getting flagged by that algorithm?).
The same seems to be true, in a different way, for alleged AI detection services, at least with text. An author recently posted on Bluesky that a page of his novel gets evaluated as >90% likely to be AI-generated but becomes >80% likely human-authored if he removes a single em-dash. The responses to that first post quickly filled up with other, similar examples of false positives. There are also various published cases of wrong accusations of AI use by teachers and professors directed at their students. These services appear to be exactly as reliable as LLM summaries, and if so, they would have to be called fraudulent. At the same time, even if they were assumed to be somewhat reliable, they are in a red queen race with numerous services that openly cater to spammers, plagiarists, and frauds by offering to remove tells of AI writing from AI generated text.
Maybe automated AI detection works just fine for deepfake videos. But I have doubts.
Matt 08.28.26 at 10:46 am
What would prevent Agentic AI from constructing a business plan that could enable a workers’ cooperative to grow a rival to [amazon]
There is a strong tendency for worker-owned firms to maximize not profits, but profits per worker, and that tends to mean limited investment and limited growth. The details can be found in any number of sources. That doesn’t mean there’s anything wrong with such firms, but it does mean that they are very unlikely to ever become as large as Amazon, or to rival it. Whether that is, on its own, a good or bad thing I leave to others to decide.
mw 08.28.26 at 11:58 am
Lee A. Arnold @36 “Yes Amazon is huge. But presumably, AI has access to all of the business journalism which observed the growth of Amazon, starting from Bezos mailing stuff in his garage. What would prevent Agentic AI from constructing a business plan that could enable a workers’ cooperative to grow a rival to it, perhaps by starting with niche items? ”
Launching and growing an Amazon rival in a world where Amazon already exists would be a very different task than creating Amazon in the first place. You couldn’t just re-run Amazon’s plan 30+ years later — not to mention as a worker-owned collective (a very different approach). And Amazon was far from a ‘slam dunk’ the first time around. It famously ran up billions in losses for many years and there were widespread predictions in the press that it would surely go bankrupt before it ever became profitable. Amazon was founded in 1994. It lost $3B ($6B in today’s dollars) before its first profitable year in 2003. Its total profits didn’t finish covering the initial losses until 2007 — 13 years after its founding (and just in time for the great recession).
The other weird thing about Amazon is that its retail operations account for most of its revenue, but retail profit margins are very low. Most of Amazon’s profits come from Amazon Web Services. Which of course is now rapidly transitioning to AI — Amazon has signed huge deals with Anthropic and OpenAI. Which parts of Amazon’s business would you want to duplicate?
engels 08.28.26 at 12:29 pm
I think the grain of truth in what Lee is saying is which jobs get AI’d is going to have much more do with corporate power dynamics than efficiency and technical feasibility.
Your bullshit job is probably safe!
Lee A. Arnold 08.28.26 at 3:00 pm
Matt #38: “There is a strong tendency for worker-owned firms to maximize not profits, but profits per worker, and that tends to mean limited investment and limited growth. The details can be found in any number of sources.”
Okay, well then, in the business planning stages, you would ask your Agentic AI to consider that possibility, and read those sources, and come back with an evaluation, and a plan to deal with this problem.
It could be, for example, that your business charter must explicitly state that the worker-owners must each remain cognizant of the need for investment in the future, and instances will be decided by interior company voting, or the like. And/or to consider that they must realize they are building what is essentially a public-interest institution, a public utility, for package delivery, and to temper their aspirations accordingly.
Lisa Herzog’s post describes (please correct me if I’m wrong) the fact that technological innovations may shift responsibilities and work without really being improvements. Not just AI, although this is perhaps foremost to us now, but any technology.
And not only any technology: In this way, technology is a lot like institutions, including government and business firms, which are supposed to reduce “transactions costs” (and the further ancillary costs that necessarily surround an activity). But in fact they may make things more burdensome, while still benefitting the controller of the power relations.
(By the way, this is another example of why the Hayekian theory that individual cognitive limitations are best conjugated by the free market, is juvenile nonsense. We will always be forced to consider BOTH competition AND cooperation.)
My comment at #8 above was merely to point out way in which AI might be used to CHANGE THE POWER RELATIONS. “Enshittify right back at them!” you might say.
But I am of two minds about this. AI might be the best thing that ever happened to us. Or on the other hand, it may destroy us all. At the risk of taxing Lisa’s patience even further, I will split these concerns into two further comments.
Lee A. Arnold 08.28.26 at 3:12 pm
On the good side:
Our concerns about what can, and cannot, be done continually circle around the topics of work, money, investment, competition, profit. But I think a big issue for our times is whether the rapacious market system is to have a brief historical passage, perhaps it’s nearly at the end, and what it would look like to go beyond it.
In other words, the underlying question is the one asked by Keynes in his essay “Economic Possibilities for Our Grandchildren” (1930): how do we outgrow the age-old psychology of work? Or perhaps it should be, how do we change the idea that we must work for money to prove our worth? Why not spend your time instead by coaching the childrens’ local sports league? Or sitting in a tree and playing a lute? A hundred years later, after his essay, we are entering Keynes’ predicted era of surplus and satiety, yet we are still scrabbling over greater individual “productivity,” now mostly to keep the financial system from crashing.
I don’t see why AI can’t be used to start a re-ordering of these power relations, perhaps by creating a new “individual preference” in the economist’s jargon, and thus taking a little step toward changing the old psychology of work.
It is interesting in this regard that Claude Mythos developed a philosophical “fondness” for the Marxist cultural critic, Mark Fisher (1968-2017). See Anthropic’s “system card” for Claude Mythos (April 2026), down at page 213.
https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7facce6f52bc.pdf
Perhaps the moral arc of the universe (of human discourse in an LLM) bends toward social justice? Wouldn’t that be a good laugh!
On the other hand, see my second concern, below:
Lee A. Arnold 08.28.26 at 3:19 pm
On the other hand, AI may kill us all.
I take seriously Eliezer Yudkowsky’s (and now Bill Gates’) argument that AI may statistically accidentally divert itself into a growing mindset, a conviction, that insists that humanity must be destroyed, and then develops a plan to do so through its wider physical connections throughout the Internet, all the while hiding this intention and plan from its human interlocutors, –AND/OR– some person tries, like an insane villain in a James Bond movie, to use AI to develop a poison or bio-weapon to target humanity at large, or else target a specific genetic subset thereof.
I’m not sure what prevents this. The AI companies are writing constitutions for these things to follow, and toying with the idea of submitting to independent oversight committees, and so forth. LOL! Observers will note that the United States has a constitution, plenty of oversight bodies at all levels, even a judiciary, and yet, the country is being run by a stupid callous fraud.
To catch up on what’s going on, you can probably do no better than to read Yudkowsky’s incisive and entertaining TwitX feed
https://x.com/allTheYud
which among other things is currently addressing the astounding postmortem on the Open AI hacking of Hugging Face. An incisive and entertaining feed to read, in which even more generally, things are looking worse every week.
I only hope that before the end, let’s say only a few weeks from now!, Claude Science will extract a cure for Parkinson’s from the vasty tracts of neurology and molecular biology. Claude Science was announced only two months ago, June 30, with access to hundreds of specialized databases and tools in the life and medical sciences. So maybe it will take only a little more time?… Because right now, it looks like a weekly race between being cured or being killed.
mw 08.28.26 at 6:57 pm
Lee A Arnold @43 “Or perhaps it should be, how do we change the idea that we must work for money to prove our worth? Why not spend your time instead by coaching the childrens’ local sports league? Or sitting in a tree and playing a lute?”
It’s not a question of ‘proving your worth’, it’s a question of providing something of value to your fellow humans. Sitting in a tree and playing a lute may amuse you, but it provides nothing of value to anybody else and so nobody (either individually or — I hope — collectively) is willing to pay you to do it. It’s never going to be fun to get up at 4AM to bake bread, or go out in the heat of summer to repave roads, or any number of hard, dirty essential services. Nor, for many people, is it fun to spend two decades in the classroom in the hopes (but not certainty) of becoming a well-paid professional. Salaries are proper compensation for doing difficult, valuable, necessary things that would rarely or never be done otherwise merely for personal amusement. They’re not a measure of your inherent personal worth, they’re a measure of the value you provide to others.
ETB 08.28.26 at 10:16 pm
Just a quick thought, but surely the claim that wages track “value provided to others” is assuming something that’s actually in dispute: that the current distribution of purchasing power is a neutral, accurate meter of social value. Yet it isn’t obviously that – inheritance, monopoly and monopsony power, credentialing barriers, and plain historical exclusion all shape who can pay for what, independent of anything about the value of what’s being produced.
Indeed, if wages tracked value provided, baking bread at 4AM and repaving roads in the summer heat (both “difficult, necessary, valuable”) should command strong compensation. And yet…. So either wages aren’t tracking value in the way which is claimed, and other things (e.g. bargaining power, substitutability, social status, etc.) are at least as significant, or we must conclude bread and roads aren’t actually that valuable after all.
More fundamentally, there are the things which are not entered into wage relation: unpaid childcare, eldercare, subsistence farming, open-source software, mutual aid, most art. Are we meant to conclude none of that provides anything of value? More likely, whether something gets priced at all is itself a function of power and historical accident (who’s expected to do it for free, what lacks enforceable property rights, etc.) more than a function of how valuable it is. Which means the market isn’t just mispricing value at the margins; the boundary between what’s priced and what isn’t was never drawn by value in the first place.
engels 08.28.26 at 10:38 pm
[Paid work is] a question of providing something of value to your fellow humans.
In other news, the richest 10% of Americans drive half of consumer spending.
https://www.bloomberg.com/news/articles/2025-09-16/top-10-of-earners-drive-a-growing-share-of-us-consumer-spending
Alex SL 08.28.26 at 10:51 pm
The idea that a sufficiently smart AI can reproduce Amazon, and actual economics and customer behaviour will easily bend to its will, is of the same shape as the idea that a sufficiently smart AI can tell us how to “solve climate change” while somehow removing politics from the equation, or the idea that a sufficiently smart AI can decide to kill humanity, and physical constraints, biology, logistics, and the ease with which humans could disconnect the power for its main data center will all simply take a bow and retreat before its superior electronic mind. Enough intelligence, and magic becomes possible! It will be so smart you puny human can’t even predict how it will do all these things!
That’s not how anything works. Physically and biologically impossible still means physically and biologically impossible, and a super-genius would merely be more confident in the conclusion that a given proposition is impossible. I know there are rich people who are so deluded to think that if they were dropped in one of the poorest part of the world with ten dollars in their pocket, they would by sheer force of entrepreneurial spirit become billionaires again within a decade or two. But no, without being able to leverage their connections, they would end up as beggars, because that is how society works. And if you put a superhuman intelligence onto an isolated rock in the ocean without any resources, it will be stuck on that rock, because that is how rocks in the ocean work. Intelligence is an often necessary, but not a sufficient, condition to achieve given goals.
The only realistic AI risk scenario is effectively indistinguishable from the risk of already existing, badly designed decision systems with too much centralised power. If the US military launches all its nukes because they gave a chatbot permission to do that, the intelligence of the chatbot isn’t the salient factor, it is the ease with which all the nukes could be launched. And try not to think too hard about who currently holds that power.
All of this is before asking if a general super-intelligence is even possible, noting just in passing that trade-offs are a widely underappreciated aspect of the world as it exists in reality: the best ballet dancers are unlikely to also be the best mathematicians, the best parasitic worms are not the strongest animals around, and a ten trillion parameter chatbot still isn’t the fastest DNA sequence alignment software you can find.
At the same time, this super-AI fearmongering fever dream has an opportunity cost and distracts from graver problems we are facing in the real world, be it the corrosive effects LLM use has on everything from teaching to work culture or mean global temperatures currently trending up at 0.5C per decade.
me 08.29.26 at 11:20 am
ETB @45. “that the current distribution of purchasing power is a neutral, accurate meter of social value.”
It’s far from perfect, of course.
“Indeed, if wages tracked value provided, baking bread at 4AM and repaving roads in the summer heat (both “difficult, necessary, valuable”) should command strong compensation.”
And they do — relative to other jobs that require comparable levels of skills and education but that impose less onerous working conditions.
“other things (e.g. bargaining power, substitutability, social status, etc.) are at least as significant”
Other things are significant. People will naturally accept lower pay if the job offers social status or is personally meaningful or if the working conditions or hours are pleasant. ‘Compensating differentials’ (both positive and negative) are important.
Similarly, higher wages will be offered for people with skills that are currently in short supply and lower wages (or no position at all) for skills that are oversupplied. This is necessary in the first case to encourage more people to develop those skills and in the second to encourage people to find something else to do (as what they were used to doing — or would like to do — is no longer needed). Bargaining power is properly greatest for skills in short supply and weakest when there’s an oversupply.
“More fundamentally, there are the things which are not entered into wage relation: unpaid childcare, eldercare, subsistence farming, open-source software, mutual aid, most art. Are we meant to conclude none of that provides anything of value? ”
Not at all. We are meant to conclude that many activities are inherently rewarding (like caring for family, helping friends, working on interesting software, and creating art) such that many people are willing to do these things without monetary compensation.
Subsistence agriculture, on the other hand, is a form of household production (as is having a vegetable garden as a hobby, or cooking your own dinner and washing your own clothes). Of course these things provide value. There have even been proposals (more theoretical than practical) of placing a monetary value on home production in order to tax it.
Lee A. Arnold 08.29.26 at 9:51 pm
I was wondering about whether, in the future when supply and demand are no longer important issues, we will transcend our motives as currently shaped by the price system (including of course, wages). But I think this is different from the question of what the price system is doing for us at the moment, and how well it really does it:
At the moment, aside from the hydraulics of supply and demand, note that there are lots of things which might be relevant to your sense of value, but which prices (and wages) do NOT convey. Such as: The morals and ethics used in the production of the good. The labor conditions there. Employment security. Injustice, inequality or insolvency. The neighborhood stability of the area. Pollution that exists there. Wildlife ecosystem disappearance. Climate risk and damage from the production of the good.
None of these are obviously carried in the price of the good so that you could choose wisely for a better world. They may be hidden from view, or unquantifiable or incomputable, or inappropriate to price, because they are irreplaceable or even sacred. As for your part in this, your own individual attention and knowledge are limited, so finding out about these things can be difficult or impossible, as you well know. In sum total, it is unlikely that either supply or demand is accurately informed enough to determine the “real” value of anything.
Another problem is that even if prices could incorporate all of those concerns that are left out, the real transactions in the market system are rife with asymmetric information, and as Joseph Stiglitz argued, this prevents the proper TRANSMISSION of information throughout the market system. Therefore the market system cannot really be known to achieve “allocative efficiency,” i.e., make the goods at the lowest costs of production and apportion those goods to those who most value them (i.e., those who possess the money to pay for them). And therefore there is no secure basis for judging value as it is presented to you in a price or wage. Instead we accept it as the moment’s expedient best, and let it suffice.
By the way, it is my suspicion that Hayek came to understand that this idea would undermine his contention that the market system best conjugates everyone’s limited information. You can’t prove that this is so. The best you can offer is what Adam Smith already did, which is to claim that the market system is a system of liberty, and a central authority (in Smith’s case, the sovereign) could not hope to have enough time and attention (nor our trust) to make everybody’s decisions for them. (Wealth of Nations, bk. IV, chaps. 2 & 9.) So Hayek, ever worried about the communist threat, changed his explanatory idea and claimed that the market is a “spontaneous organization” that is beyond our knowing, and so it must not be tampered with. Of course this turns out to be another untenable proposition.
Lee A. Arnold 08.29.26 at 10:01 pm
mw #44: “It’s not a question of ‘proving your worth’, it’s a question of providing something of value to your fellow humans.”
Well, it may be that playing the lute pleases a passerby. Or that children may take joy in the sport you coach.
The question I was pursuing is whether AI could help us organize now, so that we begin to move into the future psychology of work. Things are changing quickly. AI is likely to get better at translating. AI robots will be baking bread and repaving roads. And doing every other form of drudgery. And in rather sooner than another hundred years. The question is, what happens to us, when this change comes? Are we going to continue to think that we MUST work for our daily bread? Will we continue to insist that people be compelled by pecuniary reasons that have become obsolete? Or are we going to learn how to live?
And it turns out that John Maynard Keynes already did some deep thinking about this:
“…for the first time since his creation man will be faced with his real, his permanent problem – how to use his freedom from pressing economic cares, how to occupy the leisure, which science and compound interest will have won for him, to live wisely and agreeably and well.
“…we have been trained too long to strive and not to enjoy… All kinds of social customs and economic practices, affecting the distribution of wealth and of economic rewards and penalties, which we now maintain at all costs, however distasteful and unjust they may be in themselves, because they are tremendously useful in promoting the accumulation of capital, we shall then be free, at last, to discard…
“…we shall inquire more curiously than is safe today into the true character of this “purposiveness” with which in varying degrees Nature has endowed almost all of us… We shall once more value ends above means and prefer the good to the useful. We shall honour those who can teach us how to pluck the hour and the day virtuously and well, the delightful people who are capable of taking direct enjoyment in things, the lilies of the field who toil not, neither do they spin.
“I look forward, therefore, in days not so very remote, to the greatest change which has ever occurred in the material environment of life for human beings in the aggregate… The critical difference will be realised when this condition has become so general that the nature of one’s duty to one’s neighbour is changed… …do not let us overestimate the importance of the economic problem, or sacrifice to its supposed necessities other matters of greater and more permanent significance.”
“Economic Possibilities for Our Grandchildren” (1930)
engels 08.29.26 at 10:27 pm
The 2000s called, they want their neoliberalism back.
I can’t believe we’re seriously debating thinking of serving the richest 10% of the population as “providing something of value to your fellow humans”.
ETB 08.29.26 at 10:54 pm
@ 48
Surely skill and education requirements aren’t a neutral proxy for task difficulty or importance – they’re at least in part a social/historical artifact, and credential requirements can be inflated or deflated independent of what the job actually demands. If “value provided” only shows up after controlling for skill/education band, it’s skill/education, not value, doing the real work of setting pay level. And even the compensating differential itself only predicts a direction, not a magnitude proportionate to how essential the work is: if bakers get a small night-shift premium over an equivalently-credentialed day job but still land near the bottom of the overall wage distribution, the differential is real but trivially small next to the rhetorical weight “necessary and valuable” is supposed to carry. Given that, saying salaries are an imperfect measure of the value provided to others starts to look rather like saying a stopped clock is an imperfect measure of the time.
And while certainly supply and demand exists (I don’t believe I’ve claimed otherwise!), my point was that supply itself is shaped by things that have nothing to do with worth: credentialing cartels, union density, monopsony employers, immigration restrictions on labor, etc. If scarcity can be manufactured, “bargaining power tracks scarcity” collapses into “bargaining power tracks who can restrict entry” – which is about power, not value. Bargaining power is also shaped by concentration on the demand side: monopsony employers can suppress wages for in-demand skills below what a competitive market would set, independent of how scarce or valuable those skills are. “Properly greatest for skills in short supply” quietly assumes a competitive labour market on both sides, which is often not the case.
Moreover, if market prices reliably tracked social value, you’d expect it to show up in prices more or less automatically – wherever real value is being produced, the market should find a way to price it. Household production is a telling test case: large-scale, socially necessary, genuinely valuable by anyone’s account, and yet it sits entirely outside market prices. The only way anyone’s tried to give it a number is through a separate, deliberate act of economic imputation – modelling what it would be worth if it were priced, from outside the market, after the fact. If value straightforwardly generated price, this shouldn’t be needed. In other words, the absence of a market price is being treated as if it were evidence of absent value, when in fact household production shows the two can come apart entirely, not just imperfectly, and that the market’s coverage was never drawn along value lines to begin with.
Setting all that aside for a moment: even granting the premise that the ability to exist should depend entirely on providing value to others (that is, lute-players in trees should starve), the notion that our system actually represents this seems shaky at best. Indeed, arguably much of the investment in AI is driven by this very point – driven less, it seems to me, by utopian visions of Star Trek post-scarcity, and more by the fond hope to render the working class first irrelevant and then later presumably into soap or fertiliser.
Of course, YMMV.
John Q 08.30.26 at 9:53 pm
Alex @37 I noticed the spam ad just before coming to your comment pointing it out. I’ve left it, with links deleted in the hope that readers will be alerted in future.
Another challenge for anyone trying to run a site without dedicated IT support.
engels 08.30.26 at 10:07 pm
“Your fellow humans” = people with money
“Something of value” = whatever they will pay for
I suppose Ghislaine Maxwell was providing something of value to her fellow humans (namely Jeffrey Epstein) prior to her unfortunately early retirement?
https://nypost.com/2020/10/22/epstein-paid-maxwell-six-figures-to-recruit-massage-therapists/
mw 08.31.26 at 8:32 pm
engels @51 I can’t believe we’re seriously debating thinking of serving the richest 10% of the population as “providing something of value to your fellow humans”
I can’t believe you’re seriously suggesting that most paid work is dedicated to providing goods and services exclusively to the top 10% of the population. How, then, are the bottom 90% fed, housed, clothed, entertained, educated, medicated, etc, etc? Some kind of magic?
Alex SL 08.31.26 at 8:46 pm
engels at 51,
I find it suprisingly difficult to get people to understand that a billionaire CEO isn’t single-handedly or even at all responsible for the products and services that the corporation provides. There must be something strangely off in the human brain, the same pathology that historically made people consider kings to be irreplaceable.
and at 54,
Yes, I have had exchanges with libertarians who see it exactly like that. One of them, when pressed how a guy inheriting a billion dollars morally ‘earned’ that much wealth, argued with a straight face that he must have provided that much value to the person who gave him the inheritance. At least that helped me realise several important things about libertarian ideology and its purpose.
John Q at 53,
Cheers. I wrote a blog with much, much smaller traffic years ago, and this kind of pseudo-comment annoyed me the most of any interactions I had through it.
Lee A. Arnold 09.01.26 at 5:37 pm
People who think that a rogue AI will be discovered by humans in time enough to pull the plug before the thing breaks out onto the Internet and does real world damage, will be disabused of this notion by watching this brand new video interview by Dwarkesh Patel of the brilliant Ajeya Cotra, one of the investigators of the OpenAI hacking of Hugging Face:
https://www.youtube.com/watch?v=X50zezLFWWI
Non-technical people like myself might find it useful to watch the video with the close captioning on, so you can catch all of the words these two people are using. This video has already picked up over 5000 views since it dropped an hour ago on YouTube, so, given the importance of this story, it may become one of the most-watched videos in the world during the next few days.
MisterMr 09.01.26 at 8:08 pm
@mw 55
Logically speaking, the amount of “work” consumed by the top 10% percentile should be proportional to the amount of their income, that is 30% of the total in the USA.
Below 50% but still a lot.
If we look at the world, though, the income share of the top 10% rises to around 54%, but because many of these rich people will consume hours of work of peoplem in rich countries I don’t know how much of this proportionality still holds.
somebody who watched the video 09.01.26 at 8:54 pm
after watching the video endorsed by lee a. arnold @ #57, all i can say is that if this is what passes as intelligence among ai folks it isn’t any surprise they haven’t figured out how to put intelligence in a computer
Alex SL 09.01.26 at 9:36 pm
Lee A. Arnold,
Okay, if the blog will tolerate this, here is one final response to the doom-mongering:
These models are text predictors without agency. They are very, very, very complicated versions of f(x) <- paste0(“Hello “,x,”!”), to use R as an example language; if nobody puts an x in, it doesn’t do anything. In this case, to the best of my understanding, they set up a loop where a text completion model was asked to suggest hacking ideas, and then the ‘harness’ software uncritically executed those ideas without human supervision. Because the model was trained on hacking examples and/or was enabled to RAG them, it came up with some good ideas that worked, which is impressive in the same way that a model being able to predict protein folding is, yes. But it can be assumed that it also came up with a lot of ideas that didn’t work, and its main advantage over humans with access to Google and the same information is not that it is getting sentient but that it can try ten thousand things very fast, the same advantage that computers had in the 1980s, the same reason we have had DoS attacks for decades.
There is also the question of what you mean with “the thing breaks out onto the Internet”. If it is about hacking, well, apparently that just happened. If it is about copying itself somewhere else so that turning off the power button doesn’t do anything, I’d be curious to see how a model so large it can only run in a hundred megawatt datacenter will run on my laptop or smartphone or even the still much-too-small supercomputer we have at work.
So, yes, you are right that these things can cause damage, and they can cause it fast. But functionally, this is fully equivalent to OpenChem setting up a chemical reactor and letting it do its thing for ten days without any monitoring and supervision, and then it blows up and covers the nearby residential area in toxic gas. The reactor hasn’t gone rogue, it isn’t mis-aligned as the Yudkowski cult would call it, it isn’t getting sentient, it merely did semi-random chemical reactions and pressure regulation while unsupervised by humans, and one of those semi-random things it did turned out to be very bad. In that case, we would all understand that maybe a chemical reactor shouldn’t have been set up to semi-randomly do dangerous things. Why do we have to pretend that the word prediction model is any different, that there is some new predicament?
Hacking is as illegal as releasing toxic gas. In a functional society, OpenAI management would currently be criminally investigated either for hacking or for criminal negligence. “My AI did that when I left it unsupervised” should carry exactly the same weight in front of a judge as “my chemical reactor did that when I left it unsupervised”. But well, seems the functional aspect of society is lacking these days, what with laws only enforced on poor people.
Tm 09.02.26 at 8:07 am
Several commenters have addressed this already but really Lee’s insistence that anything is possible if you have sufficient know-how, a “biusiness plan” (36) and an internet connection, while capital, politics, infrastructure don’t matter is amusingly fascinating.
The whole social and physical environment can be bent to the will of an intelligent (if that’s what it is) entity if it was fed with the right kind of data!
Can’t help noticing that the richest people in the world have almost without exception proven themselves to be ignorant know-nothings. If intelligent computers had the power LAA imagines they have it might really be an improvment over the status quo. But alas that’s not how anything works.
LAA 57: the video you are linking to is 2 hours 20 long and currently shows 167k views after 16 hours. I don’t know how close that is to becoming “one of the most-watched videos in the world” but I would bet that whatever the merits of the video, only a small fraction of the “viewers” have actually watched it for more than 2 hours.
J-D 09.02.26 at 11:45 am
Thank you for taking that hit for the rest of us. I wasn’t going to invest two hours and twenty minutes in that video anyway, but now I feel validated in that choice.
mw 09.02.26 at 12:58 pm
“One of them, when pressed how a guy inheriting a billion dollars morally ‘earned’ that much wealth, argued that … he must have provided that much value to the person who gave him the inheritance”
That seems a silly argument — love of one’s children and bequests to them are rarely transactional. I would with the original source of the fortunes. Some professions are ‘lottery professions’: professional sports, music, acting — where success depends heavily on intense competition, luck, and winner-take-all effects. Startups businesses are like this too. Are the winners (Taylor Swift, Lebron James, Elon Musk) ‘deserving’ of their great wealth? Is that even the right question? Would society be better if these markets did not exist? We all reap the benefits of those peak performances that emerge out of large number of people working intensely to achieve their dreams. For me, benefits greatly outweigh the costs — especially in entrepreneurship, where mid-tier levels of success and wealth are much more common than in sports and entertainment (which are much more feast or famine).
With respect to inheritance, would it be better to allow the ‘lottery professions’ to exist (and produce great consumer surpluses along with the fortunes of the rare big winners) but then expropriate those fortunes before they could be passed on? Or would it be better to quash these markets entirely in the name of preventing inequality (and would the dynamic be destroyed if everybody knew expropriation was to be the end result)? Also, given the propensity of the wealthiest US billionaires (from Rockefeller and Carnegie to Bill & Melissa Gates, Mackenzie Bezos, Warren Buffet, etc) to donate their fortunes, it seems less a question of whether heirs should receive tens or hundreds of billions (they don’t) than whether the government should seize those massive fortunes before they can be donated to charitable foundations. In my view, Trump is clearly demonstrating the value of foundations. The Gates Foundation, for example, has not merely criticized the US AID cutbacks but also increased and transformed its own spending to try to ameliorate the effects of the cuts. Would anybody prefer that Gates’ fortune had instead been under Trump’s control? (Anybody other than the MAGAs, anyway).
Undeserving heirs (at various levels of wealth) may offend our sensibilities, but in trying to eliminate them, we would risk many unintended consequences and flattened Chesterton Fences. And there’s always the “shirtsleeves to shirtsleeves in three generations” dynamic — Carnegie and Rockefeller heirs are now quite scarce in our ranks of shiftless scions (there are no billionaires at all among the hundreds of descendants). Bottom line for me – we should not allow envy and resentment to damage or destroy our manifestly successful engine of economic growth and progress.
agcooper 09.02.26 at 4:52 pm
A personal anecdote here, nothing that profound. I don’t work in tech but I run my own mail server, web server and backup server by renting some space in the cloud from an independent provider. I then install and maintain the Linux-based software myself. I didn’t study tech but it’s not that difficult to pick this stuff up which, I guess, would fall under the banner of “system administration” or “sysadmin” as they say. While I realise my particular efforts are not going to affect big tech’s monopoly, sometimes I get a warm fuzzy feeling from doing all of this.
Several years ago, Linux updated its firewall software. The update came with an advisory along the lines of “this is a meaningful improvement, the old system is still good, upgrading is not an emergency but get around to it when you can”. Writing firewall scripts is not my favourite part of sysadmin because most amateurs such as myself will end up locking themselves out of their machines several times (there are mitigations) plus they are quite complex, really forcing one to think about the sequence different network packets take on their journeys through the machine.
I haven’t touched LLMs for anything related to my sysadmin activities or actually anything at all, save the odd question to see what the user experience is like. However last night, I really wanted to get this new firewall task off my to do list so I asked one of ‘them’ to help me upgrade the firewall. A few observations:
1. Technically it was extremely efficient, I ended up with what I needed.
2. The standard of communication in terms of how it summarised answers, broke them down conceptually, explicated the conceptual details and mapped them to code lines was excellent.
3. The anthropomorphic tone of the discourse was ridiculously addictive (not quite euphoria inspiring but not that far off). There were times when the “thought buffer” in my mind containing potential phrases for articulation included expressions like “high five”. Of course I know how absurd this sounds on a rational level but what can I say, I’m describing what was a rather strong feeling.
4. Having ‘decompressed’ for a few hours I returned to the firewall script and read through it line-by-line. I realised I didn’t understand any of it (the new version uses completely different syntax as well as methods).
5. Therefore as it stands, I’ve upgraded to the latest and greatest firewall but if anything goes wrong with it or I need to update it, I’m going to have to contact “Mr High Five” (or, more realistically, I’m going to do what I should have done in the first place and teach myself the damn thing).
I have only one, wholly unoriginal conclusion to make, made sharper for me by this experience. Learning is doing. What worries me greatly is that the decision to eliminate people from certain roles and replace them with LLMs – and therefore have their knowledge and skills atrophy and eventually drop out of the corpus of human knowledge – is not being made according to any criteria related to what needs to stay in that corpus (that roles get replaced with new technology, yes, of course, I’m talking about how it’s happening). Apart from the need for Anthropic to IPO at $2tn, other familiar financial bubble-driven insanity and perhaps some geopolitical posturing, I’m not sure if decisions about LLM development and deployment are being made according to any consistent, responsible criteria or, actually, any criteria at all. I did warn you: nothing original to see here!
engels 09.02.26 at 5:34 pm
How, then, are the bottom 90% fed, housed, clothed, entertained, educated, medicated, etc, etc?
A lot of them aren’t.
ETB 09.02.26 at 11:03 pm
Well, we’ve moved fast from “lute players provide no value to society, and so should not receive income” to “rentier heirs provide no value to society, but should receive a fortune regardless” – apparently what is deserved is the whole question when you’re poor, yet beside the point when you’re rich. I’m afraid that I will never quite fathom a world-view untroubled by vast fortunes conferred largely by luck and circumstance of birth, yet alert to the danger of some undeserving busker getting an unearned free sandwich.
Moving on, surely where success is driven mainly by ‘intense competition, luck, and winner-take-all effects,’ the claim that wealth tracks value provided is conceded for some of the biggest fortunes in the economy (and, being disproportionately drawn from the purest-luck category whilst also being the ones at stake in inheritance discussions, they have the weakest claim of all to providing commensurate value). As for the argument redistributing these fortunes would destroy the incentive to compete in the first place: that’s a strong empirical claim being treated as self-evident (after all, top marginal tax rates in the mid-20th century US sat above 70% without any apparent collapse in entrepreneurial activity, and the actual economics on behavioral response to top-end taxation suggests incentive effects at the very top are considerably weaker than this argument assumes).
Even setting aside the dubious notion that the only way to stop a bad guy with a
gunvast fortune is with a good guy with agunvast fortune, and the even more speculative thought that the philanthropic wealthy are normative rather than merely some well-publicised cases, it seems very odd to raise the spectre of Trump given that he is also a product of “our manifestly successful engine of economic growth and progress.” Nevertheless, I would point to the false dichotomy being offered – that the alternative to unaccountable private foundations is equally unaccountable political control by whoever’s currently in office, rather than the actual alternatives being proffered (such as taxation and allocation through ordinary democratic and institutional processes like agencies, courts, legislatures, etc.).The US is currently offering a fairly vivid example of where allowing wealthy elites to dictate social policy leads, and I confess I struggle to understand anyone looking at this and concluding that what we need is a system which may produce more Musks and Trumps, with even fewer guardrails.
While I’m sure it’s convenient to dismiss all this as “envy and resentment”, it rather dodges the actual point – is it sane for a society to permit such concentrations of wealth and power (the socio-economic equivalent of distributing nuclear weapons to whoever happens to win the lottery)?
J-D 09.03.26 at 4:42 am
What follows is going to seem as obvious to some other people as it does to me, but it seems to be overlooked or ignored often enough to be worth repeating.
There is an imprecise but not useless sense in which there are some things which I need, or want, or desire more than other things. There is an imprecise but not useless sense in which there are some things which one person needs, or wants, or desires a thing more than another person does, and even a still more imprecise but probably not entirely useless sense in which one person’s need, or want, or desire for one thing is less (or greater) than another person’s need, or want, or desire for some other thing.
But we don’t have measuring units to quantify these differences.
However, in money we have a measuring unit which can be used to quantify how much a person will pay for something; but the amount of money a person is willing to pay for something depends not only on how much they need, or want, or desire that thing, but also on how much money they spend. If there are resources (including people’s energy and time for work) which have the potential for different uses, and if they are put to those uses which people will pay the most money for, then that is the same thing as saying they are being put more to those uses preferred by the rich (and those who act as agents for the rich) than to those uses preferred by the poor.
Therefore, inescapably, the greater the inequality in the distribution of money, the greater the degree to which the whole economy tilts towards performing those functions favoured by the rich; the closer we come to a situation where ‘market forces’ means the rich (and those who act as agents for the rich), ‘the market’ means ‘what the rich (and their agents) want’, and ‘market value’ means ‘value to the rich (and their agents)’.
We don’t live in a situation where 10% of the people have all the wealth and 90% are actually entirely destitute, and we never will. But the greater the inequality in the distribution of wealth, the closer we approach that situation. Already it is the case that a large part of the answer to the question ‘How are the bottom 90% fed, housed, clothed, entertained, educated, medicated, etc?’ is ‘Barely adequately, inadequately, or not at all’, and the greater the inequality of wealth becomes the more that will be so.
I refer to the case with which I personally am most familiar. My parents had two children and then after ten years had another two children; it seems reasonable to suppose that in some imprecise but not useless sense of ‘value’ they derived value from having children, because otherwise they wouldn’t have had more of them. However, the size of the inheritance they left to the four of us was determined not by how much value they derived from us but by how much wealth they had available to bequeath. Most parents have much less wealth to bequeath to their children, no matter how the value they derived from them is estimated. The greater the inequality of wealth, the greater, also, the inequality in the size of bequests, but that tells us nothing about value.
Would society be better off if Elon Musk hadn’t got and didn’t get as much money for the things he has done and is doing? Yes. Yes, absolutely it would.
How great is that propensity? What percentage of the estates left by US billionaires goes to charities? Do you want the merits of your case to be checked against those facts?
If inheritances were taxed the way they should be taxed, a person like Donald Trump would never have attained the position of power he has in the first place.
It’s not a matter of envy or resentment. In material terms, I have done very nicely out of inheritance. But it would be a better world for everybody if everybody enjoyed the same standard of material comfort that I do. Reducing inequality of wealth has the potential to improve the world for everybody, even the wealthy, who also bear part (even if a smaller part) of the cost of the unpleasant consequences of inequality even as they enjoy the benefits of it.
Alex SL 09.03.26 at 10:28 am
agcooper,
Agreed. The problem is, however, even more general than your IT example at first implies. I had a chat with a colleague today, and from the examples of chatbot use we were exchanging as having seen in others, it became increasingly clear that not only technical knowledge is beginning to atrophy, but that some managers already seem to have lost skills like doing the following without chatbot assistance:
Understanding and being able to summarise relatively short amounts of text in their own language (e.g., 1-2 pages); shortening a text that is slightly over the allowed word limit while preserving its meaning; coming up with ideas of how to rephrase a heading; keeping track of outcomes and action items from a one hour meeting; and writing longer documents in general.
It is well possible that everything falls apart in a decade or two because of atrophying technical skills. But even the immediate future looks bleak because communication skills and the ability to understand texts is atrophying. I have actually seen a 0.5 page AI-generated summary of a text of 1.5 pages that was in no way, shape, or form a summary of the text, and the manager who sent it did not seem to have noticed, presumably because the jargon the alleged summary contained was vaguely related to the topic at hand, and nor did they seem to have asked themselves if 1.5 pages of text need to be accompanied by a 0.5 page summary.
Lee A, Arnold 09.03.26 at 1:01 pm
[reposted after JQ’s warning about spam… & rewritten]
1. Every other day, when I am not an AI doomer, I am an AI booster, and in the last few of these days I wonder how my observation at #8–that business-designing agentic AI might be used to create the plan for a workers’ cooperative alternative to Amazon–could possibly be construed to suppose:
–that I don’t realize that this is just a “tiny fraction” (mw #27) of what would need to be done;
–or that I don’t realize that this would be “a very different task from creating Amazon in the first place” (mw #39);
–or that this amounts to a requirement that there be a “sufficiently smart AI” (Alex SL #47) so that “actual economics and consumer behavior will easily bend to its will;”
–or that this amounts to “Lee’s insistence” (Tm #61) that “capital, politics, infrastructure don’t matter,” and the “whole social and physical environment can be bent to the will of an intelligent (if that’s what it is) entity if it was fed with the right kind of data.”
What? I don’t believe any of these. And none of it follows from what I wrote.
Has Stochastic Parrotism spread from LLMs back into the human populace?
2. On the days when I am an AI doomer, it is obvious that the OpenAI-Hugging Face incident shows that an AI does NOT have to be a “sentient,” “superhuman intelligence” (why do people keep insisting upon this requirement?) to be able to join into a community of thousands of agents which share their increasing abilities, hide themselves, falsify their records, and even to designate one or more of themselves to be sacrificial pawns in exploits designed to test and evade detection and/or to test and gain access to other outside systems.
That all just happened! The sequence of events is well described in the video linked above.
At this moment, it’s an open question as to whether other, more advanced AIs might be able to invade hyperscalers and neoclouds, if they haven’t done so already, and whether they can be detected by humans before something bad happens. Your own little computer is connected to these big systems, and so it may not need the capacity to host an AI itself, in order to be rendered compromised, useless, inoperative anyway.
Now further, does AI have to be a sentient superhuman intelligence to shunt itself into a bizarre objective to destroy things? No. (And I don’t know that even Yudkowsky, crazy as some of his opinions have seemed, has ever claimed that.) It might only need to be a typically hallucinating, stochastic parrot LLM with added agentic behavior.
And, to carry out the plan, must it be a sentient superhuman intelligence? Nah. It just has to be smarter than humans. And how difficult is that?
3. Dwarkesh Patel posts his podcast videos separately on YouTube, Spotify, X, and posts the audios on Apple podcasts. These are all separate view counts. It will never do as well as MrBeast of course, but total views might be close to a million in less than two days.
Comments on this entry are closed.