Back in 2022, after my first encounter with ChatGPT, I suggested that it was likely to wipe out large categories of “bullshit jobs”, but unlikely to create mass unemployment. In retrospect, that was probably an overestimate of the likely impact. But three years later, it seems as if an update might be appropriate.
In the last three years, I have found a few uses for LLM technology. First, I use a product called Rewind, which transcribes the content of Zoom meetings and produces a summary (you may want to check local law on this). Also, I have replaced Google with Kagi, a search engine which will, if presented with a question, produced a detailed answer with links to references, most of which are similar to those I would have found on an extensive Google search, avoiding ads and promotions. Except in the sense that anything on the Internet may be wrong, the results aren’t subject to the hallucinations for which ChatGPT is infamous.
Put high-quality search and accurate summarization together and you have the technology for a literature survey. And that’s what OpenAI now offers as DeepResearch I’ve tried it a few times, and it’s as good as I would expect from a competent research assistant or a standard consultant’s report. If I were asked to do a report on a topic with which I had limited familiarity, I would certainly check out what DeepResearch had to say.
Here, for example, is the response to a request to assess whether Canada should adopt the euro. I didn’t give much in the way of prompting except to request an academic style. If we ignore developments post-Trump (which wouldn’t have been found by a search of academic and semi-academic publications) it’s pretty good, at least as a starting point. And even as regards Trump, the response includes the observation “If, hypothetically, … political relations with the U.S. soured badly, the notion of joining a stable currency area might gain some public traction. ”
So, just as Github Copilot has led to big changes in coding jobs, I’d expect to see DeepResearch having a significant impact on a lot of research projects. That in turn implies an increase in productivity, which had been lagging in the absence of major new developments in IT for the decade or so before the recent AI boom.
None of that implies the kind of radical change often tossed about in discussions of AI, or even the kind of disruption seen when an existing analog technology is suddenly subject to digital challenge – the classic example was the “quartz crisis” in the watch industry. Coding has been progressively automated over time with the development of tools like online code libraries.
Similarly, research processes have changed year by year over my lifetime. When I was starting out, reference aids like citation indexes and the Journal of Economic Literature were brand new, and only accessible in libraries. Photocopying articles was sufficiently expensive and painful that you only did it for the important stuff (Daniel Ellsberg’s account of the work involved in producing multiple copies of the Pentagon papers gives you an idea). Today, a well organised research assistant with reasonable background knowledge could do the same job as Deep Research in a couple of days, and without needing to leave home (to level the playing field with AI, I’m assuming the RA is free to plagiarise at will, as long as the source is cited).
The other big question is whether efforts like this will generate profits comparable to the tens of billions of dollars being invested. I’ve always doubted this – once it became clear that LLMs were possible, it was obviously possible to copy the idea. This has been proved, reasonably conclusively by DeepSeek (no relation to DeepResearch), an LLM developed by a medium-sized Chinese company at a claimed cost of $500m. It charges a lot less then ChatGPT while providing an adequate substitute for most of the current applications of LLMs.
The likely collapse of the current AI boom will have some big economic repercussions. Most importantly and fortunately, it will kill (in fact, is already killing) the projections of massively expanded electricity demand from data centres. In combination with the failure of Tesla, it will bring down the market valuations of most of the “Magnificent Seven” tech stocks that now dominate the US share market. That will be bad for the US economy, but to the extent that it weakens Trump, a good thing for the world and for the long-term survival of US democracy.
{ 40 comments }
Daniel 04.30.25 at 8:13 pm
GDP of U.S. declined in the first quarter. How likely is the U.S. to be in an official recession by July?
Alex SL 04.30.25 at 10:48 pm
As I wrote on Bluesky just a few days ago:
“I ask ChatGPT a question, answer is wrong. I use it to extract data from text, the responses are so unpredictable that it is impossible to automate the process. I ask for suggestions, the suggestions turn out not to match the criteria I gave in the query.”
Seriously, my query to suggest five journals in my field that would publish a free app even if its source code cannot be fully open-sourced resulted in the suggestion of two journals from outside my field and three that explicitly did not accept manuscripts unless the code is open source. When I asked it about the native range or area of origin of one of the worst, best-known weeds in the world, which is all over its training data from Wikipedia to many research papers, it got that simple question wrong. It just never works for me.
So, I am all for being more efficient using scripts to automate busywork away. But based on my concrete experience, I just don’t see how I can ever trust an LLM, how I can ever use it and not immediately repeat all the work it did myself to ensure it didn’t do it wrong. Maybe that is just because I never tried the right model or agentic system or whatever, but this line of argument increasingly reminds me of Sophisticated Theologians who insist that an atheist will become religious if they only read this one book. Didn’t convince you? Try this book. Still nothing? But now this book here will definitely convince you. Etc. Sorry, I will stop now, still an atheist, because I have better things to do than read twenty equally unconvincing apologist books; and similarly, at some point I have to give up and conclude that LLMs are fundamentally nothing more than statistical models that try to guess what a response to my query might on average look like without any understanding of what I meant or what the response means.
I do not doubt that there are uses for generative AI, like generating garbage e-books and low-quality images at scale. A colleague says it helps him document his code. But even putting aside environmental impacts and the fact that the training of the models is intellectual property theft at a historically unprecedented scale, the great risk in my eyes is that people will use the models while not realising that the outputs are full of errors, and once nearly everybody has got used to that, hardly anybody will be left who knows or checks anything and notices the problem. Surely, that will be a use, but in that sense, there are uses for computer viruses, bloodletting, and asbestos.
If I were asked to do a report on a topic with which I had limited familiarity…
That is a scary sentence in this context, because of Gell Mann Amnesia. I am somehow immune to it, perhaps because I am a cynical misanthrope: I immediately lose all confidence in any newspaper, colleague, Tesla CEO, or software tool once they say or do the first very stupid thing and never respect them again, which is precisely why I do not trust LLMs at all, see above. But many other people approach LLMs in exactly this way: oh yes, when I ask ChatGPT something in my area of expertise, it gets nearly everything wrong, but it knows so much in other areas! It is so useful when I have limited familiarity with something!
DCA 05.01.25 at 12:34 am
Zoom (in the US) now has AI built in; if you allow recording, it produces a summary. Looking at it for a couple of meetings of a local community board I am on, it does a good job of transcribing what is said and, from this, an OK job of organizing the recording into something coherent. But it has trouble knowing what to record in detail and what to summarize, and can miss whole statements.
John Q 05.01.25 at 1:45 am
DCA – yes, I saw this feature on Zoom, just after writing this post. Rewind still has some advantages as you can go back over past actions.
But it’s another example where AI seems like email – very useful for some purposes, but no company is ever going to make much money out of it.
Karen Lofstrom 05.01.25 at 4:03 am
DeepL (and to a lesser extent Google Translate) will do time-saving first passes on translation. The output absolutely requires human vetting. LLMs make mistakes. But even professional translators will use AI this way. When I was editing scientific papers, one of my main clients saved a lot of money by writing in Mandarin and using DeepL to translate to English. She looked over the result herself and then sent the draft to me. I found errors and mistakes, but the paper only took three hours to edit, as opposed to previous papers that took nine or ten. I think she wrote a better paper when she was writing in her own language.
Mike Huben 05.01.25 at 1:27 pm
I just tried DeepResearch (thanks for the link) on the subject of my research, the wasp family Evaniidae. I asked for comparisons between some pairs of genera.
Almost everything it said was either equivocal or wrong. Things that are REALLY clear, such as morphological differences from my own publications, continental distributions of the genera and the fact that hosts are not known for most genera were either wrong or imaginary.
I can’t imagine anybody using such a bad tool.
oldster 05.01.25 at 3:40 pm
“this line of argument increasingly reminds me of Sophisticated Theologians who insist that an atheist will become religious if they only read this one book.”
Right — a related scam is played by invoking the apologist’s favorite LLM.
“Oh, Google’s AI told you a load of bollocks? Well, no wonder — you have to use Claude!”
“Oh, Claude told you a load of bollocks? You have to use OpenAI!”
“Oh, OpenAI told you a load of bollocks? Well, but you used Version 2.0 — that was your mistake. Version 2.5x works wonders!”
Somehow they never get around to explaining why Google has already put live and online, a system which spouts falsehoods left and right. Doesn’t it seem like a problem that everyone who is well-informed knows that Google’s AI is crap, but Google still builds it in to every search, and maybe a lot of people have not been informed that its results are crap?
In a way, it is worse than Elon’s “self-driving cars” scam, because not only is there the perpetually repeated promise of a functioning system some time in the future, these non-functional systems are now out on the road, so to speak, and driving down every street and alley.
Add to that the fact that Russian bots are seeding the web with massive dumps of training data containing lies, esp. about Ukraine. Bad information is going to be driving out good at lightning speeds — Gresham’s LLM.
https://www.washingtonpost.com/technology/2025/04/17/llm-poisoning-grooming-chatbots-russia/
Peter Dorman 05.01.25 at 6:01 pm
I want to strongly endorse JQ’s point that, whatever its merits and shortcomings, the various flavors of AI will be unlikely to generate the kind of profits that would give the current avalanche of investment even a modest rate of return. This seems so obvious to me that the hard part is figuring out why it isn’t obvious to the people who are throwing billions of dollars around betting the opposite.
So here’s a hypothesis. The digital world give us representation of the real one. We can manipulate those representations in extremely complex ways and apply the results to the physical world. There are some industries, especially those related to finance and media, where the final products are themselves digital, and for them improvements in extracting and processing data, forecasting, estimating parameters in algorithms (or selecting the models themselves) can be highly remunerative. But for the rest of the economy the main value added lies in the actual doing, physically, of stuff. The AI hype applies the results of the first set of industries to the whole thing. Why? Computer work is immersive and, making things happen on your screen gives you the feeling that you have manipulated the actual world. People who spend all day in front of a screen are susceptible to this illusion.
In a related context, I wrote about how the sci-fi novel Snow Crash anticipated this development and ignited the dreams of a legion of teenage online addicts, some of whom grew up to manage venture capital funds, found tech firms etc.
But there has to be more, doesn’t there? Wouldn’t such a massive misallocation of capital need a more complex explanation?
On a different note, I think the most glaring weakness of existing AI, again in all its flavors, is its inability to assess its own risk of error. It’s this meta level that, up to now, humans can address but not machines. By error I mean not just the dispersion around a point estimate but the all the potential error associated with mismeasurement, the wrong model, etc. AI enthusiasts, like our friend Elon, seem blithely unaware of this.
Derek Bowman 05.01.25 at 6:14 pm
@Alex_SL,
Yes, I agree, and it’s refreshing to hear such a clearheaded take. Similar to your comparison with theology, I’ve compared the academic AI evangelists to those who report receiving great insights from their psychic advisor or from tarot readings. ‘Oh no, you didn’t go to the right one…’ or ‘Of course you have to use the right technique…’
I didn’t think it was possible but the AI enthusiasm has made me even more disillusioned about academic humanists than my own experience with their/our inability to deal with the material conditions of their/our own working lives.
JimV 05.01.25 at 8:51 pm
I feel (for what little worth it has) that LLM’s have been vastly over-hyped the same way advertisers over-hype every product, but are still impressive in this regard: they can do things (e.g. get B’s and A’s in several different graduate-course exams, and point me in the right direction on coding questions) which no average human with an IQ of 100 can do. High-IQ experts in a given field find them incompetent, but I expect those experts would also find the average human to be incompetent in their fields. Granted, they cost a lot to train, but I doubt the average human would produce comparable results with the same amount of training; and once trained, they can be easily and cheaply copied.
I think the next steps are to add modules with other functions to them, and make training happen (more) continuously.
Alex SL 05.01.25 at 10:50 pm
Mike Huben,
Well, that is just it. I had a very long bsky discussion with an genAI-promoter who insisted “The objective reality is that they’re used in healthcare and finance and education and legal and various other industries. I don’t know what exactly you were doing but they’re highly accurate for many tasks.” I already made them angry by writing that it always fails for me, personally, so I did not want to offend further by giving the logical response: are LLMs highly accurate for many tasks, or are the users who find them highly accurate the ones who are so ignorant and incompetent that they do not notice the inaccuracy? In other words, I can totally imagine many people using such a bad tool – if they are deep in Dunning-Kruger territory. And there are many such people, unfortunately.
Okay, maybe it is the former, maybe it works better outside of biology. But all I can go by is my own lived experience, and LLMs suck so badly that I find it extremely difficult to believe they suddenly work well when applied to law. I mean, I once didn’t find an answer to a coding problem on Stackoverflow, so in desperation I tried an LLM. The resulting code chunk ran without error but did not do what I asked for. Maybe I am singularly unlucky, but it always fails me, every time.
I also listened in on two events promoting genAI just yesterday, and the experience was jarring. The first was opened like a very cliche salesman repeating the terms “revolutionise” and “supercharge” over and over without specifics. The second speaker discussed a concrete use case, albeit again at a very high and abstract level, and then said, I showed this new system to a subject matter expert in my team, and he says the output is “reasonable”. Okay… that didn’t sound revolutionary to me. At the second event, one speaker said how amazing it is to use genAI to summarise a meeting recording and then spent the rest of his talk outlining how he had to enter people’s names himself because the AI garbled them all up, and he had to do so much checking and correcting that I felt it would be easier for me to write down notes myself during the meeting (which is what I do, of course).
This is so bizarre. I feel like I am watching a car demonstration where they cannot get the car to move at all, and then it spontaneously explodes, and everybody, even those hurt by the explosion, subsequently says what an amazing car that was and they want to buy one of those and it will change everything for the better. Are we experiencing some kind of mass delusion?
hix 05.02.25 at 1:24 am
Chatgpt keeps surprising me in both directions. My recent attempt to use deep research for a rathe unsusual question “estimate the replacement costs of some German theme parks”, was quite catastrophic, for example. The estimate for the replacement cost of 4 star plus luxury hotels per room were made based on an article about the costs of very basic accommodation in another park per bed 10 years ago. The rest was no better. So chatgpt is bad at answering very specific questions with no good public answer, and you need not be an expert in the field to spot it is doing a bad job when such errors happen. Does not mean it is bad at other more or also less generic things with a public answer. Psychiatric diagnosis look scarily accurate, for example. Chatgpt also sometimes does maths the 1+2 level wrong, and still it usually solves quite complex text based University finance assignments correct.
Feric Jagger 05.02.25 at 7:28 pm
Sure, for the current time. The collapse of the current AI boom may come, I suppose. The current AI boom. But unless AI hits some wall that it can’t get past — and I can’t see why it would — it’s going, in time, to be able to do any and all symbol-manipulation type tasks better and cheaper than any of us can. “Time” may be ten years, or 20, or 50. But these are rounding errors in the scheme of things. Just because nobody wants to believe this doesn’t make it unlikely. It could turn out well for humanity. But it won’t.
MisterMr 05.02.25 at 9:21 pm
@Peter Dorman 8
“But there has to be more, doesn’t there? Wouldn’t such a massive misallocation of capital need a more complex explanation?”
It’s quite simple IMHO: there are some big tech firms that have very big profits, but they don’t have something profitable to invest in. Plus they know that in informatics apparently it’s winner take all. So they bet all their money on the thing that looks like it might be the new biggie.
I mean, AI maybe won’t create that much profits, but it’s certainly more useful than crypto, so there is progress.
KT2 05.03.25 at 1:54 am
“AI and the fatfinger economy” by Cory Doctorow, rhyming with JQ.
JQ: “The likely collapse of the current AI boom will have some big economic repercussions. Most importantly and fortunately, it will kill (in fact, is already killing) the projections of massively expanded electricity demand from data centres. In combination with the failure of Tesla, it will bring down the market valuations of most of the “Magnificent Seven” tech stocks that now dominate the US share market. ”
Me?
Change “likely collapse’ to shakeout and slowdown, and light regulation following. Collapse would have to see for example, OpenAI’s touted “500m users” ‘sycophants’ – oops! – user base collapse.
“April 29, 2025
“Sycophancy in GPT-4o: What happened and what we’re doing about it” … “We designed ChatGPT’s default personality to reflect our mission and be useful, supportive, and respectful of different values and experience. However, each of these desirable qualities like attempting to be useful or supportive can have unintended side effects. And with 500 million people using ChatGPT each week, across every culture and context, a single default can’t capture every preference.
“How we’re addressing sycophancy…
https://openai.com/index/sycophancy-in-gpt-4o/
AI and the fatfinger economy”
by Cory Doctorow
CD: “Of course, bosses understand that their workers will be tempted to game this metric. [Bonus vs clicks] They want to distinguish between “real” clicks that lead to interest in a new video, and fake fatfinger clicks that you instantaneously regret. The easiest way to distinguish between these two types of click is to measure how long you watch the new show before clicking away.
“Of course, this is also entirely gameable: all the product manager has to do is take away the “back” button, so that an accidental click to a new video is extremelyhard to cancel. The five seconds you spend figuring out how to get back to your show are enough to count as a successful recommendation, and the product team is that much closer to a luxury ski vacation next Christmas.
“So this is why you keep invoking AI by accident, and why the AI that is so easy to invoke is so hard to dispel. Like a demon, a chatbot is much easier to summon than it is to rid yourself of.
“Google is an especially grievous offender here.
…
“This is an entirely material phenomenon. Google doesn’t necessarily believe that you will ever want to use AI, but they must convince investors that their AI offerings are “getting traction.” Google – like other tech companies – gets to invent metrics to prove this proposition, like “how many times did a user click on the AI button” and “how long did the user spend with the AI after clicking?” The fact that your entire “AI use” consisted of hunting for a way to get rid of the AI doesn’t matter – at least, not for the purposes of maintaining Google’s growth story.
“Goodhart’s Law holds that “When a measure becomes a target, it ceases to be a good measure.” For Google and other AI narrative-pushers, every measure is designed to be a target, a line that can be made to go up, as managers and product teams align to sell the company’s growth story, lest we all sell off the company’s shares.”
https://pluralistic.net/2025/05/02/kpis-off/#principal-agentic-ai-problem
Alex SL 05.03.25 at 8:52 am
oldster,
Yes, I have had exactly those conversations!
Feric Jagger,
Well, the current discourse tends to be about the current bubble: over-promises, ethical issues, and unprofitability of LLMs, specifically. Even if a very differently designed super-AI wipes out all research and management jobs in thirty years, the more imminent problem is the misery we may face when (not if) OpenAI goes bust and drags a bubble-inflated stock market with it.
But regarding future super-AI, I am very skeptical. On the one hand, we already have computers that can do numerous tasks at least faster and cheaper than we can do them. That is also what we really want and need: automate away the tedium so that we can focus on the interesting stuff that needs human decisions and creativity. We do not actually need general AI for anything, because it would only replicate what we already have between our ears.
So, the question is not whether AI can symbol-manipulate quickly, but whether AI can achieve reasoning, understanding, and creative thinking at human or super-human level and do so while being more economic than it is to raise and educate a human or several humans who do the same work. I find the first very plausible; I see no reason in principle why an electronic ‘brain’ shouldn’t be able to do what a biological brain can do, given the right architecture, only LLMs are very much not that.
The problem is the second part. I would be extremely surprised if AI does not at some point hit a wall of some kind, because physical reality is full of physical limits, diminishing returns, and trade-offs. You can be the fastest land animal or the strongest land animal, but you cannot be both at the same time. It is well possible that super-AI can be achieved but only at the cost of ten trillion dollars per year; or that super-AI can be achieved but only by constantly burning out and replacing an unmanageable number of chips; or that human-level AI can be achieved but only with an architecture that is about as slow-thinking and prone to cognitive bias as a human’s organic brain is, and so on. Assuming that it will simply happen without any trade-offs is like watching the first automobiles in the 19th century and being certain that, logically, by the year 2025 they will go at 12,000 kilometers per hour.
All I can say is that the assumptions of the singularitarian AI bros (which is not necessarily what you are talking about) run into the Fermi Paradox. I find the paradox trivially explainable under the assumption that god-like singularity AI is impossible: organic beings cannot survive interstellar travel, and that is that. But their assumption means that clearly, some other species should have created a technological singularity 300 million of years ago, and the resulting AI drones should have colonised this planet 298 million years ago.
J-D 05.03.25 at 8:55 am
What if it’s 100, or 500, or 10,000 years? Would you still call that a rounding error in the scheme of things? If we actually knew for certain what AI will be doing twenty years from now, then it would make sense to be doing some planning now with that future in mind; but it’s not worth doing any planning now for a future that isn’t going to arrive for 10,000 years. So a prediction–even an accurate prediction–that AI will achieve some level ‘in time’ is of no practical value if there is no indication of the order of magnitude of the time required.
steven t johnson 05.03.25 at 8:03 pm
If I google something, an AI summary pops up on top. So far, my impression is that it’s very much like someone automated wikipedia, convinced that the conventional is the Good, the Wise and the Beautiful.
But…on another board, someone said they put a long post of mine through ChatGPT to translate my post. To my eyes it sounded like my high school English teacher had rewritten it. It seemed surprisingly accurate but everything was ground smooth like a pebble in a rock tumbler.
marcel proust 05.03.25 at 9:40 pm
steven t johnson @18: …but everything was ground smooth like a pebble in a rock tumbler.
For your readers, this may well be a feature, not a bug ;)
You should share the original and the digested versions of that long post. We can get an authoritative answer by submitting both to ChatGPT and asking which one sounds better. For a second opinion (perhaps less biased) opinion, we can solicit Claude’s answer as well.
Edward Gregson 05.05.25 at 5:51 am
Alex SL @16
Obviously nobody knows if general AI is possible, practical, etc. It could happen tomorrow, in 10,000 years or never. But if you accept the materialist notion that the brain is a machine that performs operations on its inputs and memory to produce outputs, and that some other assemblage of matter that is constructible by humans now or sometime in the future could replicate those operations, then it seems obvious why someone might bet, in the absence of evidence to the contrary, that it could be more economical than a human.
The AI would presumably have internal workings/state/memory legible to its builders enough to be constructible by them, so if necessary the effort of raising it can be avoided by simply forking or copying one that has already been raised. Similarly if no other avenue of improvement is available, then improving the AI is at worst a matter of improving the speed of the hardware it runs on, which again must be human-legible if it was built in the first place. And the AI has the advantage that it occupies the design space of “things that perform thought-like operations and that can be assembled out of matter” and not the vastly more limited design space of “things that perform thought-like operations and that assemble themselves out of matter if the right swamp gets hit by lightning (or whatever) and you wait a billion years.” It seems like anticipating such strengths for a hypothetical designed brain is just applying the Copernican Principle to the human one.
Alex SL 05.05.25 at 10:48 am
Edward Gregson,
This is all highly speculative, so one may beg to differ about where the burden of evidence lies. Still, a couple of responses.
As a materialist, I share the notion that there is no supernatural spirit to the brain. Whether the brain is fully described as performing operations on inputs and memory is another question. It is well possible that an architecture that replicates what is desirable about human cognition necessarily replicates some of the downsides of the human brain, be it fragility or cognitive biases; the trade-offs that I mentioned. It would be nice to still have a cake after having eaten it, but such is life.
It is well possible that AI isn’t going to be more economical than humans for general intelligence tasks. Again, automating busywork, no problem. But even current LLMs, which are nowhere near human versatility, understanding, world model, and reasoning, do not currently look like a profitable business, and claims that they can become profitable rest on hopes that training and inference will somehow get much cheaper or that users are willing to pay tens of thousands of dollars per year for a license for a “PhD level” agentic system, which is, it turns out, the salary of a PhD level human who can also, in addition to ‘symbol manipulation’, do the kind of tasks that require eyes, legs, and hands.
One central problem here is that inference doesn’t scale. Many other applications get cheaper per user as the number of users increases. The computation cost for inference, however, increases in a linear fashion as the number of users increases, although there are then, admittedly, at least per-user savings for training. This will be the same for future AI. More to the point, I see no reason to assume that, if LLMs are currently burning billions of dollars per year, hypothetical, much more complex future iterations of AI ten or thirty years from now will suddenly be much cheaper to run; that sounds like asking for a miracle. Maybe at that level of complexity, biological neurons are simply more economical than electronic chips, just like metallic nanotech robots are a silly idea because at that scale, metal cogs don’t behave the same way they do at macro scale, and enzymes already do exactly what those nanobots were envisioned for. And if we try to replicate that efficiency, we may end up above again and buy in the downsides of biological neurons, like not being able to copy them and occasional brain tumors.
It seems very likely that at the level of complexity required to surpass human cognition, the electronic mind will not be legible to its developers. Even now LLMs aren’t built, they are trained without the developer really understanding what is happening in detail in every virtual neuron. This will get worse as complexity increases. Still, as long as it is all encoded as weights in a model file, that file can be copied. The question is then if that architecture can ever achieve more than just statistical guesses (and can ever be economical), or if a different approach is needed that will be even less legible and maybe cannot easily be copied. Who knows – speculation.
Improving speed only improves speed. That is of some value, but it doesn’t get us to understanding, a proper world model, and reasoning. And here too will be physical limits. Quantum computing is the great hope in that regard, I assume, but so far it is, shall we say, aspirational?
Finally, a key misunderstanding of singularitarians is that the super-AI that they anticipate will merely have to think very fast and very cleverly and thereby solve all manner of social, political, or scientific problems. That isn’t how anything works. The solutions to social and political problems are already known, we merely lack the political will or wisdom to implement them. What is the AI going to say except “stop burning fossil fuels” and “sorry, Mars isn’t habitable, your planet is all you got”?
Scientific problems do not get solved by thinking very fast and cleverly but by conducting empirical field observations and experiments, and therefore scientific progress will ultimately happen at the speed of building labs, reactors, observatories, and glasshouses, at the speed of pipetting and sequencing, at the speed of growing and screening GMO cultivars, and at the speed of designing and testing prototypes. The hypothetical super-AI would very quickly and cleverly develop a hypothesis, and then it would ask for the funds to build or be given access to the equipment or infrastructure needed to conduct the experiment that confirms the hypothesis, competing with funding requests for other priorities.
The singularitarians’ hope is that the AI will do magic, but that is not a serious proposition. A lot of the AI discussion is magical thinking.
MisterMr 05.05.25 at 12:47 pm
About the “general AI” thing, here is my point:
We humans have a brain that works in a certain way, can learn a lot of things, but of this a lot we dislike, many of the things we learn turn out to be wrong, etc., because we learn through trial and error.
We do however value some things that we learn, like e.g. being good at math, or being a good cook, which are the results of the trial and error.
I think there is a name for those two parts of intelligence but now I can’t find it, let’s call them “basic intelligence” (the trial and error part, that even a newborn already has) and “learnt intelligence”, the fact that we can understand more things thanks to our pre-existing knowledge.
A “general intelligence” thing is, by definition, trying to replicate the “basic intelligence” part, but somehow people seem to think it will be awesome in the “results” part (learnt intelligence), so it will be very good at math, or a good cook, etc..
This is IMHO a big category error.
Furthermore, when we speak of our basic intelligence, the way we learn things is influenced by many things such as desires and emotions (because it is a trial and error thing, so there has to be a way to judge the results of each trial, which implies a valutation system). While it is probably possible to replicate this stuff artificially, it wouldn’t necessarily lead to results we like/are interested in.
But the idea of a true AI is very romantic, so people prefer not to understand that the differences between ChatGPT and a true AI are really basilar.
Also, I partly blame Turing: his concept “intelligence as computational ability” expressed in the Turing Machine idea was very foundational for informatics, but sucks if you try to define “true AI” that way.
Tm 05.05.25 at 12:47 pm
Peter 8: “I think the most glaring weakness of existing AI, again in all its flavors, is its inability to assess its own risk of error. It’s this meta level that, up to now, humans can address but not machines.”
To be fair, most humans can’t, and many of those who fall in that category are big fans of LLM output.
“Wouldn’t such a massive misallocation of capital need a more complex explanation?”
I keep wondering what’s happening in the financial markets. Meta and Google are still actually highly profitable companies but Tesla is valuated at absolutely ridiculous levels despite a profit drop of 70% and no realistic chance of regaining the lost ground. I think we simply have to reject the hypothesis that capitalism is about rational profit maximization. The current crop of capitalist investors is comparable to the rationality of medieval princes hiring alchemists to turn lead into gold, except at a much different scale.
Tm 05.05.25 at 2:08 pm
It’s increasingly clear how poorly equipped we ordinary humans are to predict what horrible damage genAI is going to cause – which very likely will more than outweigh anything positive (like easier access to foreign language texts) that might come from them.
“Self-styled prophets are claiming they have ‘awakened’ chatbots and accessed the secrets of the universe through ChatGPT”
https://www.rollingstone.com/culture/culture-features/ai-spiritual-delusions-destroying-human-relationships-1235330175/
Gar Lipow 05.05.25 at 4:18 pm
A take specifically on Large Language Model’s rather than other types of AI:
https://softwarecrisis.dev/letters/llmentalist/ The author makes a case that when it seems to be working it is an illusion, that essentially it is recreating the mentalist tricks of cold readings, the same methods by which “psychics” hook their believers.
somebody who remembers that each and every one of these ai guys firmly believes your average black person has an iq of 48 05.05.25 at 8:42 pm
There won’t be “general intelligence” because it doesn’t exist, and thinking about it for five minutes tells you why: intelligence is socially determined, and as society and its values and needs shift, intelligence changes as well. The average computer boy, a vicious racialist seething with grievances against women who wouldn’t even say hi to him when he stood directly in their way and shouted, is easily impressed by being told to put glue on his pizza, or code that doesn’t work properly, specifically because it’s on a computer screen. If it was told to him by a human he’d consider them mentally ill and if it the human was black he would call the cops to have them shot, and if it was a woman he would…well, best not to elaborate. Remembering this critical element – that nobody working to make computers “artificially intelligent” thinks black people can be “intelligent” at all – helps you understand why they keep hitting their egg shaped heads against the wall the way they do, and why another 400 billion dollars will be poured down the drain chasing this dream. They have essentially infinite money, why should they ever stop trying the same thing that hasn’t worked so far?
Tm 05.06.25 at 3:46 pm
Very interesting 25 and this reminds me of a point Martin Gardner used to make often: that intelligent, higly educated, scientifically minded people are often easy to deceive by “psychic” con men because they think they are too intelligent to fall for a con, and because they don’t know the tricks, they don’t know the methods of manipulation, they don’t understand how easy the mind can be deceived.
steven t johnson 05.06.25 at 3:47 pm
Given how poorly natural intelligence is understood, it seems premature to expect much from artificial intelligence.
Human intelligence is not a mind=program running on brain=hardware. The brain is interpenetrated with the rest of the body, primarily via the central nervous system. But the brain’s cranial nerves connect pretty directly to the outside environment. I gather that some of the processing for vision takes place in the retina. Is that body or brain, much less mind?
As I understand it, a graph of encephalization (proportion of brain to body mass) against longevity shows a positive correlation. This to me suggests that one function of the brain is not to think but regulate the body, with positive effects for the organism. Drawing a distinction between the brain and the body in this respect is like declaring that the thermostat is not part of the heating system?
Part of that regulation is locomotion. The human sensorium seems to my introspection to be very much about a point of view to guide movement. I suppose you could think that this display is equivalent to the tag on a mall map marked “You Are Here.” Maybe that’s why sensorium doesn’t function normally when you are asleep?
What these simple-minded questions point to is, what is the point of copying sensorium for a computer that doesn’t even move? AI would not just be Artificial, it would be Alien, no? The mind is not the ghost in the machine, I believe. Thus AI would just be the ghost. I think of most chatter about AI as a program for putting souls in bottles. The upload my mind into a computer types are pretty upfront about that. The thing I want to know is, why does anyone want to create a person in a computer, where they will be effectively blind/deaf/mute quadriplegics? The only answer I can come up with is, because whatever thinking a computer does may be valuable precisely because it is not human. Different perspectives offer insights, different tests of reality if you wish.
marcel proust@19 Fortunately for me, this observation misses the mark, as I obviously don’t have readers!
steven t johnson 05.06.25 at 4:06 pm
marcel proust@19 By a coincidence, after commenting earlier today, I ran across this: https://www.richardcarrier.info/archives/34471
It is not the Carrier post that is relevant here, it is the example in the comment by Al Cannistraro, which reported asking an AI, Perplexity Pro in research mode, to comment on Carrier’s post. Judge for yourself?
JimV 05.06.25 at 7:54 pm
“Furthermore, when we speak of our basic intelligence, the way we learn things is influenced by many things such as desires and emotions (because it is a trial and error thing, so there has to be a way to judge the results of each trial, which implies a valutation system). While it is probably possible to replicate this stuff artificially, it wouldn’t necessarily lead to results we like/are interested in.”
Seeing this, I felt the need to comment on it from my perspective: the AI’s people are talking about are based on neural networks, which are automated trial-and-error systems. All the goals and evaluation procedures are part of the code, designed and programmed by humans. Examples of interesting results are AlphaGo (beat the then world Go champion in a tournament including a move that none of the watching experts had seen before and of which the (now former) champion said, “When I saw that move, I knew I would lose the tournament”), AlphaFold, AlphaTensor, and many other useful applications. LLM’s may be the least useful applications because the goals and validation systems aren’t well-defined yet. They can probably be useful as talking encyclopedias and translators but as general intelligences they are at the Model T stage, of which people used to say, “Get a horse.”
engels 05.07.25 at 11:43 am
Kagi, a search engine which will, if presented with a question, produced a detailed answer with links to references, most of which are similar to those I would have found on an extensive Google search, avoiding ads and promotions.
Enshittification in 3… 2… 1…
MisterMr 05.07.25 at 1:14 pm
@JimV 30
But if the goals and validation systems were well defined, it wouldn’t be an “all purpose” intelligence.
steven t johnson 05.07.25 at 3:56 pm
JimV@30 Isn’t a fairly reliable translator huge? In the long run I suspect it can change the texture of everyday life as dramatically as satellite communications. In our world for many decades there’s been the feeling that every language needs its own flag (and army and currency.) What happens when your language just needs a phone app?
engels 05.08.25 at 1:19 am
What happens when your language just needs a phone app?
As a resident of a country without linguistic borders with the US my guess would be “Americans end up running everything…”
John Quiggin 05.08.25 at 6:50 am
Engels @31 Kagi is paid-for. That’s not a guarantee against enshittification, but it provides a pretty strong incentive, given the dominance of Google which is free if you don’t count the ads and tracking. If Kagi stop being better than Google, their customers will stop paying.
abo 05.09.25 at 1:58 pm
I think Meta has explained how they will make AI profitable. Use it to create chat buddies with specific personalities to befriend people. Once the chat buddies are trusted, use them to advertise or recommend products which the user can buy, or to persuade on social or political issues, as covertly or blatantly as needed according to the character of the user. This is our future. It will make money, for those companies who move first and achieve first dominance.
steven t johnson 05.09.25 at 3:34 pm
engels@34 Once the sun never set on good prose. Tomorrow the sun will never rise again over truthful language. Yes, very sad.
engels 05.09.25 at 5:32 pm
Perhaps I should find it reassuring that when Mark Carney wanted an example of somewhere that was “never for sale” (to politely dissuade your President from “buying” Canada) he chose Buckingham Palace.
Jim Harrison 05.09.25 at 9:12 pm
LLMs basically automates what cranks and MAGA folks call doing your own research and has many of the same predictable consequences.
KT2 05.10.25 at 7:22 am
“Put high-quality search [simulated search] and accurate summarization together and you have the technology for” … model training sort of with Albaba’s ZeroSearch.
Another training cat, amongst the billions of US for profit brute force AI, soon to be fungible, model pigeons.
The key phrase “simulated search” is doing some very heavy lifting! Yet another arrow in the quiver of ai model development for a wider set of developers.
ZeroSearch “slashing training costs by 88 percent”… “Beyond cost savings, this technique gives developers more control over the training process. When using real search engines, the quality of returned documents is unpredictable. With simulated search, developers can precisely control what information the AI sees during training.”
8 authors. Tongyi Lab, Alibaba Group.
“Alibaba’s ‘ZeroSearch’ lets AI learn to google itself — slashing training costs by 88 percent”
Michael Nuñez
May 8, 2025
…
“Outperforming Google at a fraction of the cost
“In comprehensive experiments across seven question-answering datasets, ZeroSearch not only matched but often surpassed the performance of models trained with real search engines. Remarkably, a 7B-parameter retrieval moduleachieved performance comparable to Google Search, while a 14B-parameter module even outperformed it.
“The cost savings are substantial. According to the researchers’ analysis, training with approximately 64,000 search queries using Google Search via SerpAPI would cost about $586.70, while using a 14B-parameter simulation LLM on four A100 GPUs costs only $70.80 — an 88% reduction.
…
https://venturebeat.com/ai/alibabas-zerosearch-lets-ai-learn-to-google-itself-slashing-training-costs-by-88-percent/
Github code & model
“ZeroSearch: Incentivize the Search Capability of LLMs without Searching
7 May 2025
…
“Extensive experiments demonstrate that ZeroSearch effectively incentivizes the search capabilities of LLMs using a 3B LLM as the retrieval module. Remarkably, a 7B retrieval module achieves comparable performance to the real search engine, while a 14B retrieval module even surpasses it. Furthermore, it generalizes well across both base and instruction-tuned models of various parameter sizes and is compatible with a wide range of RL algorithms.”
https://alibaba-nlp.github.io/ZeroSearch/
Paper:
https://arxiv.org/pdf/2505.04588
Comments on this entry are closed.