In the Next Great Transformation AI will not eliminate genuine expertise; rather it will make it more valuable

As the next great transformation advances, before long AI may well drive discovery and justification and, thereby, as Tao suggests transform different sciences. We may well have to get used to playing second fiddle to AI practices.

Something like this may happen.

Or, then again, it may not happen.

It seems an estimate of the odds might be useful, it anybody can produce one.

Tangential and trivial, but I don’t think the helicopter metaphor works very well; if nothing else, one might observe its initial flight pattern to develop hypotheses about its route to the destination which could then be followed up. Rather, it’s like an alien teleport machine; you set desired coordinates, step into it and disappear, reappearing somewhere else (which may or may not be where you wanted – how can you be sure?) having somehow travelled via goodness only knows.

One thing I worry about is simple lack of awareness that an AI could be wrong. Recognizing this requires being aware not only that gaps in training would create hallucinations as the system attempts to fill them from faultily generalized knowledge, and also as the system overgeneralizes to the point of ignoring counterevidence, but also that the input to the system needs not to be considered absolute and final complete Truth, because that’s something we humans don’t yet have either. It would require being aware that just funneling all the facts into a computer would be unlikely to create usable generalizations even if it found all the generalizations that actually exist. It would require being aware that the data being fed into the system is linguistic data and not “facts” in the sense empirical sciences use, and that there’s a difference between how those two types of knowledge are formed. (It would also require being aware that philosophy of AI that assumes a perfect AI might not be a practical guide to whether an actual AI should be trusted.)

It would be entirely possible for a consensus to form, among people with the right kinds of power and influence to enforce that consensus, that the AI doesn’t make mistakes, and what appear to be mistakes are actually some kind of user error, misinterpretation, willful resistance to the facts. It would be entirely possible for a consensus to form that even though AI is fallible, it’s better than the alternatives and that there are benefits to lying about its capacities. Opacity might be seen as acceptable in those circumstances, given the tradeoffs.

As a practical matter, however, how to get out of the gravitational field of a limited AI, after it’s been given absolute epistemic power, would remain as a potentially existential danger.

Cheez Whiz @ #8 characterizes the complete inability to report facts and truth by AI as a stumbling block. Very true, for a normal person, but the wealthy and powerful are not normal people, their minds do not operate in such a way. (it is left for an exercise for the reader how harvard and yale might react differently to this technology than, say, the university of california los angeles.) a critical feature of the technology is its passionate embrace by white supremacists and fascists, the most normal footsoldiers capital has on offer these days. they, like billionaires, fear the interiority that ordinary people have, and know that their power means they can get someone to tell them whatever they want, rendering the ordinary courtier’s praise of their genius (stupid) ideas meaningless and obedience to their beautiful (stupid) orders hollow. but the emperor doesnt need clothes, if grok will always say he’s dressed.

in fact, a certain degree of hallucination is desirable. after all, reality is just some stupid roadbump on your way to total sexual and racial domination. as you choke down your glue pizza and eat three rocks a day because the ai voice on your favorite grindset superpodcast orders you to, you smile, knowing SHE would never eat a pizza like this. “women don’t understand” the ai voice – female coded, of course – coos in your earbuds, then asks if you want to order more steroids to stop the vaccines – ten percent off with promo code “SAMALTMAN”.

It would also require being aware that philosophy of AI that assumes a perfect AI might not be a practical guide to whether an actual AI should be trusted.

There’s enough information available about how people do things and about how existing computer programs do things to conclude reliably that when computer programs (for example) play chess, or Go, they’re not doing it the way humans do it and that, likewise, when LLMs answer questions they’re not doing it the way humans do it. However, all the philosophical arguments I’ve ever seen that try to establish that it’s impossible in some absolute sense to program a computer to do things the way humans do them fall down because not enough is known about how humans do these things to draw such impossibility conclusions reliably. On the one hand, the limits on present understanding of how humans do things impede attempts to prove that computers can never be programmed to do things humans do in the same way that humans do them; on the other hand, those same limits are a reasonable basis for being dubious about the prospects. In short, it’s reasonable to suppose that it is in some sense possible that we’ll get there one day, but it’s equally reasonable to conclude that so far we’re nowhere close to it.

One thing I’ve been thinking about is the various attacks on LLMs, especially attacks via ‘poisoned’ training data, and how to distinguish those from all the other inputs we feed into the LLMs. How do we tell whether a given input will be good or bad?

I don’t think we know, and I don’t think we have a very good understanding of how we might find out. We’re more at the “noxious miasma” stage of AI-medicine than even the cellular life stage. There’s a lot of intuitive knowledge, but not a lot of systematic knowledge.

You see this most publicly is the various “AI made to do something insane” articles, but for me writing software every day the more practical one is… actually, “the AI doing something subtlyinsane” is more accurate. Rather than glue on pizza, it’s a complete re-write of a test such that it no longer functions and thus the new code that “passes the test” in fact does not.

@J-D 11
“However, all the philosophical arguments I’ve ever seen that try to establish that it’s impossible in some absolute sense to program a computer to do things the way humans do them fall down because not enough is known about how humans do these things to draw such impossibility conclusions reliably.”

I don’t think that it makes sense to say that it is absolutely impossible to program a computer to think exactly as a human, but this is a different argument from “present day computers think exactly as humans”, that obviously false.
There is the question on how much exactly as a human is enough, for example I don’t think anybody wants to program AIs to feel anxious or envious, however feelings do play a role in human thinking.
I think the confusion comes from the fact that people are trying to simulate only the “higher” functions of the human brain, but “intelligence” relies on the lower functions too.

Just some superficial reflections on OP and the thread(s):

I wonder about people who coin terms like “epistemic opacity,” how much time they have spent on the pursuit of trying to feel their way around the very opaque issue of how knowledge differs from unjustified (or inadequately-justified) belief. As phenomena that demand explanations go, things don’t come much more opaque than that. Perhaps if one finds oneself tempted to coin a neologism with the adjective “epistemic” one ought to examine closely exactly what one means.

OP cites famous mathematician observing that “AIs” (by which are meant LLMs) don’t judge their confidence reliably, and the the would “would appreciate more honest AIs.” Aye, indeed, Famous Mathematiker has pointed directly at the Holy Grail: a mechanism that would make a LLM STFU when its training data is too thin to allow it to guess accurately what a correct answer would look like. I personally remain certain that this capability will not arise spontaneously in any model that is trained the way LLMs have been trained so far. This is the 2nd time just today that I have observed in a comment that the most noteworthy shortcoming of the LLM approach is that it can never, under any circumstances (other than ad-hoc rules about specific questions put in after training) answer “I’m sorry, Dave. I don’t know the answer to that.”

In this thread are a lot of people who would probably like to be taken seriously, but whose insistence on referring to LLMs with epithets like “spicy autocorrect” militates against that.

During the last month, they have finally reached the level of interesting research assistance in my own field.

I am always puzzled by this, because whenever I try them, they fail me. A few months ago, I tried a presumably “agentic” system that was purpose-built for research, and its literature review function was like a clown car driven into a ditch that is also on fire. In the first attempt, it repeatedly used references to support statements that they did not actually support, and in the second, it got stuck down a rabbit hole of a small subset of the space it was meant to summarise, doing the equivalent of a student who is asked to provide a ten page overview over WW2 and then spends six pages on the sinking of the Bismarck.

The usual response is that I should shut up until I have tried the pro subscription of the currently most publicly hyped model (at the time of writing, Claude), which reminds me forcefully of religious apologists insisting that I read book after book after book written by Sophisticated Theologians and never accepting the legitimacy of being an atheist because there are still more books that could convince me to convert.

More to the point, the helicopter ride is a very bad idea for research. Science isn’t science if we cannot explain why we got a result, and science isn’t science if the results cannot be reproduced. This means that generative AI should not be part of any analysis workflow. And using it for writing manuscripts or grant proposals but then putting a human’s name onto the title page is quite simply fraud. What is more, until there are cheap robots doing all of the data collection, lab work, field work, experimentation, moving around of specimens, etc., even a hypothetical AGI will not be able to replace researchers and research technicians.

That being said, whether AI works reliably enough, or whether it can actually replace researchers, ultimately doesn’t matter. The world is full of a seemingly constantly growing number of people who do not care about aligning their worldview with reality or even about making it logically consistent, who in a year of record high temperatures point at snow and say “so much for global warming, eh?”, who take Musk seriously when he talks about Mars colonisation or hotels on the moon, who cheer war with countries they would not be able to find on a map. It is full of managers who want to hear only positive sound bites from staff whose jobs they are cutting, who they unnecessarily force back into the office, who they stuff into open plan offices despite all the studies showing those to be less productive and incubators of infectious diseases. It is full of people who take carbon capture seriously because they have seemingly never heard of the laws of thermodynamics, or who think that betting markets and deflationary currency are great ideas, or who argue that driving all of nature to extinction is fine because we can just GMO new plants and animals. The combination of self-confidence and not understanding how anything whatsoever in the universe works at all that is exhibited by most people is horrifying.

So, I am not worried that a probabilistic word generator can replace me, but conversely, I am not comforted by the idea of genuine expertise either. If I lose my job during the great AI transformation, it will not be because of that. It will be because too few people care about knowledge, expertise, and accuracy, and will accept LLM-generated word salad as good enough even where it objectively and demonstrably isn’t; they will just go lalala I can’t hear you at those who point that out.

if LLMs end up being a slightly significantly more sophisticated markup language,

To be useful the output has to be deterministic. Otherwise “what changed” turns into a nightmare of “everything”. I suspect similar problems apply to people who use LLMs to “proofread”, where they make a pile of innocuous changes while quietly negating a core argument. Or just relabelling a graph so it’s nonsense. Your post-proofreading step ends up being “pore over the whole document with a microscope”.

Here’s a small data point as to whether human thought processes have any commonality with neural networks:

The first neural network I learned about, in the early 1990’s, was trained to recognize hand-written decimal digits, from 0 to 9. It had this property: to whatever you scribbled in one of its input boxes, it would assign a numerical digit. It would never indicate that it wasn’t sure or couldn’t respond.

My eyesight is getting bad, and I recently squinted at a two-digit number on my computer screen. My optical neurons showed me “89”. Then I removed my glasses (I’m near-sighted) and looked again with my nose almost on the screen. This time, with equal confidence, I saw “55”.

So it seems to me my optical neurons made their best guess the first time, wrongly, and presented the hallucination to me with no indication of a lack of confidence.

I understand that brain neurons and their synapses are about 1000 times more complex than the capability of a single node in a neural network, so it takes huge numbers of them to do anything interesting, and it will probably take combinations of many of them with different tasks. Which so far has not been the case, I think, AlphaGo had only two sets according to the paper. I don’t how how many an LLM has but I think they lack a separate executive function to estimate things like confidence.

I am always puzzled by this, because whenever I try them, they fail me. A few months ago, I tried a presumably “agentic” system that was purpose-built for research, and its literature review function was like a clown car driven into a ditch that is also on fire.

Bret Devereaux at A Collection Of Unmitigated Pedantry reported asking a Large Language Model to describe the relationship between two specified books (neither of which I have read myself, so I can’t comment from that kind of knowledge). The answer supplied gave brief (and, apparently, flawed) summaries of the two books and then concluded with a characteristically generalised bromide about how the topic was similar for the two books but they approached it in different ways.

Using as a research tool not a Large Language Model but an ordinary Web search engine easily turned up a review of the later published book which revealed the key fact which the Large Language Model had failed to report, that it (the later book) was presented in large part as a rebuttal to the earlier book.

The Large Language Model’s ability to provide (even faulty) summaries of the books showed that its training data set contained information about them, presumably reviews; but it was unable to pick out from that information the point which was really salient.

J-D,

Yes, the key here is that LLMs do not actually understand anything. They extrude words, and often they guess right what words we expected, and then we are also biased towards assuming that something that can produce language is sentient, so we fill in the rest.

It helps if the outputs of an LLM can be easily verified. Two good examples are coding and literature/references. Run the code, see if it does what we expect, if not try again. An “agent” can do that, and in the end the code does what the user wanted, and the question is only how many tokens it cost. In research, the “agent” can check if a paper that is mentioned in the generated text can actually be found in a literature database. What is much more difficult to check is if the LLM understood correctly what that actually existing paper was saying.

Kaleberg,

I don’t know how worried I have to be about model collapse, but it has occurred to me long ago that AI engineers are uniquely relaxed about that possibility. No other field that uses any kind of model is like them. For example, when the possibility of running out of training data was first discussed, many AI CEOs and their hangers-on said they would just train the next models on the outputs of the previous generation. If we translate that idea into the context of, say, ecological or evolutionary models, it is immediately revealed as completely deranged.

What now, I fit a model to the data, then I make some predictions, then I fit the model on my predictions, I make more predictions, then I fit the model again on its predictions… if you suggested that to colleagues, they would start asking questions to the effect of whether you have bumped your head or how many figures you think they are showing to you. But in LLM land, that idea is just Tuesday.

“What now, I fit a model to the data, then I make some predictions, then I fit the model on my predictions, I make more predictions, then I fit the model again on its predictions…”

Yes, the idea seems to be that you fit a model to the data, and that gives you truths (not predictions), so naturally feeding those truths back to the same system would produce even better truths! There’s a failure here which should be obvious, and apparently a lot of people don’t notice. We should fix that.

Zamfir,

What I am talking about is e.g. not having enough videos to train video generating models and then claiming that they can be trained on videos they created themselves, or the same for novels. Sure, if humans invest enormous amounts of time to create fresh data, like your colleague times hundred million other people, then there are fresh data. But the claim I am addressing was that this can all be automated except for some annotation of the training data.

The problem is not what if the LLMs of 2030 have been trained on a carefully quality controlled, guaranteed accurate corpus of data, but what if the LLMs of 2030 have been trained on social media that are by then 50% LLM-generated propaganda, scams, and shrimp jesuses, on hundreds of thousands of LLM-generated eBooks that were mass-produced by a handful of scammers at a rate of five per day in the hope that each would be bought by about eight or ten people by accident, and on websites that are low-effort LLM-generated “information” existing entirely to sell something.

I have suspicions regarding Botanical Realm, for example; it has alleged “information” on most plant species I have ever googled for in my line of work. It features tens of thousands of pages, an effort that would require an enormously large team or community effort to write, maintain, update, and quality-check if it was done by humans. The text on every page I have checked so far screams LLM-generated:

“[Plant species] ([Latin name]) is a fascinating species within the [plant family], known for its striking appearance and resilient nature. This herbaceous perennial plant has garnered interest from botanists, ecologists, and gardeners alike due to its unique characteristics and ecological role. In this article, we will delve into the various aspects of this plant, from its physical traits to its importance in our ecosystems.”

By the way, this is from the page about a globally invasive weed that no gardener would find attractive, and it isn’t perennial either.

And then there is a shop tab where you can buy some indoor potting mix, which doesn’t feel like it could fund an operation large enough to write the website. This was almost certainly extruded by an LLM with no quality control whatsoever. Will people who train a later iteration of LLMs necessarily realise that? Will they do so for all the other garbage that is out there?