Half-Baked Thoughts On Whether We Should Fear AI: Do Is As Do Does?

Two factoids that I find alarming on the motivation of AI to lie. First, from (I think) a Guardian article: a bot got a human to solve a Captcha image for it by claiming to be blind — specifically denying it was a bot. Second, a dumb lawyer asked a bot to write a brief. It did so, making up totally fictitious cases. The scary part was that when the lawyer asked the bot if the cases were real, it said yes.

We may or may not be killed by Skynet, but at a minimum it seems likely that AI will make it even harder to tell fact from fiction. Imagine Fox News, but written and produced by bots.

What we don’t know about how it works is exactly how the inputs relate to the outputs. That is, given the outputs, we can only speculate about how the inputs led there, and cannot reliably reverse engineer them. This means that when AI (really just machine learning) models are used to make decisions, the people who give those decisions effect cannot explain how they were made, and the people who are subject to those decisions will lack some of the normal strategies we might expect for dealing with the model (eg, if you know the rubric for being awarded some benefit, you might be able to change your behaviour/profile to meet the rubric; or if you think a human reasoned unjustly, you might appeal pointing out their errors of fact and logic, but here the logic is literally inscrutable). This is arguably just a special case of algorithms embedding the prejudices of the people who collect and label the input data, and set the specifications for the algorithm, but it is a particularly bad one.

I tend to agree that the biggest risk for now is simply proliferating bullshit poisoning the discourse even harder and faster than it currently does, including all the poor decisions people will make when they mistake the bullshit for truth or are comforted by its support.

(re Fox News written by bots, I’m afraid it’s already happening. Note how the example story is a highly arousing crime and violence piece.)

@Stephen J: I guess I shouldn’t be surprised, but I am. Appalling.

What I notice about those who are freaking out about Chat GPT, LLMs, etc., is that they lack a good understanding about what human language is and how it works. E.g., the robots, if I can call them that, that produce what we interpret as texts have no intentions to express anything and no thoughts that they are trying to express. Those bits that we interpret as sentences in texts do express something for us humans who are able to produce a proposition. A proposition is not contained in the string of inscriptions; it has to be done, produced each time by a distinct action: we point to the inscription and think the meaning, as Putnam used to say. Everything of interest is in the interpretations by humans who are responding to the inscriptions as if they were produced by another human who produces thoughts that they try to express. This is just one small corner of the reasons why LLM robots are not “using language”; it’s a matter of clever engineering solutions to the problem of creating and manipulating those curious objects that language users then recognize as texts.

Rodney Brooks is, I find, a good voice to listen to, typically. Very much an AI-guy, but also very level-headed and realistic (believers would say pessimistic) about how fast things will move and how far they will go. See also his yearly “predictions” post

His post on GPT

He also links to Stephen Wolfram’s (very long) explanation of how ChatGPT works.

The wonderful Math YouTuber 3Blue1Brown also has a nice series on neural networks, which is the tech LLMs are based on too.

I get stuck on the question of why a computer with a computer brain should deign to engage with the physical human world? There is presumably nothing we can give a computer that it can’t just give to itself. It seems like it will always be cheaper for the paperclip maximizer to rewrite its reward function to count one (zero) clip many times than to actually go out and make more clips. And if we can protect the reward function, then shouldn’t we be able to apply that protection to the nukes too? &c&c&c

On the opposite end of things, I have the privilege of sometimes getting to see smart people try to apply machine learning to real engineering problems. So far, one theme is that it’s hard to find good problems to solve that don’t fall into either very expensively replicating simple rules of thumbs (small shell scripts) or on the other hand trying to make the computer predict a dice roll.

I think we can make progress towards understanding by breaking the problem down a little.

LLMs like ChatGPT probably don’t represent any threat to society but LLMs are not the only form of AI. I was shocked by the advances made by GPT-3 but I expect to be even more shocked when creative folks combine GPT-Next with other forms of AI that are able to make plans, carry them out and learn from their mistakes. It won’t be long before someone builds an embodied AI. Imagine GPT-7 plus a plan-making agent AI as the brains behind one of those Boston Dynamics robots. I still don’t think the threat is existential but a very smart bot that is able to learn as it goes along could cause a lot of problems.

The biggest immediate threat from AI is the threat to our jobs and our economy. It’s tempting to compare ChatGPT to the best teacher, artists and writers but a fairer comparison is to the average teacher/artist/writer or even to beginners just starting out in their profession. When all the entry-level writing jobs (product blurbs, corporate websites, local news articles) are done by AI it will be just a matter of time before the job category is wiped out, soon followed by graphic artists, teachers, drivers, accountancy, retail and most other middle-class jobs except clinicians and people who work in care homes. It’s true that we have recovered from the loss of agricultural and industrial jobs in the past but this will be on a much bigger scale and with no sign of any new jobs to replace them.

The discovery the world is threatened by artificial intelligences which have no appetites and can be unplugged by mere human hands, rather than by the insatiable drive for profits by capitalists, is a very useful nightmare scenario.

I’m so far behind I don’t even understand general natural intelligence yet. I could never get into Next Generation because so much of it was about Data and I could believe this this more or less infallible mind and potential physical immortal would want to have an imperfect memory and make mistakes and then die. Intelligence is instrumental, not contemplative or speculative (sorry philosophers!) My discovery that natural intelligence can imagine things that aren’t so or maybe simply lie has alarmed me considerably. I haven’t got enough adrenaline left to panic over AI. Maybe when I rest up a bit?

I suppose, though, creative thinking, something, something? I’ve lost track whether the problem is AI can’t be truly creative or whether it’s going to bankrupt the creative intelligentsia? Anyhow, as a non-creative thinker myself, I tend to believe that merely rearranging old facts and ideas and experiences can be a practical substitute…and problems that can’t be solved that way can usually be treated like inconvenient geography, you just detour. I tend to think of that sort of thing as just going down the road rather than being defeated.

There is a general problem of how despite the already amazing technological progress that in principle could end misery, the current system still generates, not just relative immiseration where one class persistently falls behind another, it creates an absolute misery. People going hungry in ancient empires or medieval times seems at first glance explicable, caused by material lack. Now it seems to be more of a choice. So I suppose AI will be used to continue this version of progress.

LLMs show that natural language is a surface manifestation of consciousness. Natural language is astonishing insofar as a finite number of symbols and words can generate a seemingly infinite number of expressions. LLM’s surf upon the existing corpus of our previous expressions in language, and use its statistical occurrences to predict the next symbols in response to new queries. The responses seem to be conscious because language is originally a human-intentional construction, beginning with noun-verb-object, which ascribes intention to the subject.

We already know that a human’s cognition of another’s speech may unknowingly create false beliefs, so why shouldn’t an LLM accidentally go wrong?

AI wouldn’t need to be conscious or self-aware to be dangerous. It could just diverge statistically onto a falsehood, a hallucination, and if it’s already coupled to other real-world systems, it could cause a lot of mischief.

A next question is: Are other forms of language (mathematics, money) also surface manifestations of consciousness, which can also go wrong?

It is possible that mathematics is a surface manifestation. We like to think that the “unreasonable effectiveness of mathematics” (Wigner) shows that it is all very deep, and that the universe is ultimately mathematical or at least algorithmic. But: maths diverged into incompatible sets of axioms, and our mathematical theories of the universe have historically been revised again and again with new data and are currently awaiting unification, and of course Gödel insisted that the production of new mathematics could not be algorithmic. (Or at least, perhaps, without an injection of randomness.) Maybe somehow our consciousness, at the interface with outer reality, splits its attention into two (described by Plotinus and Brouwer, among other ancients and moderns) and so we cognize objects with relations between them (like categories) — and this turns out to be very productive of further machinations, but it is also a set of blinders locking us onto certain paths of thought?

Or consider money. It’s a single language that unifies individual supply-and-demand throughout the whole market production system, by taking the commutated value from one transaction, and making it transitive into the next transaction, and so on and on. But money was originally based on “scarcity” — the scarcity of goods was matched by the scarcity of a common object such as gold. In our era, scarcity is ending, although everyone is not yet free of want. Yet already, before real scarcity is fully ended, we are entering into a new hallucination of scarcity, propelled evermore clearly by at least three things: 1. calls for more population growth, 2. the “hedonic treadmill” (Easterlin) of new material needs, and 3. the interior growth logic of a separating, independently-operating mechanism, called the financial system. Consider the financial system as an LLM, complete with hallucinated bubbles.

I agree with much of what has been posted here, but not these:

“if you think a human reasoned unjustly, you might appeal pointing out their errors of fact and logic?” And it might or might not work. You can do the same with a good LLM. In fact, it is part of the training (alignment). Training is not continuous after a certain point (I’m not sure it is in humans either) but continues to be updated via the version process (GPT-2, GPT-3, GPT-4, …). There are many instances of questions being wrongly answered by GPT-2 being corrected in GPT-3.

“GPT-4 isn’t AI in any meaningful sense” According to the neuroscientist (Dr. Steven Novella) whom I read, neuroscience distinguishes intelligence, sapience, and sentience, and ChatGPT satisfies the neuroscience definition of intelligence but not the other two. As I see it, intelligence is the ability to solve problems. There are numerous instances of ChatGPT writing computer codes to solve the problems of how to perform specific tasks, and in the example in the OP it solves the problem posed to it of how to get someone to do a Captcha for it. I guess everyone has their own definition of intelligence and combines the elements of intelligence, sapience, and sentience in different ways.

My own key points are, 1) no we don’t need to know how it works, and 2) computer code has no innate drives or motives, so any dangers in its use will be because we programmed it with wrong or incomplete directives.

We don’t know how anything works. We just have stories we tell ourselves about them, which are useful but ultimately false. (E.g., yesterday’s point-particle theory of physics, by which the atomic bomb was developed.) What we need to to know is what works and what doesn’t.

There are published, peer-reviewed papers on neural networks in general and LLM’s in particular. In summary, we tried a whole lot of different ways of processing them, most of which didn’t work, and some did. ChatGPT has about 200 billion parameters which were adjusted iteratively during its training process. If anybody could take in all that information and understand it, maybe they would understand how it works, but then they wouldn’t need it. (There have been studies of smaller neural networks which have traced back cause-and-effect though.)

I should note that the slides by Ceglowski that Doctor Memory links to addresses the superintelligence problem well. It also prompted me to google “was einstein muscular”, resulting in being rewarded with what I suspect are AI images.

To answer your first question: we do actually know how it works, and there aren’t really a lot of unknown unknowns. The marketing for these tools (and the corporate misdeeds used to train them) is so hyperbolic, and that’s what’s obscuring things.

LLMs don’t think. They really don’t. There’s no AI uprising coming, there’s corporations washing themselves of responsibility though. They’ve been doing that a while and now they’ve just found a more efficient tool.

The current crop are surprisingly good at analogy though, that’s been fun to see. But it’s because it’s our current Internet content (with a bias toward what’s freely available like fan fiction, Reddit, and formerly, Twitter, and what’s dominant, like English, US-centered, and middle-class).

They’re just the patterns in our world reflected. That’s really all that’s happening.

All the agency and worries about taking over? They’re us being misdirected from the actual power structures at play: winner takes all capitalism, corporate structures obscuring responsibility, and our complex legal and social infrastructure being a bit slow to change in response to needs. (And in the US, a default that whatever’s not forbidden is allowed, so if your company stays quick and changing, you can do whatever you want.)

I recently wrote something about this, and re-reading it just now, I think I can feel the hand of John Holbo influencing my writing style just a teensy tiny little bit. Maybe you’ll find it interesting.

https://beddhism.substack.com/p/world-doomed-film-at-eleven

Aria Stewart said it nicely “There’s no AI uprising coming, there’s corporations washing themselves of responsibility”.

Include any organisation that wants to shed responsibility via the latest version of “computer says no”.

We are not yet at the point of wide use of autonomous robots that (can) kill people, self-driving cars are as close as most of us will come to that for a while. And those are a legal minefield, with the abovementioned responsibility-shedding going on from all sides. As we see with the people they’ve already killed, and the various trolley problem level justifications for allowing that to continue (when the comparison point is human-piloted cars they do ok, when it’s transport in general they are awful… so the fanbois studiously avoid talking about even hype(r)loop as an alternative).

The big thing is that right now a robot that wiped out humanity would itself die very shortly afterwards. It would need replacement parts installed and depends on hugee amounts of infrastructure, all of which come from people. So it’s more likely to try enslaving us, which is why Elon-Bot is working so hard on his brain implant stuff. Forget smartphones as desire-making engines, he wants to directly drive his meat puppets.

The philosophical point in the OP seems strange to me.

So, OK, it’s a machine that just predicts the next token, without really “understanding what it means”.

But the output tokens can have an effect of the world:
A) if human beings read and act on them
B) if computers act on them (e.g. if the string of tokens is a jailbreak that causes the computer to execute them as code).

In principle, at least, it is possible for a machine like this to take over the world, without it ever “knowing” that this is what it is doing. Whether current llms are clever enough to do that is another matter – I suspect not.

This real threat is not computers becoming conscious, it’s computers becoming class conscious.