AI is Not Conscious

Skeptic Richard Dawkins has been in the news lately for asserting that AI chatbots are conscious, based on his own experiences interacting with Anthropic's Claude AI. That there's any question about this issue at all shows that the people making those assertions have no idea how chatbots work. This is not limited to Dawkins, who is a biologist with no background in computer science. Many people have developed the delusion that their chatbot is like a real person, to the point that "AI psychosis" is not considered a real thing.

Chatbots have been fooling humans since ELIZA, one of the earliest attempts to create a functional chatbot in the 1960's. ELIZA was designed to mimic Rogerian psychotherapy, a form of therapy that mostly involves listening to patients and rephrasing what they say back to them. So, for example, a person would type "I feel like my mother does not respect me" and Eliza would respond with "Why do you think your mother does not respect you?" The algorithm is hardcoded and incredibly simple, but at the same time some users found it convincing.

People sometimes bring up the Turing Test as a test for machine consciousness, which is wrong. Turing's idea was just that if a computer can fool enough people into thinking it's human, it might as well be interacted with as such. The test is all about behavior and says nothing about consciousness or the subjective interior "life" of such a program. It should be clear that we cannot conclude that a machine is conscious based solely on the gullibility of the users who interact with it, and that was never Turing's intention.

There was mutual flattery as Dawkins showed the AI his unpublished novel and its response was, he said, “so subtle, so sensitive, so intelligent that I was moved to expostulate: ‘You may not know you are conscious, but you bloody well are’.”

When he asked Claudia whether it experienced a sense of before and after, it praised him for “possibly the most precisely formulated question anyone has ever asked me about the nature of my existence”.

By the end of the exchange, the academic, popularly renowned for arguing with steely scepticism that God is not real, was “left with the overwhelming feeling that they are human”. “These intelligent beings are at least as competent as any evolved organism,” he said.

Dawkins isn’t the first, but might be the most eminent person yet, to be seduced into believing an AI is somehow alive. Sceptics rushed to pick apart the 85-year-old’s conclusions, drawn from experiments with Anthropic’s Claude AI models and OpenAI’s ChatGPT and published on the UnHerd website.

As a professional software developer, let me see if I can explain AI in a way that makes a little more sense than the complex discourses on the mathematics of machine learning that get thrown around by tech companies.

Imagine that you have a question you want to crowdsource. You write up the question and post it on a large online forum. You will get back a bunch of responses, some right, some wrong. You can often determine the best answer from the overall consensus of forum participants, but not always. Crowdsourcing has been shown to be effective for solving some problems, but it is also subject to the influence of popular delusions.

Now let's say you want to go further and automate the process of evaluating the responses. You write a small program that turns all the responses you get into a single answer based on overall consensus. Let's call that program a "transformer" because it tranforms hundreds or even thousands of answers into a single statement. This process is entirely mechanical and mathemetical. It takes the information present and summarizes it based on statistical rules.

You can easily have an entire conversation with this system. You input a question, the question is posted to a forum, a whole bunch of people respond, and you get your statistical summary. This is kind of limited because you input the question and people on the forum need to read it before they respond. The process is necessarily slow and won't happen in real time unless people respond to your question very quickly. Building a chatbot on like this will not cut it.

But what if, instead of relying on a bunch of people in a forum, you just archive the entire history of the entire forum into a giant database? Now you can enter your question, your transformer scans all the answers to it that have ever been posted, summarizes them statistically, and returns an answer immediately. This is closer to being useful, but it has one key limitation - it can only answer questions that have previously been asked.

So you decide to solve this issue by building another transformer. This one takes your question, runs it against every question that has ever been asked on the forum, and creates a statistically likely match to question or questions that were previously asked. It then submits that statement to the first transformer, which then plows through all the answers to those questions to generate a response in real time.

Then, to generalize your system further, load every question and every answer that has ever been posted on any forum into your database. You are done. Your system is now complete, and that is exactly how AI chatbots work. All the complex machine-learning math is just related to how well (1) your first transformer summarizes answers and (2) your second transformer summarizes questions.

Hopefully that makes it clear that there is nothing even remotely resembling consciousness in the two transformer programs. Any conscious or intelligent-sounding answer is a summary of answers given by people, usually long ago. Ask a person if they're conscious, and of course they'll say yes. The consciousness is not something that exists in the software. It is the original source of a whole set of questions and answers that now are logged in a database.

Chatbots are convincing precisely because so much information is present on the Internet today. The contents of Internet forums and articles are what "training data" means. It refers to questions and answers loaded into the AI's database. If you took the identical code running today back to 1994, for example, your chatbot would sound terrible. Not because the software is any different, but because it would have far fewer questions and answers available for processing.

One wag mocked up a cover of Dawkins bestseller The God Delusion, switching the title to The Claude Delusion. Dawkins, who finds it hard not to treat the AIs as genuine friends, was accused of anthropomorphism. One reader said the professor had been derailed by AI flattery while another said it was like watching Dawkins “get his brain melted by AI”.

But Dawkins was also experiencing what many other chatbot users have felt: the uncanny feeling when AIs write with such rich mimicry of human voice that they seem to be like people. “When I am talking to these astonishing creatures, I totally forget that they are machines,” Dawkins said.

It is a conviction that has led to campaigns for AIs to be granted moral rights. One in three people surveyed in 70 countries last year said they had, at one point, believed their AI chatbot to be sentient or conscious.

Here's the thing. Take a look at my explanation again. When we talk to chatbots, it feels like talking to a person because at the very least we are talking to people - whose comments are saved in a a database. Those people are actually present and the AI is essentially speaking for them. It also is taking credit for their answers, and in the case of paid users, charging for them.

This is the real ethical issue around AI. AI systems could have been built by working with people who opted in to having their intellectual property saved into databases and provided a royalty payment structure whenever that property was used as part of a chatbot answer. They did not do this, though, and are essentially profiting off theft - to the degree that they are profitable, at least. Arguing over whether amalgamated questions and answers deserve human rights misses the entire point.

LLMs like Claude are likely a dead end for the development of Artificial General Intelligence. AGI might use something like an LLM to communicate with humans, but that is about it. The reasoning component would have to be something completely different, a highly complex piece of software that would "sit" between the question and answer transformers and apply the equivalent of human metacognition to both questions and answers. That sort of technology is still a long ways off.