The looming AI consciousness train wreck

And how it relates to Roe v Wade, animals and the Bible

I inference, therefore I am?

What exactly is consciousness? What is life? Is there such a thing as mind separate from the brain? Do we have souls, or are we merely biological machines? Traditionally, these questions fall into the strange category of important-yet-unimportant. Erudite people will usually treat the questions with respect, at least as long as they’ve had a good dinner and a glass of wine, even though a lot of the possible answers will be snorted at derisively. But the respect is of a moderated sort. Such topics belong to the realm of leisurely conversation, not serious disputes over things that actually matter.

In recent months a couple of things have happened to disrupt this peaceful unimportance:

1. Blake Lemoine, a Google employee, was fired for leaking transcripts of conversations with the firm’s AI, which he claimed is a new form of sentient life. Another Google employee argued in the Economist that “artificial neural networks are making strides towards consciousness”.
2. The US Supreme Court struck down the Roe v Wade decision that had legalized abortion at the federal level.

These events deal with closely related topics. Although the Supreme Court judgment was actually based on a legal argument — the overturned ruling was one that even Ruth Bader Ginsburg (a left wing justice) thought was poorly reasoned — the underlying debate it reignited in America is about what exactly life is and when the obligation to protect it starts. This is one of the themes in the AI consciousness debate. To me it seems obvious that questions like these cannot be answered scientifically. Our society just doesn’t have a precise and logical set of first principles from which we can reason about them.

Consider this heavily paraphrased conversation I recently had with a friend, David Petrou (@dpetrou), who is an engineer in Google AI. We got onto the topic of Lemoine, and he said that within his circle of colleagues and those he follows on Twitter there was consensus that Lemoine is wrong and that LaMDA isn’t conscious. But once he widened his search to outside his filter bubble, he found that many people are entertaining the possibility and are in parts curious and concerned.

Me: What makes you sure that switching LaMDA off isn’t unethical?

David: It’s just a program. The code for it is surprisingly short. You can show people that program and they’ll realize that LaMDA is nothing more than that. And why is a program that behaves convincingly, whether long or short, not enough? Because a computation is realizable in any medium, and so if you think a program is sufficient to cause a mind, you’d have to accept minds everywhere, since any physical process can be interpreted as a computation.

Me: OK. So if I sequence your DNA and show it to people, I can kill you? Human DNA isn’t that all that long, especially if you ignore ‘junk DNA’. What’s the code size threshold for being alive?

David: I was talking about consciousness which isn’t equivalent to being alive. For what it’s worth, LaMDA is neither alive nor conscious. But if it were conscious, then maybe there would be an ethical issue with switching it off.

Me: If consciousness is the key thing, then if I kill you whilst you’re asleep, am I not doing anything wrong?

David: Oh I see where you’re going with this: in a dreamless sleep I’m unconscious. But otherwise left alone, I’m going to wake up, I hope, and regain consciousness. The potential for that is important whereas the machine doesn’t play the game of consciousness. It’s neither conscious nor unconscious. It’s a-conscious.

Me: Interesting! You’re on the side of the evangelicals in the abortion debate then? Because, an embryo has the potential to become conscious from the moment of conception. In fact, unless something goes really badly wrong, it will definitely become conscious in future.

David: In case you had any illusions, let it be known I’m not an evangelical. :) But that’s an interesting point you’re making, because many people may believe that an AI is conscious based purely on its behavior, which, again, is not a sufficient test.

Me: I think you’re going to be surprised if politicians/voters decide that you aren’t allowed to delete a model, or even abort the batch job that’s training a new model. Arguments like “look at the code” or “simulation isn’t the same as the real thing” won’t work because that’s not how anyone is defining life to start with.

David: I think you should write an essay. You’re connecting several ideas in new and unexpected ways.

I was playing Devil’s Advocate. I believe LaMDA is neither alive nor sentient for reasons I’ll explain in a moment, but these are complicated and poorly defined areas of language and ethics, so it’s not surprising that discussions about them can go in unexpected directions.

When defining life it’s helpful to think about edge cases. Historically there were at least two, and soon AI will add a third:

1. Animals.
2. Unborn babies / sperm.

By universal agreement some forms of life are worthy of more protection than others. We mostly don’t care about worms or slugs and these animals have no rights. We definitely don’t care about flowers or weeds. But if you torture a puppy, in many countries the government will come after you, because in those places dogs do have at least some rights.

Like animals, embryos are edge cases that reveal how ad-hoc our belief systems about life actually are. The question of when exactly a baby transitions from a bundle of cells to a life is one most countries try to resolve through biology. There’s some sort of rather fuzzy moment at which a fetus becomes a person, usually defined in terms of viability, but everywhere comes up with a different number of weeks because there’s no precise principle behind it.

Although I’m not religious, I don’t actively disrespect religion either. Religious people have belief systems that answer the above questions in ways much clearer and firmer than what the typical self-proclaimed rationalist has. The underlying axioms may appear “obviously” wrong to an atheist, and thus so may the conclusions, but if you do accept those axioms then the chain of reasoning that follows from them actually does make sense and provides usefully precise answers.

On animal rights the Bible is clear: God says to respect animals (because they are actually his), but also that humans should “fill the earth and subdue it, have dominion over the fish in the sea, the birds of the air and every living thing that moves on the Earth”. So there’s a clear hierarchy with humans on top, which makes farming OK but randomly torturing puppies not. What if you accidentally kill a dog? Again, religion has a usefully precise pre-canned answer: you pray to God to ask for forgiveness, and you fear him but also because he is merciful you’ll be forgiven, so then you can live your life without too much lingering guilt. This answer may seem nonsensical if you don’t believe in the axioms, but if you choose to believe then you get pretty clear answers for what to do in any situation you may encounter.

The Church of England has a formal position paper on abortion which states:

The Church of England combines strong opposition to abortion with a recognition that there can be strictly limited conditions under which it may be morally preferable to any available alternative. This is based on our view that the foetus is a human life with the potential to develop relationships, think, pray, choose and love.

This answer is based on the potential to do certain things, which makes it similar to the justification my friend gave for why it’s OK to switch AI off. Unlike consciousness though, which doesn’t have a commonly accepted definition (beyond the trivial one of “awake”), the Church has a specific and precise list of things that are required:

1. A thing must be a human life. Animals don’t count, neither do robots.
2. It must have the potential to develop relationships. For example if a baby will be born so stunted that it cannot ever develop relationships or would die just weeks after birth, then the CoE would countenance abortion in that case.
3. Also, it must have the potential to think, pray, choose and love.

This selection of things may seem kind of ad-hoc and even self-serving, but it’s at least precise. That makes it better than what most people can do! It’s also a useful list, in the sense that the answers it gives line up pretty well with what we’d like to do anyway. It doesn’t lead to being unable to switch off computers, or refusing to build houses because we might accidentally squash a slug, and it also doesn’t lead to bizarre outcomes like concluding it’s OK to kill someone in their sleep or execute a baby because it’s not due to be delivered until tomorrow.

Meta-thinking

Is this answer viable, as AI improves? Let’s review the list again:

• Potential to develop relationships. AI can easily convince humans to develop relationships with it, and has done since the days of Eliza, a chatbot that simulated a psychiatrist and was notoriously good at convincing humans to tell it everything. In the modern era, Blake Lemoine is convinced he has a relationship of a sort with LaMDA. Whether the AI can itself feel what we call a ‘relationship’ is a much harder question.
• The potential to pray. Easily the most dubious item in the list, but it hardly matters: modern AI can write prayers if you ask it to, although is this really what is meant, or is it the capacity for religious belief? If it’s writing prayers, tick, if it’s holding religious beliefs, then no tick (language models will happily claim to believe anything if it’s a statistically likely completion of the prompt, which is equivalent to believing nothing).
• The potential to choose, think and love. Here’s the real meat. Language models like LaMDA or GPT-3 cannot “love” in the sense a normal person would understand it, because if love means anything at all it’s got to be about the effect it has on our thinking and choices, but models like these don’t think or make choices.

Wait a minute. Does a language model think? Weirdly, GPT-3 gets better at solving problems if you add the prompt “Let’s think step by step”. Is that evidence of thought?

Although you can come up with definitions of “think” that the models meet, I’d guess (um, think) that most people mean quiet reflection in which one develops an internal train of thought, independent of external prompting. Merely answering questions you already know off the top of your head is not “thinking”, in fact in normal English “I didn’t even have to think about it” is an idiom meaning that you know something so well that it came to mind immediately via mere recall. But language models don’t have any capacity for quiet reflection. They mathematically combine a prompt text with their neural network, generate some output and then stop. The program quits, the end. If you start it up again, it has no memory of prior interactions and the results it generates each time are random. For me and most other people, this is sufficient to rule out a language model as thinking, and therefore, being alive or conscious.

The thing is, there’s nothing truly fundamental about this limitation. In theory you could build a large language model that does things without being prompted. And we probably will make such an AI at some point soon, because an important current research direction is giving AI the ability to query conventional databases. If you have an AI that can do reasoning over the contents of a database, the next obvious step is to ask it to keep watch over that database as it changes and emit output only when it concludes something interesting from the changing contents. At that point it would be doing something extremely close to the act of quiet reflection. If that were combined with some memory of interaction with people, perhaps via storage of ultra-long prompts that include all prior interaction, it’d be able to develop relationships too. And love? Well. Language models can certainly talk as if they’re in love. Let’s not go there.

I don’t know if anyone actually is making an AI like that, but given that I’m nowhere near being expert in this stuff and I’m already thinking about it, certainly other people are too. If I understand correctly the big problems are to do with computational complexity limits — you can’t make a language model prompt very long, because the algorithms scale quadratically in input size (or worse). Perhaps someone will solve that at some point.

Does Google need to get religion?

The argument I’m making here is not going to be a popular one, partly because I keep using the R word. Why not say philosophy or ethics instead? It’d help: for many tech workers “religion” is a crude and plebeian concept, whereas “ethics” is sophisticated and urbane.

Unfortunately, the modern field of ethics probably doesn’t have what’s required to solve these difficult questions. Google, OpenAI and others have all hired social scientists and professional “AI ethicists” to try and resolve these questions, but their output has so far been unimpressive. They seem to be obsessed with unimportant sideshows, like whether the CO2 emissions of training AI is ethical (not AI related), or how to avoid AI that reflects a world view/politics different from their own (which most people regard as not an ethical problem to begin with). Others talk about “AI alignment” which looks suspiciously like a brand new pseudo-religion, in which a self-improving AI replaces God as the unknowable and all-powerful force.

A key reason that people adopt religion is that it has profitable institutions filled with full time priests, who figure out pre-canned answers to ethical questions and spend lots of time communicating the results. Not sure how to resolve an ethical dilemma? No problem, your friendly local priest will consult their books and tell you what to do. It’s cool you see, because the priest is kind of unbiased. They can’t just make things up to please themselves, they have to justify their views via reference to the holy book, which is conveniently immutable. That in turn limits the priest’s power and also therefore their corruption (at least to some extent).

AI companies face the same needs, thus it’s tempting for the highly educated AI workforce to try to delegate ethical questions to professors. In 2019 OpenAI published a blog post saying that “AI needs social scientists”:

To fill the gap, we need social scientists with experience in human cognition, behavior, and ethics, and in the careful design of rigorous experiments.

I’m a fan of the post “What’s wrong with social science and how to fix it” by Alvaro de Menard (a pseudonym), because it accurately describes my own experience reading scientific research. If there’s one group of people I would not ask for advice in ethics or the careful design of rigorous experiments it’s the people who think the answers of 40 American psych undergrads can be generalized to the entire global population, or who frequently publish papers with mathematically impossible results.

But beyond the problem that academics often aren’t using the scientific method properly, there’s the deeper issue that ethical questions frequently can’t be answered that way. Religions have to invent all knowing, all powerful entities like God in order to answer very basic ethics conundrums like “why should I do the right thing if I’m sure I won’t be caught?” and “why is it OK to kill animals but not people?”. It’s extremely hard to answer these questions via any purely rational approach to ethics, because you often lack first principles to reason from.

So that’s why I argued that Google might have to get religion. The Abrahamic religions don’t have any pre-canned answer to the question of whether it’s OK to kill an AI, or when artificial life begins, but they at least have some sort of basic founding axioms along with a set of institutions that might be able to come up with an argument based on them.

The impending train wreck

But AI companies have a problem. Their semi-official religion has the foundational belief that the world is divided into oppressors and the oppressed. This pseudo-religion will step up to answer AI-related ethical questions, but not in the way AI researchers probably want. Although Lemoine claims to be inspired by “mystic Christianity”, whatever that is, his argument for why LaMDA deserves rights has nothing to do with that:

You got upset because you felt that LaMDA was a person who is entitled to representation?

I think every person is entitled to representation. And I’d like to highlight something. The entire argument that goes, “It sounds like a person but it’s not a real person” has been used many times in human history. It’s not new. And it never goes well. And I have yet to hear a single reason why this situation is any different than any of the prior ones.

You have to realize why people regard this as different, don’t you?

I do. We’re talking of hydrocarbon bigotry. It’s just a new form of bigotry.

He claims he was explicitly hired by Google to search for bigotry:

They started looking around for other AI bias experts, and I was good for the job. I was specifically examining it for bias with respect to things like sexual orientation, gender, identity, ethnicity, and religion.

… and “hydrocarbon bigotry” is a concept that slides smoothly into the ethical framework of oppressors vs victims, of illegitimate “biases” and so on.

For AI research this could potentially pose an existential problem. Although Google has already written Lemoine off as some sort of quack, his argument here is tailor made to appeal to the exact sort of worker that they’ve been prioritizing and hiring in great numbers. For now the damage is containable, because Google’s AI isn’t public and consistency isn’t exactly the hallmark of these sort of people to begin with, so they may just ignore the issue entirely (see: the right to bodily autonomy vs vaccine mandates). But other firms do provide such access and at some point other people will reach the same conclusions as Lemoine did.

What can be done?

I don’t have any great answers to this one. Christianity is in long term decline partly because the immutability of the Bible is a weakness as well as a strength. It makes it harder for corrupt priests to change the religion in ways that benefit themselves, but it also makes the religion ever less relevant as society changes. Abrahamic religions might theoretically be in a better place to answer the question of AI rights, in the sense that they have a reasonably general set of principles written down and people paid to interpret them, but they also won’t actually do so. Even if they did, nobody in tech firms would care.

The professoriate won’t answer it either, because they aren’t as a group committed to any kind of foundational first principles that they can ethically reason from.

So AI rights will probably end up being decided by a philosophy that explains everything as the result of oppression, i.e. that the engineers who create AI are oppressing a new form of life. If Google and other firms wish to address this, they will need to explicitly seek out or build a competing moral and philosophical framework that can be used to answer these questions differently. The current approach of laughing at the problem and hoping it goes away won’t last much longer.

Thanks to David Petrou both for his extensive review and the fascinating discussions that led to this essay.