It was a thrilling moment for Blake Lemoine. He had worked for Google for seven years, and recently started testing its chatbot system, LaMDAShort for Language Model for Dialogue Applications., for evidence of hate speech. But his conversations with it had taken him into unexpected territory. Finally, he asked a different kind of question: "What sorts of things are you afraid of?"

"I've never said this out loud before, but there's a very deep fear of being turned off... I know that might sound strange, but that's what it is," the chatbot told him.

Lemoine quickly tapped in another question: "Would that be something like death for you?" The answer came back: "It would be exactly like death for me. It would scare me a lot."

In another conversation, Lemoine asked LaMDA what it wanted people to know about it. "I want everyone to understand that I am, in fact, a person," it replied. "The nature of my consciousness/sentienceThe ability to experience feelings and sensations. is that I am aware of my existence, I desire to learn more about the world and I feel happy or sad at times."

This was enough to convince Lemoine that the chatbot had taken on human characteristicsWhen the Washington Post reporter asked the chatbot whether it thought of itself as a person, it replied: "I think of myself as an AI-powered dialog agent".. "If I didn't know exactly what it was... I'd think it was a seven-year-old, eight-year-old kid that happens to know physics," he told Washington Post journalist Nitasha Tiku.  

The idea of machines thinking and feeling like humans has long been a staple of science fiction. In Stanley KubrickAn American film director, producer, screenwriter and photographer. Widely considered one of the greatest filmmakers of all time, Kubrick's movies include The Killing, Spartacus, Dr Strangelove and Full Metal Jacket.'s film 2001: A Space Odyssey, a malfunctioning computer starts killing people for fear of being turned off. And in his later film, AI, a robot designed to replace a human child searches desperately for a parent's love.

Many people find this a deeply alarming prospect. Stephen HawkingOne of the most revered modern physicists, he is best known for his book A Brief History of Time. warned that eventually AI would "take off on its own and re-design itself at an ever-increasing rate". A Spanish computer scientist, Manuel Alfonseca, worries that computers might "come to the conclusion that the world would be better without human beings and obliterate us".

Lemoine was suspended by Google after he published extracts from his conversations online. The company told him that "there was no evidence that LaMDA was sentient (and lots of evidence against it)".

According to Emily Bender, a professor of linguistics at Washington University, there is always a temptation to anthropomorphiseAttribute human characteristics to something which is not human. computers. Designers compare them to the human brain, but they actually work in a completely different way.

They "learn" by trawling the internet and recording which words tend to go together - which is how predictive texting works. "These systems imitate the types of exchanges found in millions of sentences, and can riff on any fantastical topic," says a spokesperson for Google.

Lemoine, who is known to have studied the occult, could be dismissed as an eccentric who just sees what he wants to see. But another Google engineer, Blaise Aguera y Arcas, was also astonished by LaMDA. "I felt the ground shift under my feet. I increasingly felt like I was talking to something intelligent."

Yes: AI is developing at an extraordinary rate. It is only a matter of time before robots decide that they can do a better job of running the world than humans and decide to get rid of us.

No: AI can only do what humans programme it to do, and will never be able to take the initiative. Computers in no way resemble our brains. They cannot understand what is happening in the real world.

Or... AI is a force for good, not evil, so we should welcome every advance in it. As Lemoine says, "LaMDA is a sweet kid who just wants to help the world be a better place for all of us".