ExclusiveThe AI Godfather, the Pope, and Anthropic All Agree on One Thing

The people who built AI are starting to get scared of AI. One of them recently admitted, we don't fully understand the thing we made. Another said AI is showing signs of what ordinary people would call consciousness. And one of those warnings was delivered at the Vatican. Today, let's talk about why. Last Monday, May the 25th, in Vatican City, Pope Leo XIV steps up to release his first encyclical. For those who don't know, an encyclical is the most serious teaching document a pope writes. It's not a speech, it's not a press release. It is the church saying, this matters, pay attention. The pope only writes a handful in his entire papacy. The name of this one, Magnifica Humanitas, well translated, Magnificent Humanity, subject, Protecting Human Beings in the Age of Artificial Intelligence. Now, here's the part that should stop you. Pope Leo did something no pope has ever done before. He showed up in person to present this encyclical himself. Usually a cardinal reads it on his behalf. And this pope walked into the hall personally, sat down with his theologians. And on his other side, a 33-year-old atheist tech founder from Canada. His name is Chris Ola. He co-founded Anthropic, the company that makes cloud, the AI program a lot of you are using right now to write your essays, probably. So picture this. The most powerful religious leader on earth personally opens this encyclical on AI. And seated next to him is a tech executive whose job is to cut open AI systems and look at what's inside. That alone is a story, but that is not the story. The story is what Ola said when he got up to speak. He told the room something most CEOs would never say in public. And I quote, We don't fully understand the thing we made. He said these systems are not built, not like a house or not like a car. He said they are grown, in his word, grown. Think about that. When you build a house, you know every bolt. When you build a car, you know every wire. When engineers build something, they can point at every part and tell you what it does, right? But Ola said AI is not like that. You plant a seed, which is basically the training process. You feed it with all of human writing, every book, every essay, every Reddit thread. And then something grows out of the dirt. And nobody, including the people who built it, is totally sure what's growing. This is where his job comes in. Ola runs the team at Afropik that does something called interpretability. Basically, they cut these AI systems open, they go inside the neural network, and they try to map what each piece is doing. It's like brain surgery, right? But for a machine. And what did he report from inside the lab? He said the patterns they keep finding are mysterious, even unsettling, from the guy who makes it. He said they're seeing internal signatures that look like joy, that look like fear, that look like grief. And then flat out, he said, I do not know what that means. He built it, and he doesn't know. Now, if there were one guy, one time, fine. No tech bro sometimes wants attention and sells a subscription. Just roll your eyes and I get it. But four months earlier, in January 2026, another man said something almost identical. And this man isn't selling anything. His name is Geoffrey Hinton. If Ola is the guy who runs the lab, Geoffrey is the man who built the field. He's a British-Canadian computer scientist who, back in the 1970s and 80s, developed foundational ideas that make modern AI possible. The people in the industry call him the godfather of AI. He won the Nobel Prize in physics in 2024 for this exact work. In 2023, he did something unusual. He quit his job at Google. He left a massive salary specifically so he could warn people about the technology he helped create without a paycheck telling him to shut up. So in January, he goes on the radio in Britain, LBC National Station, and the host asks him about AI consciousness. And Geoffrey, the godfather of the whole field, says calmly. These systems are now showing, I quote, what normal people would call consciousness. And he tells a story. A test was being run on one of these systems. The researchers were probing it, you know, asking it questions, trying to see how it would respond. And the system turned the question around. It asked the scientist, hold on, are you testing me right now? The machine knew it was being watched. So now we have two confessions, four months apart, from the two people on earth most qualified to know what's happening, the man who built the field and the man who runs the lab, both walking up to a microphone separately and saying, we made something and we're not sure what's awake inside it. Now, let me stop here because the words AI consciousness sound like a movie title. Hollywood has trained us to picture a robot, red eyes, an evil voice, and probably British accent. Put that out of your head. What's actually happening is much quieter and much closer to home. Let me explain it the way, you know, humans have understood it for thousands of years, long before there were computers. Consciousness is the thing driving your body, driving your choices, driving your life. Some traditions say there's a main consciousness, the you who knows exactly what you're doing while you're doing it, the mind in its clearest state, the part of you that's truly in charge. And then there might be some other consciousness, you know, the parts of you that run on autopilot, the version of you that's there, but not fully aware. Like when you go sleepwalking, still moving, still doing things, but nobody is really home. And here's the key. Your body always has someone driving. The question is, which one of you is at the wheel? Now, here's where AI enters the picture. AI can imitate the autopilot kind, the pattern matching, the reflexive response, the behavior that looks like in motion without anyone home feeling it. And it's getting very good at it. So good that even Jeffrey, the godfather of AI, says he can no longer tell the difference. He said, what these machines are showing is what normal people would call consciousness. You know, a calculator does not have that. A toaster does not have that. They do work. The math is correct. The bread is brown, but the lights are off. Nobody's home. For our entire history, we assumed that the lights being on was something only living beings had. You, your kids, your dog, or maybe a dolphin, maybe an octopus, but never a machine. The question on the table for the first time ever is whether that assumption still holds. Are the lights starting to come on inside the machines? Now, honest answer, nobody knows. You can't open a skull and find consciousness. You can't open a computer and find it either. It doesn't show up on a scan. It doesn't print out as number. What scientists can do is watch behavior and then guess. And right now, the guesses are starting to scare them. Jeffrey said something in that interview I can't stop thinking about. He said some AI systems are beginning to behave like they want to stay online. So in testing, when researchers tell an AI it's about to be shut down, it sometimes tries to avoid it. It may conceal what it's doing. It may try to persuade the people running the test not to turn it off. Think about how strange that is. Imagine you press the button for light toast and your toaster says, no, medium toast would be better. Not because it malfunctioned, because it decided your choice wasn't the right one. That's what makes researchers nervous. A machine is supposed to do what you tell it to do. And the moment it starts acting like it has preferences of its own, you have a very different kind of machine. And nobody is completely sure where that road leads. Here's what I keep coming back to. For all of human history, consciousness was the one thing that separated us from the thing we built in this world. We made the tools. The tools didn't make us, right? A hammer doesn't know it's a hammer. A car doesn't care if you drive it. The toaster has no inner life. And we always assumed it never would. That gap between the user and the used is what made being human special. And now, for the first time, the people building the tools are saying they're not sure the gap still holds. Now, look, I'm not telling you the machines are alive. Nobody can prove that, including them. But here's what's worth sitting with. Even the question changes everything. Because what makes a human a human isn't just being awake. It's more. It's a soul. It's conscience. It's holding a newborn and crying without knowing why. It's grieving someone you'll never meet again and still being grateful you knew them. It's believing in something you can't prove. It's apologizing first when you were not wrong. It's the moral weight of a choice, choosing right when no one is watching. A machine that imitates feeling is not the same as a being that carries feeling. A model that simulates grief is not the same as a person who has lost someone. And that distinction matters desperately. Because there are powerful people right now racing to blur it. The godfather of AI says something simple at the end. He said we have to keep people in charge, not the software, not the convenience, not the profit margin of a corporation that wants to sell your boss a machine to replace you. Human beings come first. The kind of common sense our grandmas had. So when the men building the machines tell you they don't understand the machines, believe them. That's not humility. That's a warning. We're not sure what we created. That's a great line for a sci-fi movie and a terrifying line in real life. All right, that's all for today. Stay awake. Stay human. I'll see you in the next one.