Masters of Disruption: How the Gamer Generation Built the Future [21]
How John Carmack is building the future of AI with the help of 8-bit Sonic.
This post is part of a longform project I’m serializing exclusively in my newsletter, Disruptor. It’s a follow-up to my first book, Masters of Doom: How Two Guys Built an Empire and Transformed Pop Culture, and it’s called Masters of Disruption: How the Gamer Generation Built the Future. To follow along, please subscribe to Disruptor and spread the word. Thanks!
I’m serializing my recent interview with John Carmack here in my newsletter, Disruptor. To read from the beginning click here.
David Kushner: With your work on Artificial General Intelligence now, what’s a typical workday for you like?
John Carmack: Most of it is still just learning: reading papers, going through books, doing experiments. I do have a framework. I actually call it my AI stage, which is my code that plays videos and old video games in emulation.
David Kushner: Which old games are you using?
John Carmack: Actually, for no strongly grounded reasons, I chose the Sega Master System just because that was a game console that I never had. I like going in and playing the 8-bit Sonic [the Hedgehog] version. I could have picked any particular one. In a lot of AI research in reinforcement learning, they use the the early Atari games as their references. I think that's too crude. I wanted something a step up from that. There's a lot of things that people have done playing video games with AI, but in some ways they're less than it meets the eye. Google, just last year, had a big deal; they called it Agent57. It was the first AI that achieved super human performance on all 57 of the Atari games that are in this well-known benchmark. That seems great, and it was. But people don't realize that there are two important factors. One is that they retrain it independently for each game. It’s not like an AI learns to play one game, and you could just pop the cartridge in and it plays any of them like that. They train it separately for each game. So the Breakout AI can't play Pitfall! and vice versa.
I will consider it a sign of legitimate general intelligence, although not necessarily human-like or human-scale, when there's an AI that starts to play the game for its own artificial curiosity, exploration reasons. It can switch between hundreds of games and learns to play all of them on its own, and it does this in a human lifespan scale. That is at the cutting edge of artificial intelligence work right now. It's close to what we've been doing, but we aren't there yet. I think that that would be a significant step towards full purpose general intelligence.
While we have these companies spending billions of dollars a year on all of the AI research, the resources that someone at my scale can get a hold of is enough of a sandbox for fully general artificial intelligence to evolve in. I don't think you need the entire internet to do this because humans have a fairly limited view of the world. I want a clan of AI raised on PBS and a clan raised on BBC media and let them go ahead and interact, play multiplayer games together, and converse with each other with human vocal modalities. I think that's enough, and that doesn't cost that much. The Cloud resources are available. I think it will take millions of dollars of Cloud GPU usage, but not billions. The big companies are spending billions now. I think they're overspending, but hey, they've got the money. It's a smart bet, because if we achieve AGI, it's going to be worth trillions of dollars just in terms of all that value that can be created. A company on the scale of Google or Microsoft throwing a couple of billion dollars a year at a small chance of improving AGI absolutely makes sense, but I don't think it's necessary. I think that I can get by not having to spend billions of dollars on it.
“I've told Elon, ‘you should give your pigs a human vocal tract and let them learn how to say a few human words. That will be a super-dramatic demonstration of the animal intelligence.”
David Kushner: What sort of AI experiments are you doing now?
John Carmack: There's a directory with probably a hundred little experiments in it, where I just try out different techniques. Right now, I'm spending most of my time on reinforcement learning. Machine learning is usually divided into three pillars, like supervised learning, which is identifying the dog species. And that's almost a solved problem now. Unsupervised learning - which is like, ‘Okay, let's just watch a year of video and see what we can learn from things like that" - that's coming along well, that's the active topic of research. And then there's reinforcement learning, which is largely the game-playing and the agent-interaction side of things. The big breakout success for that was AlphaGo beating the world Go champion. To a lesser degree, there was also the Atari game-playing stuff. Those achievements haven't been industrially-useful yet. It is going to be more critical for making an AI that interacts like a person, because we are all interacting agents that get reinforced signals from things. Reinforcement learning is where I spent much more time the last two years going through the classic texts, reading. AI, in general, has taken off in the last maybe eight or nine years. We really do look at almost a takeoff point with AlexNet and ImageNet in 2012. A lot of things in unsupervised and reinforcement learning have happened in this relatively brief time span.
People aren't going to accept an artificial intelligence until they talk with it. In many ways I think we undervalue a lot of the animal intelligence just because we can't carry on a conversation with them. I've told Elon a couple of times, ‘you should just give your pigs a human vocal tract and let them learn how to say a few human words and eventually the monkeys. That will be a super-dramatic demonstration of the animal intelligence. There's a whole interesting psychological history about people trying to draw the clear, bright line of human intelligence, thinking that all animals are on the other side of it, and then discovering, no birds do this and other animals do that. It keeps getting pushed back to these tortured definitions to try to separate human intelligence from it. It mostly comes down to familiar communications modalities. I write conventional code for things like that.
Right before this meeting, I was more or less staring at a page of graph paper, writing down small notes about how I've got something Sonic. Sonic the Hedgehog 8-bit is what I'm currently using for my reinforcement learning tasks. I've got stuff that'll play through the first couple of levels of stuff using conventional reinforcement learning algorithms. But I see the limitations and how they're not going to get me to where I want to go. So, I'm still looking around for the right combination of techniques.
Interestingly, while everybody is focused on these hot new things from the last five to eight years, there are a couple things from the eighties and nineties, older technologies that I'm kind of fond of, that people are not paying much attention to right now. I think there's a pretty decent chance that there may be forgotten insights from decades past. With virtual reality was, people talked all about it in the early nineties and it didn't work out. Then it took 30 years for it to come back around. I have reason to believe that there may be some old gems for artificial intelligence research. We had people that are every bit as brilliant as today's researchers, they were just born in the wrong decade.
“I could imagine people getting really worked up about AI and it becoming a populous movement that has serious consequences.”
David Kushner: What about science fiction? Are there any particular science fiction works that are real beacons for you with AI?
John Carmack: I still read a lot. I'm a huge fan of The Expanse. I think that's amazingly good fiction. Certainly, at the back of my mind, I do consider the Dune: The Butlerian Jihad. I could imagine that happening, people getting really worked up about AI and it becoming a populous movement that has serious consequences. I have a friend that in deadly seriousness urged me like, "John Carmack, you got to stop talking about all this AI stuff, because there's a real chance that someone could come at you about this,” and basically Sarah Connor me in some way.
David Kushner: Have you ever heard from anyone like that?
John Carmack: Back in the Doom days, we did have a couple of crazy people that came up and at least harassed people around the office a little bit. But in general, I don't worry about a lot of that. Looking at my priors, I realize I'm generally an optimistic person that in many ways lives in a bubble around a lot of things. I try not to discount the possibility, but no, I'm still pretty out there in terms of just not being concerned about things like that. Maybe that'll change in five years or something. But I think people will be surprised how smoothly the transition to artificial general intelligence actually goes.
David Kushner: I think that optimism has always come through in your work, from working on games early on to cultivating the mod community. There were a lot of people railing against those decisions of yours but you remained bullish.
John Carmack: I am a hugely optimistic person. It kind of sucks because up until COVID, up until 2020, I could just trot out the statistics about it's like, "No, you're bitching about everything's gone to hell, but this is the best time the world has ever been near any way you want to measure it.” I hate that we've got to have that little asterisk on 2020 and 2021, but I think it's going to certainly recover and exceed it by quite a bit. I styled myself as a technological triumphalist and this is why I am pretty blasé about AGI risks. Because everything we've done, all of these tools, there's always been somebody worrying about it. You know that. There were probably people going on about cultivating fire back in the Paleolithic era. I'm not really that worried about malevolent AGI. The idea of an indifferent AGI could still be dangerous in various ways and people will have to start thinking about it. But I think the people thinking about it right now are just wasting their time. Very little good is coming out of all the smart people working on AI ethics and AGI alignment. It's just not at all clear to me that the work that they're doing now is even going to be relevant when we're trying to figure something out that we don't know what it's going to look like yet.
“I'm still really high on the odds that human welfare will be drastically better 10 years, 20 years, 30 years from now.”
David Kushner: What about climate change? Are you thinking about how AI help us in that regard?
John Carmack: In terms of climate change, I was tempted to go work on something with the Elon's funded carbon capture stuff, just because it's such a crunchy engineering problem it'd be fun to go work on something like that. I recognize that the climate is changing. I still do not see it as disastrous as that people that are saying humanity could be extinct in 30 years. That's just mind-numbingly out-of-touch with the reality of the way systems work for these different areas. In even the worst cases, it means that humanity is in Canada and Siberia and so what. We're not going to hit those worst cases. When it gets desperate, there's a lot of stuff that we can do, even if they are Band-Aid fixes like atmospheric seeding or sun shades and things like that. I'm a big proponent of being able to have a control that you can work on rather than trying to culturally change everything. I think that it's going to get hotter. There will always be people negatively affected by it. But, netting out, I'm still really high on the odds that human welfare across everything is going to be drastically better 10 years, 20 years, 30 years from now. You'll be able to point out specific things where it's where like, ‘oh, this is worse. This species is now extinct. This ecosystem here is now inhospitable to its traditional inhabitants, but overall there will be more counterbalancing factors for it.
Many years ago, I went through a bunch of the evidence. I read a bunch of the IPCC [The Intergovernmental Panel on Climate Change] reports and I was making a conscious decision: Do I want to steer my life to making this a big battle I want to be fighting? I chose not to, and I still chose not to. I at least gave some thought to that as I was picking this next phase that I'm going on. Because I still think that we're going to reap more benefits than we are going to take harm from things like that. I recognize I could be wrong, it could be really damn bad, but that's just not my current assessment.
