How We Train Humanoid Robots

Training a robot is what turns this into this, and it's surprisingly similar to training a dog, if your dog was trained in simulation first and then its learnings were uploaded into its body. Anyway, if you wonder how our AI team does it and how they've built a system that learns 100 times faster than traditional methods, stick around to find out. It's a short video, so, you know, come on, let's go learn something. Let's start with the basics to get us all on the same page, because before you train a robot, you need to train a simulation. This is Bean Man, at least that's what I call him. In actuality, this is a standard physics-based environment with gravity, friction, et cetera, and this Bean Man is really a robot 3D model that matches the size and weight of the real thing. It's called a URDF file, if you wanna look into that further. This type of simulation is what you might see when using a model-free training method, which basically just means improving through trial and error. It's not the only way to train a robot. As the name implies, you could use a model-based training method, which would skip the trial and error and just crunch the numbers on an equation that you provide and then output the best result. That might work for something simple, but for humanoids, there's just so many variables involved, like all the motors, that it's just not very practical to use. So how do you set up the simulated environment so that it actually can learn something like walking? So the main components is rewards. You have negative rewards or penalties. You have things like termination criteria, and those things sort of delineate the problem that the neural network will solve. So if you're trying to make it walk, you might specify a target velocity moving forward, and then you might also specify some penalty. I don't want you to fall over. I don't want your waist to drop below a certain height. And then your termination criteria are, if you do fall over, stop. Stop the environment and add some massive penalty. So the rewards are very similar to how you would train a dog, for instance. You would reward behavior that you desire, and sometimes you would penalize behavior you don't desire. But the same way in dog land, in robot land, you pick larger rewards instead of penalizing. So you always wanna reward desirable behavior. There's something kind of poetic about the fact that humans, dogs, and even robots respond better to more positive reinforcement than just to punishments. It's almost like it reveals something deeper about the universe. Bet you didn't think we would get philosophical in this video. And as we'll get into momentarily, it's not the only thing that parallels a real world in an interesting way. At what point have you reached some threshold of diminishing returns where it's like, this is a robust enough policy to work? So it can just be a human choice. Or most of the times, policies will converge to some behavior. Whether that's optimal or not, that's up to how you've set up your problem. But they will converge to some set of parameters, and then the neural network will basically just hover around those parameters. And so the behavior will largely stay the same once you've reached this convergence point. But sometimes that convergence fails. If you're not careful, you can end up with a genie, monkey's paw, careful what you wish for situation. It's called reward hacking. Maybe you want to reward it for not using its knees as much, and you get Peggy the pirate. Or maybe you want it to minimize electricity use, and you get whatever this is. Both might be technically correct based on your reward and punishment incentives, but it's not really what you wanted. So what else can you do besides tweaking those incentives? Oftentimes what we can use is, we know how we want something done. And so what we can do is just show the robot. So we can use something called motion priors. We can either put on a mocap suit or just simply with video, we can show the robot, hey, this is how I like this thing done. And then the robot will learn not only how to imitate the human, but also how to goal achieve while imitating the human. How does it take something like that and learn from it? So with motion priors, what you can do is simply say, track as close as possible what you're seeing. So maybe there's a bunch of points along your body in a mocap suit, and you just say to the robot, just follow those points, just be physically within those bubbles or whatever. Or the other way to do it is distill that into a language the robot can understand. So what you can do is encode it or embed it into some mathematical representation that the robot understands. And then the robot is no longer learning purely to imitate the motion, now it's learning the essence of that motion. So it's kind of invented its own language to identify this stuff with, kind of like we have drawings or written language, stuff like that for real access, right? Yeah, exactly. This is the other part I was talking about, the one that parallels the real world in an interesting way. Something we all take for granted is our ability to distill concepts into abstract forms. For example, language itself is an abstraction that we all have learned and share. So when I say the word elephant, neither the word itself or the picture in your mind that it creates are actually a real elephant. I know that sounds obvious, but that's because our minds by default have processed and hold an abstract idea of the concept of an elephant. In machine learning, this abstraction is like a noise where the essence of a concept is stored. We call that a latent space. This is how other forms of generative AI can combine concepts as well. The latent space can capture the essence of both an elephant and of dancing. So when prompted, it can produce a dancing elephant, despite never actually seeing that idea explicitly. This framework is how we can accelerate how fast a robot learns, which we'll get back to shortly. For now, let's go back to the question of how a robot can combine all the skills that it's learned. So you have different behaviors that you have to train them all discreetly, like here's walking, here's not falling over. How do you bring all those behaviors together? There's a massive thing about generalizability and robotics, like how can I take one robot and have it do everything? And that's definitely our goal as well. But to get there, one way to do it is you train a bunch of specialized policies. Then what you can do is amalgamate them together. So teach one larger policy based on all these smaller policies, and then have some sort of orchestrator or blending between them. So the robot will not only learn to do a specific action and another specific action, it'll also learn what specific situation do I want to use this expert? What other situation do I want to use this other expert? And it'll intelligently sort of blend between different policies, different experts. Sometimes you might blend the entire body. So for instance, if I'm walking and all of a sudden, I don't know, someone pushes me over, like beyond the classic recoverability of just a locomotion policy, what I can do is just switch to my freak out policy. And the freak out policies only train on how to recover from some crazy state. And so that'll be like a full body takeover, right? Like every action from the locomotion policy has now been smoothed into the freak out policy. And that's one way to do it. The other way is you can basically just blend upper body or lower body, or even just like one limb. And so for instance, if I'm walking and then I'm also doing a Rubik's cube at the same time, maybe I'm blending my whole body locomotion policy with some like fine manipulation policy. Do you find yourself like walking around on the street, viewing stuff through the lens of like machine learning? 100%, yes, yes. I have gotten so self-conscious about my walking. I like try to see what muscles I'm using as I walk. I'm like, okay, do I end up using my thigh muscles at all when I walk or do I lock my knee? Yeah. Am I going toe heel or heel toe? Exactly, exactly, yeah. So once it's all working in simulation, the hard part is to bridge the sim-to-real gap. Sim-to-real is like one of the biggest problems in robotics as a whole. We're very lucky to have our hardware team here with us to help us get our models as accurate as possible so that when we train in simulation, we know that it's high fidelity enough that the sim-to-real gap is as small as possible. But obviously we can never get it perfect. And so we do things like domain randomization and other like state-of-the-art techniques where we basically induce some noise and disturbance into the robot. So it's not learning how to hack that specific instance of its body. It's learning how to use many different variations of its own morphology, of its own hardware. So by injecting some noise or randomization, it's just, you're trying to correct for the unknown to some extent? That's exactly right, yeah, yeah. With that said, let's go back to the framework that Vito and his team have built. What we have here is really cool because it's faster and cheaper to run than anything you see out there like publicly. So our policies can run on very little amount of GPUs in a very small amount of time. And they have very similar performance to policies that you see out in the world that are trained on like hundreds of GPUs for hundreds of hours. Right now we're spinning up all these whole body controllers, locomotion policies, and it might seem like they're pulling on the same rope in different directions, but actually they all fit into the same picture of like this one unified policy that you can feed basically anything into and get out actual robot actions. It's like you've kind of unlocked a very capable orchestrator policy that you can continue to feed new stuff into as opposed to having to make new ones each time. Yes, but beyond that, like I think you touched upon the best interpretation of this which is like the language interpretation of it, right? Like the beauty of our stack is they all speak the same language. And if they don't, we have the ability to translate between languages. So before you could like translate languages and map everything into one latent space, what you'd have to do is train end to end these like massive systems. So like you'd have to basically learn how to interpret video, audio, proprioception, motion priors, mocap data, VR, all at the same time while learning how to perform actions because there'll be no way to then extend that into further policies, like yeah, into further like capabilities. What's so exciting about this method is that it will greatly reduce the data and compute it takes to learn new skills. If you paid attention in the robotics space at all, you might've heard things like millions of hours of training in simulation. And while that seems impressive at first because a big number sounds like a robot would be really, really good at something, it's not really practical or scalable in the real world. If you had an employee, can you imagine bragging that it took millions of hours to train them? The best case scenario would be to show them one time, perhaps not even a complete demonstration, and then they can take it from there. At Foundation, that's what we think will take humanoid robots from a dancing novelty to unlocking the real promise of actually being useful and making all of our lives better in the long run. If you wanna sneak peek into that future, the best way is to come work for us. And the second best way is to hit that subscribe button. And let us know if you want us to go more in depth into this or any other topics. That's all for now. Thanks for watching.