Andrej Karpathy: Tesla Autopilot and Multi-Task Learning for Perception and Prediction

We rely primarily on vision. So what we do have instead is we have eight cameras that are pointed in the 360 around the car, and we obtain video streams from all those cameras, and we parse them using neural networks, and we stitch that up into a view of what's around us in three dimensions in 360. So this is more of what we see, what the car sees at every single frame, and we're trying to stitch it up. So I'm now going to go into the computer vision problem and what's necessary to get this to work. Now, we're in a multitask workshop. So first, I will talk about just a single task and make the point that even for one task, this can actually be a very difficult problem if you want this to work all the time. So 97 percent accuracy is not good enough. You actually really need high 99.99 percent accuracy for this to be actually safe to deploy. So as an example, even a basic visual recognition task like object detection or putting bounding boxes around cars, this is very difficult if you actually want to ship this and make it safe, because cars don't just look like that. They actually have a massive variety to them. So there are cars with signs, cars that look odd, cars on cars, bikes on cars, cement trucks, cars with falcon wings, boats, and other objects. Cars that take up the entire field of the view of the camera. So even your basic bounding box abstraction actually just breaks. Cars with appendages and excavators, cars upside down, cars with trailers, and then of course, heavily occluded cars. So for example, in parking lots, you just have very heavy occlusions. It's very difficult to deal with, with a very naive approach to object detection. So even in the process of getting a single task to work, there's a ton that goes into what is the architecture? Where are you putting your dilations? Are you using ResNet or DenseNet, or what does the architecture look like? What does your loss function look like? What object detection technique are you using? But also not just for the model, but the dataset itself. We have to iterate on the dataset, and we do that in an iterative fashion in something that I call the data engine, which is a process by which you iteratively build your dataset. So you have a network that you've trained on some dataset V0, and then you train your network, deploy it, and you have mechanisms by which you notice that the network is misbehaving or produces inaccuracies. And you source those examples, annotate them correctly, and incorporate them into the training set, and you spin this data engine loop over and over to actually get this task to work really well. And we don't just do this for one task, we do this for many tasks. So let me try to go into how this gets very crazy very fast. And the core issue is that there's actually a ton of things you might want to know about the environment if you're trying to drive. So as an example in this random image of a residential neighborhood, you don't just want to know about the cars, but there's, of course, a ton of things you want to know. You want to know about the static objects, not just the moving objects. You want to know about the road signs and what they're telling you. You want to know about the overhead signs next to the traffic lights, and they tell you about whether or not you can do a U-turn or anything like that. You want to know about the traffic lights and all their state, and you have to handle all the variety of those traffic lights. The lane lines, which hint at the flow of traffic. The road markings that tell you what you can and can't do in any lane. The curbs that tell you about where you may or may not want to move if you want to not damage your vehicle. The crosswalks that hint at the behavior of pedestrians in the vicinity. And lots of environment tags, as an example, like is it, are you in a residential neighborhood? Are you in a tunnel? Are you coming up to a toll booth? All those predictions are also kind of binary classification problems, for example, that we solve for every single image. So there are a ton of things you might want to know about every single scene. This is not an exhaustive list. It's also a bit of a cartoon. These are not exactly our tasks, but it just gives you a sense of, there's about 50 or so tasks that we simultaneously work on in the team. And even though I've put only a single task here such as moving objects, there's actually lots of subtasks for moving objects because I want to know a ton about all those objects. Is it a pedestrian, or a bicyclist, or a motorcyclist, a pickup truck? Is it a truck? Is it a bus? All those things have repercussions on your driving policy and your expectations about how these things might move. Is this an ambulance, a fire truck? Is it a police car? Are there sirens on? Is it a construction vehicle? Is the left blinker on, right blinker on? Are the rear lights on? Is the car cutting in? Does it look like it's parked? All those things are subtasks that you might want to know. All of these are independent predictions that you must predict on every single car. All of them have their own data engine. And so the scope of this problem very quickly just becomes quite daunting. So that's kind of what we're faced with. We make about 20 or 30 tasks, and every one of those has subtasks. So you end up with basically 100 subtasks that we all have to solve simultaneously and have to make work really well to very high accuracy requirements before we can actually deploy it. So I'll talk a little bit about some of the struggles and some of the ways that we're approaching these problems. So from the architectural standpoint first, obviously we are putting in some image and we'd like to understand a lot of things about it, but there's a lot of design space as to what goes into the black box that makes this network. So at one end of the spectrum, you might imagine that every single task, for example, could get its own network, an independent network. And all of these different choices architecturally will have their own pros and cons. So in this case, for example, if you separate out every single task to its own network, then some of the downsides are that, as an example, this will be very expensive at test time. We have a finite compute budget on the car. We want to run this at high frame rates to ensure safety. And so, of course, you might not have the computational budget to actually run 30 different tasks, 30 different components at every single frame for every single camera. Moreover, there's no feature sharing in this kind of a setup. And so if one of the tasks, as an example, might not have enough data, then you're going to actually overfit. But what is nice about this, and I'm going to go a little bit more into the team workflow of how you actually develop these networks, is that it decouples functionality. So if you only want to improve on moving objects, as an example, then you retrain your network on moving objects and someone can iterate on that and try out a different detector or something like that. And you know that you're not changing or regressing some of the other tasks. So you can sort of iterate on one thing, holding everything else constant, and you don't have to re-evaluate the entire system as much. On the other end of the extreme, we could try to have all of these tasks be separate heads, very lightweight heads, on top of a single shared backbone. So this is kind of on the other end of the extreme. In this case, this would be significantly cheaper at test time because this backbone is basically multitasking and all those things. But there are some downsides. So as an example, all of these tasks will now fight for the same shared capacity. Sometimes they fight, sometimes they actually help each other. There's a complicated binary relationship there that I will go into. But the downside is that this fully couples the functionality. So if you're just trying to improve your moving object detector, then of course, maybe you're just fine-tuning the last piece without touching the backbone. But if those features are not good enough and you actually have to touch the backbone, then suddenly you have to continue training all the other tasks. And suddenly you have to re-validate all the other tasks. So it just couples the problem and makes it much more challenging to actually evaluate. So there's a massive space of in-betweens of the architectures in between this one and the first one that I showed. And exactly how you lay out the architecture and how you plug everything together, there's a large unknown there. So you can try to explore this manually or you can try some approaches that have been developed that are automatic. Typically these approaches, like this paper, AutoDeepLab that has come out recently, they're trying to find the optimal architecture in the NASNet kind of style for semantic segmentation problem. And so they find where should the different operations be, how are they plugging together, where's the encoder, decoder, how do you upsample and all those different things. So you can imagine running something like that in our setting, but it gets a bit more complicated because we don't just have one task, we have multiple tasks. So exactly how do you lay out this architecture? How much feature sharing is there? Maybe the first three layers are shared, but from then on, they split off into different networks. How much capacity do you allocate to any single different sub-piece of this network? It's not really obvious how you actually go about exploring all these different choices. There are papers that recently have started to come out that actually study these multitask settings and all these different tasks, they're typically very heterogeneous. So some of them will help each other and some of them will hurt each other. And so there's a complicated binary matrix here in this paper of which tasks seems to be helping each other and which tasks seem to fight. So as an example for us, moving objects and traffic lights typically do not share too many features, so we see that they fight, but some other tasks actually help each other. So when you train moving objects with a second task, the accuracy on the moving objects could actually improve. So what they find in this paper is that they study basically the problem of how many backbones do I have and where do I put the heads and how should I allocate the heads to the backbones to come up with the optimal performance? And they study this as a function of your computational budget. So if you don't have too many resources computationally, then you would have a single backbone and you would hang all the heads on top of that single backbone. But then on the very bottom, let me try to point. This is kind of ambitious, but I can't see. But on the very bottom, they find that the optimal assignment, if you do have more resources, is that you actually have independent backbones and you place some of the heads on some of the backbones, but there's a discrepancy between a train and a test kind of training and inference there. So as an example, one of the backbones could be trained on five tasks, but then at test time, you only intend to run three of those five tasks on that backbone. The other two tasks are only there to create better features for that backbone, which is really interesting. So we also find similar things, except we have like 30 or 50 tasks and we try to study some of these relationships, but it's a very complicated, but we do see these complicated interactions. It's not obvious how you actually study them. In this paper, they propose a few ideas there. Now, where it gets even more complicated for us is that we don't just have a single viewpoint. We actually have eight cameras, as I mentioned, and all of those cameras, as you might imagine, are just different viewpoints, but they all have cars, they all have potentially other objects like traffic lights and so on. And so you might imagine that you actually might want to share features across those different viewpoints. So for us, it looks a little bit like this potentially, where we might have some of the first layers that are shared across all the other viewpoints, because all the viewpoints will have edges and T's and little wheel shapes and stuff like that, but then they start to split off later in the network and eventually on a per camera, you might actually split off all the different tasks. And so this is potentially what that network could look like. And then the other dimension to all of this, of course, is that not only do we have tasks that are per camera or per or use multiple cameras, but we have tasks that potentially could use context across time. So you have to imagine taking this beast of a network, stamping it out across time. And if you're using something like a recurring neural net or something, you would be sharing features across the time dimension, but effectively your network ends up looking maybe something like this. It's a bit of a beast. And then what's going on is that you have all these different tasks that this network is simultaneously juggling. So as an example, maybe I take a batch of static objects on a narrow camera, which is one facing forward. And then what that does is it subsamples a part of this network and only on that part of the network are we doing a forward, backward and an update. And the other parts of the network are held still. And then maybe we have a different batch. And in this case, it's a batch of path prediction. I'm not gonna go into the details of path prediction, but this task actually could be a prediction that is based on not just one camera, but three cameras, giving yourself a bit more of a wider field of view. And so in this case, we subsample a different portion of the network and train that. And then we subsample the, and then we try to train the cutting detector and the cutting detector might benefit from temporal information of how these cars are moving over time. And so you might activate again, a different part of this network. So while you're training, you're always sampling these different tasks and subsampling this network and training different pieces of it. And they're sharing some of the features in complicated ways that are sometimes difficult to honestly like derive as to what they should be. So there's a lot of intuition that goes into it and a lot of experimentation, of course. So those are just the architectural considerations. We have a ton of considerations with respect to loss functions. So as you might imagine, all of these different tasks will kind of mathematically plug into the same final loss. And that's what's being back-propagated. Now, when you actually look into the literature of how people deal with these very large and multitask settings, I'm just picking a random paper here, this panoptic segmentation paper that I found a few weeks ago, maybe. So what they're doing is simultaneously both something that looks like object detection, but also something that looks like semantic segmentation. And you have these lambdas here and those lambdas are hyperparameters. They are task weights. So how much do you weigh the object detection part and the semantic segmentation part? And typically what you find in these papers is that there's a large grid search going on over those task weights. So it's a brute force solution trying to identify what those weights should be to get the best mean average precision or something like that. And this works if you have two of them. It does not work if you have, say, 100 of them, which is what we do have. And so you have to do something different. A lot of it typically relies on intuition. I'd love to see other approaches, but the difficult aspect of this is that the heterogeneity of these tasks is not just on the architectural level because some tasks might need, for example, much more dilation than others. But this heterogeneity is also on the level of losses. And that's because some tasks might be in different scales. So their losses could be classification or regression. So different scales of these loss functions would have different task weights potentially. Some tasks are much more important. So as an example, moving objects task would be much more important than the road markings task. If I don't correctly predict road markings, I might take the wrong turn, but at least I'm not going to potentially crash into something. Some tasks are much easier than others. So what we see is that some tasks just converge very easily and very immediately, but some tasks have a massive amount of variety, like moving objects. They will take a long time to train. Conversely, say like a road markings task, if you're trying to identify a left arrow, that left arrow will always take on a very similar appearance, so it trains much faster. Some tasks have much more data than others or have much more noise. And so this has repercussions on the regularization strength of the different subparts of this graph. So if I have some tasks that has very few examples, then of course I can't afford to train a very thick head for it, or I can't afford to use too many parameters there, or I might want to regularize that piece of the network much more strongly. So it's not just about the weights, but also the regularization coefficients for all these different tasks. And there's no real language in the community as to how we sort of understand or measure all these different components for tasks. Like what are the things that matter? The amount of noise, the amount of data set, some kind of an importance that maybe could be tuned by the person. But how you actually come up in a principled way with these task weights, I think is like quite unknown based on what I've seen so far. So those are sort of the statics of the problem, the architecture and the loss function. There's a lot that goes also into some of the training dynamics of what makes this hard. So as an example, we are trying to get a traffic light color to work well. If you are just naively sampling your data sets from the fleet, then you might find that most of the traffic lights will always have a red or green color. And so it will be very unlikely for you to run into orange or say blue or something like that. So you might be in a setting where you have a million red and green, and you only have like 50,000 of orange or 10,000 of blue after you've tried really hard. And so what you do typically in that case is, in academia, what you'll see as standard is you might want to start to massage your batches. So it's not just about the task weights, but you can also massage the batches and make sure that you have some representation of all these rare classes due to the labeling balance inside your batch. Now this oversampling or this tuning of these batches doesn't just happen on a single batch level, sort of like within a task like traffic lights, but it also happens across tasks because some of these tasks that I mentioned are much easier to train and train much faster. So as an example, the road markings task, we might only want to sample a batch of road markings every 10th cycle, but moving objects, we might want to sample more often to train. And so we have a schedule for batches that is both on the level of tasks, but also inside a task, we have a schedule of batches and examples that we pick there. So here's our dataset as an example. In a cartoon setting, we have rows, which are different images, columns, which are different tasks, and then X indicates a presence of a label for that task and that image. As you'll note, this is a very sparse array. Typically in academia, what I'm seeing is that these are almost always assumed to be dense. So each image has all the supervision. This is not the case for us because you have to remember there's a data engine for every single task. So they'll have their own sort of instances. And so you end up with a very sparse array. There's oversampling across both tasks, but also within a task, there's all kinds of complicated oversampling. And what you end up with typically is something along the lines of sample a task from your schedule, sample a batch for that task, do a forward and backward, and do an update. And then some of the issues that you run into, as an example, and these are somewhat subtle, is that suppose I take a batch of moving objects and I did all this work of the forward pass and I do the moving object's head, and now I'm gonna do the backward pass. So suppose that this example that I just forwarded also happens to have a label for static objects. So since I've done all this work of the forward pass, I might as well want to amortize that work and actually train on the static objects if I have that label. So should I train that component or not? So you'd like to amortize the computation and train on it, but if you do, you're actually messing with the oversampling ratios that you've specified per task. So that example, you're sort of messing with the distributions of your batches in some non-intuitive ways. So what I'm trying to get at is, and I don't wanna go into full detail, but the scheduling of this is somewhat complex and there's a lot of open questions here. The other kind of interesting aspects of this are, you all know about early stopping. When you're trying to get a model to work really well, you monitor the test loss or the validation loss and you take your model whenever your validation loss is lowest. That's early stopping. That works well if you have one task. What happens when you have 200 tasks? Where do you take your model? I don't know. So your plots will typically look like this in TensorBoard and now you're trying to get your model and obviously some tasks will overfit much earlier than others because they're easier or they have more data or less data. And so this heterogeneity, again, is pretty tricky to deal with. And how do you now manipulate the tasks or the oversampling rates or the task weights or the regularizations so that, ideally, these end up in the same spot so you can just cut a clean model? It's not obvious. The last piece that I wanted to go into, just mostly for entertainment, actually, is with respect to the team workflow around how you actually collaborate on a network like this. As I described, this is kind of a beast of a network. It multitasks so many things and we have a finite team and this team is dispersed across the network and collaborating on this network at the same time. So maybe we have this notion of a task owner. So someone is working on the moving objects. Someone is working on static objects. Someone is tuning on traffic lights and the architecture there. Someone is messing with the loss function for traffic lights and two interns are working on the architecture on the bottom. So the tricky thing is that, and I haven't seen this ever before, is you have this large network but you have, say, 10, 20 people collaborating on this network and they all own different subparts of this network. And what does that look like in practice? If you have a code base, of course you have Git and you have pull requests and commits and so on, but these people are collaborating on the neural network and this is all coupled functionality. And it's really not obvious how you iterate on networks when multiple people are simultaneously changing the tasks, the weights, the data itself and everything is moving. So just a few interesting kind of things that have happened in the team. As an example, I walk around, that's me on the bottom left over there, and I say, hi, traffic lights task owner. If you could make traffic lights work a bit better, that would be great. And the traffic lights, and there are many tips and tricks to actually getting your task, the one that you own and you want to work very well, to work well. So the traffic lights owner has an idea and they go to the configuration file where you list all the different tasks. And as I mentioned, we have different oversampling ratios for different tasks. So you just go to the traffic lights and you see your oversample is just one, so you make that two. And that means that your task will be trained more. And that means your network will work better on your traffic lights task. And then you say, hey, can you prove my commit? And a random person says, okay, and moving objects person gets upset. What's going on here? Because their task will be starved of resources and the capacity of the network is finite. And so suddenly this is not okay. So this is a very obvious one the team has caught on. We don't try to manipulate oversampling ratios too much, but there are many more devious ways of making your task work better. So as an example, you go to your traffic light loss and you just multiply your loss by 10. This makes your task work better potentially, but it's not illegal, but it's definitely frowned upon. This is not okay. So, I mean, the higher point being that there's finite capacity to go around and a lot of people are trying to simultaneously get their task to work well, but there are many okay ways of doing that. And some of them cause political drama, I would say in the team. And I have to somehow like allocate capacity to the tasks, but there's no easy ways of doing that. It's not really task weights, it's not oversampling. So I don't really have language to describe how to correctly allocate capacity to tasks and interest and weights to them. Another interesting aspect that we've noticed is very often it is the case that you have all these different tasks, they simultaneously coexist in the network. You're pretty happy with everything, except for this one task that is kind of misbehaving in a certain way, you want to change that subtask. And so it has some kind of a failure mode that you want to fix. So because you don't want to touch the full network because everything is validated and signed off, you might have this idea that, what I'll do is I will just fine tune this little piece of this network and I will make this work a bit better just by fine tuning that small piece. I'm not touching any of the other tasks, nothing has to be revalidated and so on, and I'll just try to improve this in isolation. So basically you fine tune just your task and you leave everything unchanged, you create a new model and you might want to go with that. So what's the worst that could happen is the following. So what happens is you end up with a very complicated lineage of models where a person took a base model and fine-tuned it, and then a different person took their model and fine-tuned something else, and then someone fine-tuned something on top of that, and then someone fine-tuned something on the wrong base model, and now because the model has changed, so now there's a complicated chain of fine-tunings that people have performed, and this doesn't work. It starts to look something like that. So you really have to, and of course, the worst of it is that some of these models then become not reproducible. So what you can end up with, and this has happened to our team, is you end up with very good models that have gone through non-intuitive chain of fine-tunings, and you end up with a very good model that you can't reproduce, and that is the worst because you retrain from scratch and you're regressing some performance that you've had there. So it's very appealing to want to do this, but we found, for example, because of the workflow that you actually just can't. You can't get away with that. It's not a good idea. So in summary, just some of the things I talked about. Of course, we have so many different tasks. There are many considerations to the architecture, to the loss function, to the training dynamics, and how to deal with all the regularization and the task weights. There's no language that I'm aware of in academia to talk about the heterogeneity of these tasks and different properties thereof, and how to make everything work with early stopping and stuff like that. And also the most fascinating component to me is when you have networks that are being simultaneously worked on by 20 people, what does that look like? What is the etiquette? And then what are the tips and tricks and rules of thumb for how to work with these architectures so that everything works? Okay, great, thank you.