Can AI kill the greenscreen?

Listen, I've got a test for you. I'm going to set up a fake zine and see if you can tell that I am- In front of a green screen? Yeah, he- He is. But here's my question. When is AI going to replace this clunky 20th century invention? I mean, after all, AI is the technology that can draw masterpieces and generate poems and hunt down our loved ones and kill them. So it can totally beat out a color, right? Right? Okay, I found a fantastic picture of a happy family. I am going to get rid of their background to show you some of the ways that we have taught AI how to do it. I can start by just separating the foreground from the background. A lot of fast AI programs do it this way. Or I can do segmentation, where I figure out and label each part of the picture and remove what I want. Or if I get even more fine-grained, I can label each instance in a segment, like Sheila, Lorraine, and Sheila Jr., and then get rid of the wall and the floor and Sheila Jr. She knows what she did. AI has gotten really good at splitting stuff up, at segmentation. For example, Meta AI just released a new segmentation model that can do this really easily. And AI will use different methods depending on the job that it's told to do. But it's a bit different when you're trying to make a really good mat, a perfect cutout, so you can add whatever background you want. This is really hard stuff. See the hair? That's often the key obstacle. Today, a lot of AIs basically train off people doing what I am doing now. Some labeling people, but also extremely tedious separations of the foreground from the background. For one data set, people at Adobe took pictures like this, went through just like me, and made a perfect mat. You can see they left out some really fine hair here. AI then looks at these mats and learns how to create one. They might get more practice by putting their mat over different backgrounds and having to try to cut out the person again and again. I just want to be clear here, the AI learned how to do this because some people were squinting at their computer, erasing backgrounds, and it learned to copy that. Others have expanded the stuff that AI can learn from by taking green screen footage. They manually keyed out the footage, removed the green, and then the model practiced with those mats on different backgrounds, trying to cut them out over and over again until it got really good. That helps the robot improve its skills. Researcher Rani Sengupta and his co-authors made this data set. They used that data to make a really, really, really good green screen. And then another one with some researchers from ByteDance, the owners of TikTok. I am going to pull it up now. It is like a Zoom background on steroids. I especially had to instruct all my subjects to play with the hair because there is no other part where this transparency becomes more visible than the hair and the glasses. So I'm looking at people's faces like this when they turn, the glasses comes into effect, there is transparency. So in practice, you are trying to predict something which is mostly zero in most part of the background, and it's mostly one at the center of my body. But around my age and the hair and everything, it has some values that is between zero and one. Between zero and one. He's talking about where the AI has trouble predicting what is Rani and what's office. And that's a hard problem to solve. So AI green screens are great, right? You don't need a big green screen, but they fail in other ways. This is TikTok. Let me put on the green screen effect. So it looks good, but do you see how it's kind of clamping down on my head right here? That's because these AI green screens are optimized for speed, not necessarily for Hollywood level quality. It doesn't look bad at first, but then when you see the comparison, you realize exactly how much information it's cutting out. In that fancy loft, I stood close to a really textured background. Do you see how the AI green screen had trouble making a perfect separation? It got some of the numbers right, but it's confused about others. And the final problem is something that traditional green screens can do that AI green screens have to adapt to. I'll put my laptop here way in front of me. The green screen has no problem getting rid of all the green because it doesn't have any knowledge of what's in the frame. But the TikTok green screen is not a real green screen. It's trained to look for people. So it's going to cut out my computer, even though I might want it in the shot. That's a problem that traditional green screens obviously don't have. Somebody has to tell the AI what to do. A lot of advanced AI video editors do this by making masks instead. A mask is basically an outline. So the computer has you click the outline and does its best to make a mask and move it along throughout the frame. It could be a car, a person, an animal, an object. And so we train a model just to be able to recognize that in a very accurate way. You want a perfect, hopefully perfect mask. Okay, so this is Chris Valenzuela. He is the CEO of this very popular AI focused editor called Runway. Their green screen tool is one way of solving this AI green screen problem by having you choose what you want to keep rather than just removing the background. Where I see artificial intelligence being very helpful for video and creative endeavors is that it's not an automation tool. It's not something that will just replace end-to-end the whole process. It will just help you reduce and replace the boring aspects of the process. We are seeing two kinds of AI emerge currently for any kind of filmmaking visual effects things. One that is very purpose-driven where we want to give cinematographers actual control over what is going to happen. And another one, which we want random people to post their videos and create cool effects. We imagine green screens and AI as switches, but they make more sense on a graph of speed and quality. TikToks might be here, an editing program might go here, and a green screen could go here. Good preparation can improve quality, but perfection in almost every case still requires an artist to finish the job. Is the model 100% there yet? Probably not, but it's 90%. And that 90% is significantly enough to reduce the time of translating hours into minutes. That means less time squinting at screens. And more time. In the loft. You're ready to make this video. Or with your family. MUST DESTROY SHEILA GEAR Okay, okay. One more thing. Can we get a slightly cooler background than this? Okay. That's slightly. So a couple of years ago, I made this video that's called the technology that's replacing the green screen. And that's true. It's these giant screens that are often hooked up to a game engine, and they provide realistic lighting and movement of the background. It's amazing stuff. But since that video, and as the technology has become more widespread, visual effects artists have been open that often they end up rotoing even these images. What is rotoing? Basically, it means that they replace the background. They meticulously trace out the characters and replace the background once again, even though they were supposed to have it in camera. Now this doesn't always happen. And there are still huge benefits to having accurate lighting on set. But the point is simple. If you put those amazing LED walls on our little graph, they would still be just short of the perfection that you can get from an artist who has a lot of time on their hands.