What Happens Inside a Graphics Card

What does a graphics card know about grass? Nothing at all. And yet, one drew this whole valley in under a hundredth of a second. It doesn't know what money is either, but for a few years people bought so many of them to mine cryptocurrency, in the middle of a chip shortage, that gamers couldn't find one in a shop. It doesn't know what a film is, but graphics chips drew all the 3D in the one you're watching. NVIDIA is worth more than most countries produce in a year, And the company behind most of them? because most of the world's AI runs on its chips. So today we're going inside one, to see how the compute actually works. By the end, you'll understand how it works, why every one of those jobs depends on it, and how one man built an empire on a chip made for video games. The one we're opening is an RTX 3060. More gamers on Steam have one than almost any other card, which makes it the one to take apart. Start with the part you can see, because almost none of it computes. The shroud and the fans are there to move air. Under them is a block of aluminium fins with copper pipes running through it. And all of that exists for one reason. At full load, this card turns 170 watts into heat, nearly as much as three old light bulbs. Lift that off and you're looking at the board. Most of the power comes in at 12 volts through the socket on the edge, and this row of small parts steps it down to about 1 volt, which is all this chip can take. The gold fingers along the bottom connect the card to the rest of the computer. These six black squares are memory, 2 gigabytes each, and we'll come back to them. And in the middle, under a smear of paste, is the one thing all the rest is there to feed and cool. It's a piece of silicon, a little smaller than a postage stamp, with 17 billion switches on it. That's the graphics chip, and it's where the compute happens. Compute only means arithmetic on numbers, and to a chip, everything is numbers. But there are two very different kinds of job. One kind is a chain. Do this, look at the answer, decide what to do next. Opening a file is like that, and so is most of what your computer does while you click around. Each step waits for the one before it. The other kind is one small sum, repeated on a mountain of numbers, where no answer has to wait for any other. Making every dot in a photo a little brighter is one of these. When people say a job needs a lot of compute, this is the kind they mean. Your computer already has a chip for the chain, the processor, the CPU. It has a few brilliant cores, ten in ours, each working through its own chain, step after step. We opened one up in an earlier film and built an adding machine out of marbles to show how a chip does sums without knowing what a number is. It's linked on screen now. But 10 cores are far too few for a mountain, so a graphics chip goes the other way. Thousands of simple cores, 3,584 on this one, all doing the same sum at the same moment. Look closely at the chip and you can see them. The same block, repeated row after row. 28 of those blocks are switched on in our card, each holding 128 of those simple cores. The box calls them cores. They're closer to workers, because all one of them can really do is add and multiply, and they take their orders 32 at a time. One order goes out, and 32 workers carry it out together, each on its own number. So to understand a graphics card, we can set almost everything else aside. The fans, the power parts, and the memory are only there to feed the workers and keep them cool. The chip has a few specialists too. The ones the box calls, ray tracing, and tensor cores. But the great bulk of the work falls to the workers. Take all of that away and watch one worker do its job. Say it's handed three numbers. Two, three, and five. It's also handed a row of numbers from a table. One, zero, zero, and four. 1 times 2, 0 times 3, 0 times 5, then add 4. That's 6. Three more rows like that, and the sum is done. 12 multiplications and 12 additions. And you could do it with a pencil. How does a worker multiply? With switches that are only on or off. Multiplying is adding, done over and over. And adding is the trick we built out of marbles in the CPU film. Nothing in the worker knows what 2 is, or what multiplying means. Those numbers don't mean anything yet, and to the worker, they never will. We're about to give them three different meanings. So what does all that compute actually do? Take a game. A screen is a grid of tiny dots, about 2 million on an ordinary monitor, and in a game, everyone needs a new color 60 times a second. Think of a stadium where everyone holds up one colored card. From your seat you see one card and the back of someone's head. From across the stadium you see a picture and nobody in the crowd knows what it is. A graphics card paints the same way. Everything in this valley is made of triangles, and those three numbers we just multiplied are one corner of a triangle. How far across, how far up, how far away. The row we just did moved that corner four steps to the right. Put other numbers in the rows and they turn it to face your eye and work out where it lands on your screen. A special circuit then asks which dots are inside the three edges, and the workers take those dots, 32 at a time, and work out each one's color. Now take a video. A frame of video is the same grid of dots, and in an edit, a dot's color is three numbers, red, green, and blue. Brighten a shot, or change its color, and every dot goes through the same small sum. That's the same row we just did, except that the 2, 3, and 5 are now a color. Squeezing the finished film into a file is its own job, so the card has a separate engine on the chip for only that. Mining was the strangest customer of all. To earn a coin, you had to find a number that, when scrambled by a fixed recipe, came out with a special pattern. There's no clever way to find it. You just guess, and no guess depends on any other, so a crowd of thousands was perfect. It isn't our row this time. It's a scramble instead of a sum, but it's the same shape of job. One small recipe, a mountain of numbers, nobody waiting for anybody. Ethereum's recipe was also written to need a lot of fast memory, which kept graphics cards competitive even against chips built only for mining. Miners helped empty the shops. NVIDIA even built a limiter into this very card to make it worse at mining, then accidentally put out a driver that switched it off. In September 2022, Ethereum changed its rules, and mining it stopped overnight. And then there's AI. An AI model is a huge pile of numbers arranged in grids. Your question becomes a list of numbers, and that list is multiplied through grid after grid, with a small twist after each, just to choose the next piece of a word. We followed one answer all the way through a model in an earlier film, linked on screen now. Look at what most of that is. Multiply, multiply, multiply, add. It's the same row. Two, three, and five were a corner of a triangle in a game and a color in a film, and here, they're part of a sentence. The worker can't tell the difference. That's why one chip can do all of it. And it's why, in 2012, a graduate student in Toronto and his team could train an AI model on two gaming cards and win the biggest image recognition contest of its day by a huge margin. After that, everyone building AI wanted them. And one company was ready. Years earlier, NVIDIA had opened its gaming chips to any kind of sum, not only pictures, so when AI arrived, the cards and the software to run them were already there. That is how a chip made for video games ended up under most of the world's AI, and how NVIDIA's co-founder, Jensen Huang, built an empire on it. Today, our card sells for about $460. The biggest gaming card on sale, the RTX 5090, now goes for between six and ten thousand dollars. That's more than ten times the price, so what's in it? It has six times as many workers, nearly 22,000. The chip is about twice the size, and the switches on it have shrunk, so far more workers fit. It has more than two and a half times the memory, and about five times the road to that memory. It draws more than three times the power, which is why it needs such a serious cooler. It doesn't do a different kind of work. The workers are newer and quicker, but they still add and multiply in gangs of 32. Mostly, an expensive card is the same idea with more of everything. It won't make every game six times faster, because something else is often the one holding things up. But if your job is a mountain of numbers, more workers go straight And one of those numbers matters more than people think. Those six black squares are where the workers get their numbers. Gamers call this VRAM. Think of the workers as a kitchen and the memory as the shelves right beside it. Everything the workers need should be on those shelves. Every triangle and picture in a game, every frame of a video, every number in an AI model. Thousands of workers get through numbers very quickly, so the road between the chip and those shelves is enormous. On this card, it carries about 10 times as much as the road from the card to the rest of the computer. But the shelves are a fixed size. When the job doesn't fit in 12 gigabytes, Windows keeps the rest in the computer's own memory, at the far end of a much narrower road. So we measured it. We gave the card a grid of numbers, made it bigger and bigger, and timed each gigabyte. Up to 11 gigabytes, the time per gigabyte barely moved. At 16, past the end of the shelves, the average gigabyte took about 12 times as long. In a game, that can be the sudden stutter when the textures are turned up too high. And an AI model has to fit on the shelves to run at full speed, which is why AI people care more about how many gigabytes a card has than almost anything else. That's what's on the box. To see what those numbers really decide, we put this card up against a very different machine. This MacBook has an Apple M5 Max chip with 40 graphics cores in the same chip as its processor. They're counted differently from NVIDIA's, so the numbers don't compare. What's really different is the memory. There's no separate shelf for the graphics. All 48 gigabytes are shared. It isn't a fair fight. This laptop costs about 10 times what our card does, and its chip is five years newer with smaller switches and more of them. So don't watch who wins. Watch what decides it. First, the proof that a graphics card is worth having at all. We gave this valley to the RTX 3060 and to the processor in the same computer, using a Windows program that does a graphics card's job on the processor alone. The card took less than 9 thousandths of a second. The processor took more than a whole second. In the time that program took to draw one picture, the card drew 131. Now, the Mac. On AI's kind of sum, a grid 4,000 numbers wide times another, the 3060 did almost four trillion multiplications a second. If everyone on Earth did one multiplication a second, we'd need about eight minutes to match one second of this card. The Mac did seven trillion. Then light. Everything we've drawn so far fakes it, because real light bounces, and following that takes enormous compute. We had both machines follow 4,096 paths of light through every dot of this valley. The 3060 took 11 minutes and 6 seconds. The Mac, with ray tracing hardware of its own, took 2 minutes and 39 seconds. We gave both the same 31 seconds of 4K footage to squeeze down, each on its own video engine. It was close to a draw, 11.5 seconds on the 3060, 12.5 on the Mac, though the Mac's file came out a third smaller. In the rounds that take raw compute, the newer, bigger chip won. Then we reached the round that decides something else. We ran the grid test from earlier on both machines. Up to the end of its shelves, the 3060 was ahead, and at 11 gigabytes it was more than twice as fast. Past its shelves, Windows kept the overflow in the computer's own memory, and the card had to reach it over the narrow road. By 16 gigabytes, the average gigabyte took about 12 times as long. The max time per gigabyte hardly changed from 8 gigabytes to 16, because all of that fits on its one shelf, and at 16, it was five times faster. On that round, speed wasn't what mattered. It was whether the job fit. We asked the model on our card what a graphics card does. A graphics card, it told us, is like a super powerful artist. It has no idea what an artist is, either. It's three and a half thousand little machines, adding and multiplying. Give them a corner of a triangle, and you get a field of grass. Give them a frame of film, or a sentence, and they do the same kind of sum, and not one of them has ever seen grass in its life.