DeepSeek flips the script on generative AI

there are decades when nothing happens and there are weeks when decades happen as so the saying goes and this is certainly true for the final week of January 2025 in the weeks leading up to recording this episode we had planned to talk about Stargate open AI Oracle and softbank's half a trillion dollar AI investment we will but in a very different context because unless you've been under a rock you'll know that today's episode has to be all about deep seek to say the Chinese large language model has caused an upset in the market would be an understatement within days of its release it wiped more than a trillion dollars off the US Stock Market sending big Tech into a public meltdown but where did deep seek come from and what does it mean for the future of the AI sector you're listening to the it pro [Music] podcast hi I'm Jane McAllan and I'm Rory bate today in conversation with it Pro's news and Analysis editor Ross Kelly we're exploring the profound implications of deep seek Ross welcome to the show good to be back a rare podcast from the office so Ross for those who don't know can you set the scene a little bit with deeps yeah so last week um obviously the roof was blown off Silicon Valley to an extent with the release of deep se's new models V3 and R1 both of these models have really blown people away essentially our one in particular you know has been positioned as a direct competitor to to open AI based on you know it's reasoning capabilities things you know really picked up pace over the weekend and it was on Monday and this week that we that we saw things I don't want to say spiral but yeah kind of get a tad out of control I have to ask I'd never heard of deep seek before this week so we are recording on the 29th of January and until everything kicked off on Monday and carried on kicking off on Tuesday I just never heard of this company so is it that I haven't been paying attention I know they're Chinese you know say are they the Chinese equivalent of mistal the French AI company or are there complete stalking horses come from nowhere I think there's a combination of sort of factors there you know a lot of people paying you know very detailed attention to the sector will have had you know an understanding of Deep and what they're doing what they've been up to but for a significant portion of you know the global tech industry and you know Tech enthusiasts I suppose and they really have seemed to come out of nowhere and I think that's what's made it so impactful that you know all eyes have been on Silicon Valley consistently for you know the last two years of this generative AI race and then all of a sudden you know a Plucky start up from China comes out of nowhere and you know sends everyone into a bit of a meltdown I think something it's important to add here is is that um some analysts from Gartner had said that they've been tracking deep seek since sort of mid 2024 you get bragging rights for that but but as Ross has just said there are so many AI startups um and a clip that a lot of people have been sharing around semi humorously in the last few days um is a clip from 2023 where an Indian journalist asked Sam Alman what's the worry what's the concern that a startup in say India or China could rival you within the next two or three years and he had kind of shut them down and said the the compute demands cannot be met so that's not a concern for us it's not going to come from that sector it's you know it's gained a lot of traction and it kind of highlights the issue of even if you were tracking this company you're tracking dozens and it's kind of I think it it definitely has put the fear in in some big tech companies that any any one of these doesn't could be sitting on kind of the next big breakthrough what is is it they say about Pride coming before a fall it's just incredible humorous so who were the big Losers of the week so as we said things went a bit into meltdown that's not just the Tech Community but their stocks also started spiraling completely so yeah Nvidia a big I don't want to say loser here because you know there's signs that there's rallying going on in the markets but Nidia dropped around 17 points um losing almost 600 billion in market value broadcom as well equally down around 17 points a lot of the infrastructure companies here in particular or you know your Hardware manufacturers this is largely due to the fact that deepseeker they have found you know a way to really Eek A lot of efficiency out the hardware that they're using which is the really impressive bit about this that I'm sure we'll we'll go on to discuss but yeah when you look at these you know manufacturers chip makers they've they've really taken a pounding this week but I mean equally meta Microsoft um you know they they sled so did Google but yeah I think it just really shook people up and we talked about degree of hubris there for us markets to go what is going on it seemed a lot of a lot of investors got really spooked by this we can go into the technicality of it in a second as you say but I think the main thing here and the reason why Nvidia in particular but also broadcom and asml is worth talking about why they are because Nvidia is very obvious the Orthodoxy has been that in order to train larger more powerful llms you need to have more and more and more and more infrastructure and that's not what happened uh so Nvidia which is behind this kind of you know more is more it's very obvious why they dropped by 177% broadcom I messaged you Rory saying I don't understand why this is happening but they make application specific integrated circuits so they're kind of you adding into the infrastructure behind the AI software and similarly asml they make lithography machines for advanced chips so it's you a multi-tier incident if you like but as we noted these stocks are now rallying so everybody's managed to kind of know get some smelling salts and pull themselves together a bit yeah if your Twitter is anything like mine over the last day and a half you've seen a lot of arguments from people saying it's over you know this is the beginning of the end and other people are saying No this is the dip buy the dip you know Nvidia is gonna video is going to Rally in the next few days if anything this is good for NVIDIA uh this is the kind of chatter you get around a big event like this I think anyone who is professing to know which way that the stock is going to go right now I might caution them against that because I think that what we've been shown over the last few days is that it's anything but something I mean if you're taking your investment tips from Twitter X yeah I take mine from Blue Sky and Reddit personally I was about to say the exact same Wall Street bets on Reddit is the is the place to be if you want to lose money yeah hang on I'm just gonna have a rumage in my sofa for some spare change to buy in the dip as well for NVIDIA CU putting this in perspective you it's still a massively valuable company even after it had lost 70% of its market value anyway the bit that we've been teasing how did deep seek do it so what did they do Rory Ross mentioned that they had managed to sort of eek out all the power that's in the hardware that they had so how does that work so as you might be expecting it's a very complex picture um I've spent the last few days looking over the um the research PDF that that they released because this is an open source model anyone can check they're working and it's on GitHub you can run it for yourself and check it for yourself it's kind of a mixed approach the one that's got the most coverage maybe is that they managed to use we'll talk about this in a little bit as well relatively low power chips very very efficiently so there is an import ban on specific Nvidia chips for China and that's specifically imposed by the US for geopolitical reasons among them concerns that China could Advance very quickly with AI given the right Hardware so deep seek has got a around that partly by really really efficiently using memory so for example within the model they used 8bit floating Point integers inste of 32bit to to kind of explain what that means it's like if you are measuring numbers with 32 decimal points afterwards that's a very very accurate number if you're doing it with eight decimal points on paper it's less accurate but in practice that's still quite an accurate reflection of of what the number is it's a kind of good enough method and through a series of very very complex new approaches to neural networks to um reinforce learning deep seek also has kind of perfected a way to only expose the model if you like to a specific number of parameters they've basically figured out hey instead of doing these giant uh incredibly intensive training runs how about we just identify the most efficient things it would be to train this model on and train it on those it kind of cuts down that that need to do those incredibly Compu intensive runs something else that R1 uses is a mixture of experts system so what this essentially means is although the model is 671 billion parameters so it's quite a large llm only around I've seen a figure of 37 billion um and that I think that's for V3 it probably varies but only a chunk of those will ever be active at any point and the chunk that is active is the one that is pertinent to the task you're doing so if you input a math problem the model is able to go okay activate this part of the model which also means it's very performant without being clunky you get low latency responses it's just a very very efficient way of inferencing on the model so yeah a mixture of new algorithmic approaches new deep learning approaches and some Hardware trickery to kind of eek every possible bit of performance out of the model I mean it's one of those things as you're describing it it just seems super obvious that you should do it this way but but I guess it's one of these things where scarcity breeds efficiency many listeners will be aware that there are some trade embargos on China from the US and this does include some AI chips so officially deep seek used 2,000 Nvidia h800 chips which are legal to import for China however in 2023 so before this trade embargo came in langang W Fang who is the main financier for deep seek and is his CEO said that his hedge fund had stockpiled a100 gpus before the trade ban what is the importance of all this Rory so this is the this is the kind of controversial part the big stock market shock in the last few days the reason that the reaction has been so severe as you pointed out earlier as Ross mentioned with broadcom and you said with asml is it kind of decouples the more is more approach that companies have been taking to AI for the past few years so Nvidia has enjoyed such U Market success because it's kind of the sole producer of chips that kind of Meet the compute requirements to train AI that's been their main selling point is you need to keep buying our chips and you need to keep building bigger and bigger uh AI data centers and AI compute clusters to train uh fancier and fancier models in making the claim that using relatively underpowered chips specifically chips that China can can still import not even the Frontline Flagship chips that Nvidia is using today in making the claim that using just those deep seek has been able to make a model that Rivals some of open ai's best models it scared a lot of investors because it it throws into question that whole approach the reason there's there's controversy around this is some people are throwing into doubt the the honesty of the claim that deep seek is making that it did just use those underpowered chips so as you point out Yang Wen Fang made this claim in an interview on Chinese social media in 2023 was talking about stockpiling chips it's unclear if that was for this or um there have been some suggestions that that could have been for separate Financial activities because these chips are also really good for Predictive Analytics and in the finance sector we've also had some even bigger claims so for example Alexander Wang who's the uh CEO of scale AI went on CNBC to claim that deep seek had access to 50,000 a100 gpus that were stockpiled in direct contravention of the import ban he hasn't provided any evidence for that claim and there's a a kind of ongoing argument now with you know people saying okay maybe they have a larger cluster that we don't know about maybe they're kind of blurring the truth a bit when they they're saying it was just these 2,800 chips and also those who were saying okay but you you can't now come out and say actually it was all because they they had so many more of our chips than they're claiming it it comes across as a bit of a sour grapes argument from some of the uh big AI companies in the US it sounds to some critics a bit like they're mad and uh they're having to make up excuses for why actually the Deep seek isn't as good as they're claiming um so it's it's a very very tricky kind of spot [Music] so Ross R1 is the Deep seek product that kicked off all the excitement last week and this week V3 is the model on which it is based allegedly it was made for $6 million is that likely I think there is tactical framing here to an extent there's no doubt that this has been built you know at a significantly cheaper cost compared to you know what lot of organizations in America compared to a lot of organizations in the US and the West perhaps have been doing but I mean the V3 paper you know specifically mentions that those costs only include the official training of deep seek V3 and you know it goes on to quote excluding the costs associated with prior research and ablation experiments on architectures algorithms or data so there is an argument to be made there this has been made on a Sho string budget for sure but you know there are other Associated costs there so it's not as if it's you know just been bootstrapped in a garage so be like me turning around and going hey I got this pen free in wh Smith all I had to do is buy these other two pens get to wh Smith buy the random piece of chocolate that they want to sell to me and you know get back home again so you if we ignore the extra 15 20 quid that I spent then yeah it was free yeah and then you take into account Staffing costs and other various Associated costs there and then there it is going to add up but I mean regardless I think you know is still very impressive even if you were to combine all of those costs I would Hazard a guess and say that it's been still significantly cheaper compared to what the open AI has poured into model development or perhaps anth or you know any of your big tech companies in the US so Rory you mentioned earlier that Sam mman who is the CEO of open AI shut down the idea that you know anybody from China or India or you know any of the bricks could come up with an idea that would rival open AI this doesn't seem to be such a laughing matter now or or something to scoff at does deep seek scare AI I would be surprised if if deep seek didn't scare open AI to be honest I think you know we talk about some of the biggest losers of the past week in terms of market cap um that's what we can measure obviously open AI isn't isn't measurable in that sense but openai was undoubtedly one of the biggest losers of the past week mainly because while for the past year there have been models you know meta's llama um alibaba's quen which have kind of come for the raw benchmarking titles that open AI originally got with things like GPT 4 open AI then pivoted to reasoning and said oh about 01 03 we've got the reasoning Crown now we know that regardless of how much the budget was evidently on a lower budget because they haven't poured billions of dollars into producing deeps V3 you can do reasoning too and I think it's also open source which undermines open ai's route to profitability as well because I put out a post on LinkedIn on Monday saying when all the dust Setters around this there's a lot of questions to be answered but ultimately what open AI will be worrying about is if you can do reasoning if you can do good enough reasoning cheaply why wouldn't you and I think a lot of companies now will be going yeah well you know maybe maybe when GPT 5 comes out it'll be a bit better do we need the most advanced model or should we just get this free one off GitHub and it will do the job enough so I think yeah it really undermines their whole profit model and I think it also kind of blows a hole in their narrative I mean we'll get onto Stargate but the narrative that Altman's been pedling for the past year of oh did I say billions I meant trillions of dollars that we need to produce the best AI it really kind of blows a hole in that argument yeah because some of the most recent pronouncement that we've had from Oldman has been all like oh you know how we did this super premium tier actually we're not charging enough for it we're still losing money hand over fist and I think that's you know there's a bigger issue at play there as well around you know what Rory mentioned around models like meta have produced um with llama you know coming for open AI specifically and targeting those Benchmark wins everybody's been focused on each other it's just been a one-upmanship sort of situation for the last two years in Silicon Valley Google releases something and there's that sort of running joke where when you know Google's getting ready to to make a big announcement open AI comes in and you know takes the wind out their sales in the AI space and so yeah well everybody's been just hyperfocused on what's going on in like the Bay Area and you know there's been a lot going on elsewhere in the world there was a bit of excitement with mistl remember when you know they first burst onto the scene I remember a lot of people talking about wow this could be like a European counter to open a eye and and and anthropic and there was a lot of excitement around around that but then it all died down a little bit and now we've seen an even bigger even more impactful situation unfold and people are you know just losing their minds over this it's interesting that you use the phrase rest of the world so I've seen this described as the spotnik moment so from the Cold War nobody saw it coming that the Russians were going to be the first ones to get a satellite a working satellite into space will come on to living cosmina and astronauts maybe another time but uh that you it really has rocked everybody do you think that that is a fair comparison or has the media of which we are part got a little bit carried away I think it is fair I I mean Rory I'm sure you'd agree um you need to remember what happened after the Sputnik situation though it was again a game of one upman ship between the USSR and and the United States and the West I think we will potentially see that between you know us and chinese-based companies um in years to come the US came up with the Gemini program and the Apollo program and I think we're going to see that new model newer model out does it newer model rinse and repeat I think you know the concerning thing for perhaps us companies though is that look at what Chinese companies have been able to achieve on the budgets they have and they don't necessarily play by the same rules so I think think that might be a concerning sort of point for a lot of execs and a lot of AI companies and startups in the US yeah and I mean you may point at Gemini and Apollo and the Luna Landings and so on but I'd also point at the fact that for a very long time until SpaceX it was so rockets and Aran that were servicing the International Space Station so that's true worth adding here um I was at Google Cloud NEX last year and I specifically asked one of the lead devs Gemini if if they were consciously nodding to the Gemini program in naming it Gemini and he said that they were and it felt like a precursor to you know precursor to the the kind of advanced model which would be um which would be Apollo uh not not saying that Google would necessarily name it that but it's it's definitely meant to evoke that call it a Ryan instead yeah exactly I feel like all of the the Sputnik and and Space Race um comments I agree with Ross These are accurate they're they're more accurate for capturing the mood I think than anything because these tech companies have convinced themselves that they are essentially NASA you know that they are going to reach that kind of f moment of artificial general intelligence or or just the most profit that you can generate from AI I think what's a bit different here is that relative to the amount of funding and we'll get on to funding in a sec relative the amount of funding that's actually gone into the combined you know models of open AI anthropic even including things like mistol which have received billions in funding deep seeks even if you look underneath the claims and say well it's probably a larger number it's a bit more like if postwar Britain had turned around and said oh surprised we're sending someone to the Moon they're on their way right now you know and the whole rest of the world are gone what we didn't even know you had a kind of had a space program I think not to undermine the work that Chinese researchers are doing in AI but again to come back to that point of no one really knew where to look it's not like there's one standout Chinese company that is the equivalent to open AI that everyone was kind of waiting to see if they would get there first it's it's definitely caught people by surprise turning back again to open Ai and the question of whether or not they are concerned by Deep seeking what they've done apparently Mark Chen the chief research officer at openai has said that deep seeks research paper demonstrates that they have independently found some of the core ideas that open AI did on their way to 01 so that's their newest version but as you pointed out Rory there's no way to prove whether or not this is true because open AI doesn't share its research it is not open source whereas deep seek is very much hey come and look around you know check out what we've done Bloomberg has also reported that open AI is claiming that it has evidence once again it hasn't shared yet that deep seek used its models to train R1 so as I say you know open AI hasn't presented this evidence yet assuming it has it but whether or not it has you to play Devil's Advocate a little bit is this an ironic protest on behalf of open AI given that they have very much relied on open access to all the information in the world and have argued that everything should be available free yeah I think there's a bit of righteous indignation here shall we say you know open AI has been absolutely trolling the web for for a couple of years now and you know it's it's it's no secret either how many lawsuits are there on the go at the yeah I I mean I've I've looked at Rory there and I look of there's there's a couple but at the very least there's maybe a bit of a tactical play here from open Ai and seeing this is a really great achievement by them but we think we're actually the real winners here we're we're the victim but we're also going to be the winners here um very trumpy in in in its style I guess you could say we lost but you know we won bigly actually we won yeah I think open AI kind of has no other option here it's sort of what you would expect them to say I think it's it's worth mentioning here I mean in the introduction we kind of teed this up Stargate what is Stargate other than an excellent 1990s sci-fi film and television series so Stargate a landmark announcement I'll say because right now the actual reality the project is is very much in doubt it's an announcement made by Sam Alman CEO at openai Larry Allison CEO Oracle and masay yosi's Son the CEO of soft Bank announced that they are pulling together $500 billion dollar half a trillion dollars of investment um into a giant compute cluster to train AI models they were going to build it in Texas um they've announced 10 projects that are on the way right now and I think they said they were looking for an initial 10 more and these would essentially form a giant training cluster for Frontier AI models a lot was made about that and like I said in the introduction I had expected that that's what we would be talking about this episode because I hadn't seen um uh this is last week uh obviously the big reaction over deep seek hadn't happened yet so um it's definitely taking the wind out of their sales on that at the time there was a lot of back and forth over whether they had the money Elon Musk waited in and said I have on good authority that they they don't have 500 billion to spend on this project and yeah I I think musk also has obviously a horse in this race because xai has just built a it's its own giant data center with I think 100,000 h100 gpus um again going in on the argument that more is more that you need these giant infrastructure projects to leverage the the the future AI models so I I think think there is a there is a fear in those statements actually you you ended up you know just validating our model and our model was trained on this giant cluster and therefore you know we need another $500 billion do to to deliver the next thing it's definitely throwing that into doubt yeah I I do not know what the department of government efficiency would think about um you know sp00 billion while startups in China are bootstrapping this I think there' be a lot to see there so one person who can be said to have taken all of this quite calmly is satin Adella who posted a link to the Wikipedia page for the jevans Paradox on X following the market reaction and he wrote Jeff's Paradox strikes again as AI gets more efficient and accessible we will see its use Skyrocket turning it into a commodity we just can't get enough of do you think that this take is this a good take you he seems very upbeat about the whole thing for those who don't know jeevan's Paradox is the idea that as something gets more efficient while the short-term reaction might be oh we actually need less of this because it's more efficient what that actually does is it drives the market because it becomes cheaper to do the thing so it comes from a case in which a new coal engine came out so it was like oh coal demand will go down because it's it more efficiently uses the coal but what it actually did of course was massively drive up coal use because now it was cheaper to run things on coal so more companies were using it so what nadela is saying here is if it's cheaper to run AI that will actually increase demand for gpus and increase demand for data centers because it's more accessible and every company will want to be doing it um I think it's a hopeful reading of it I think we are likely to see something like this and and and a line that we have to draw here is between train and inference right deep seek is cheaper to train it's also much cheaper to run than a lot of the models and and the majority of AI data centers aren't there to train models they're there to influence them they're there to run them whenever someone puts a a prompt in so there will be always inferencing demand even if AI becomes less compute intensive to train once it's trained you still need to run it somewhere there will always be those influencing demands and some I did just want to mention which is an area of interest to watch and maybe something we'll do a podcast on um in the coming months is the while deep seek used some Nvidia chips depending on who you ask they either use a100 or H 800s but they definitely use Nvidia chips to train it they're not influencing on Nvidia chips they're influencing on Huawei 910c chips which are made in China and they are while they're not entirely reling on Chinese supply chain yet because as we mentioned earlier um lithography machines come from Europe China is working on that too so there's also a bit of Home Pride here and a bit of National Security implications for China where of course it wants to secure its own supply chain for chips and not at all be reliant on companies like Nvidia so the inferencing picture is is interesting here because as Nadella has pointed out the inferencing is is going to continue to drive growth in the AI space that's kind of where we'll need a lot of this investment to be going anyway and also deep seek is already proving an interesting model for Chinese Chips being very much good enough for um inferencing an advanced model well unfortunately that's all we have time for today but Ross thank you so much for joining us great to be back great to be back at the office with Rory as well a rare occasion as always you can find links to all of the topics we've spoken about today in the show notes and even more on our website at itpro.tv [Music]