How AI code is changing software development – interview with OpenAI

IT leaders are carefully assessing the extent to which AI-generated code can make a difference in their business. On the one hand, AI developers promise their tools can enable faster code deployment and free up time for developers. On the other, it can be difficult to know where to start with AI tools, particularly if you want total reliability in your code. How can enterprises make best use of AI code, and what do these tools mean for developer teams of the future? You're listening to the ITPro podcast. Hi, I'm Jane McCallion. And I'm Rory Bathgate. In today's episode, we're joined by Colin Jarvis, Head of Forward Deployment Engineering at OpenAI to discuss internal code generation. Colin, welcome to the show. Thanks for having me on. To start with, could you tell me how much of OpenAI's internal code is now generated using CodeX? Yeah, great question. So we've seen the amount of code written by our engineering team increase, I think, almost like 7x in the last few months. We have a chart that we look at where our headcount has largely grown linearly, but the amount of code has grown exponentially. And we've seen that across pretty much every engineering team. But especially in the Forward Deployment Engineering team, I think we've really leaned into CodeX because it's made it, I guess, quicker to deliver for customers, but also to do a lot of the internal work that we normally didn't have as much time to do. In terms of tracking, you said 7x there, the amount of code generated, is that on a per-token basis? Or is that actual deployed code that's being measured there? Yeah, good question. I've seen the chart on it. I'm not sure exactly how it was measured. But I know within our team, we look at the number of PRs that are raised and the amount of lines of code that are edited, for example. And we do, I guess, at least within FDE, shift a significant amount more code as a result of using CodeX. We generally integrate it in our CI pipelines, so we generally see it come through as code that needs to, at times, be reviewed by us, but generally comes through in that respect. Does OpenAI use CodeX for all of its major product launches or not? I can't comment on every engineering team in OpenAI, but the general principle that we work to now is, if you can use CodeX, do so. And remove yourself from the loop of having to manually write code. So I would say a significant proportion of our product code is likely written with CodeX. I know, again, speaking to the F4 deployed engineering team, in most cases, we use CodeX as a core building block that we design around with our applications now. So typically, almost everything is done with CodeX, although in some cases, just due to the, for example, some customers require us to write completely within their infrastructure, and some of them don't use CodeX within their infrastructure, so in those cases, we can't. But in the majority of cases, we do. You mentioned, obviously, the rate at which developers are using CodeX is increasing, and the amount of different products and tools that OpenAI is using CodeX to create is increasing. Is this overall speeding up the rate at which OpenAI is shipping new products? It does often, the claim around AI code is that it can increase that kind of time to deployment. Yeah. So I view this in a couple of ways. So I'd say, certainly, it allows you to write a lot more code. I think what we find in, again, I'll maybe frame this using the F4 deployed experience of this, is that in the F4 deployed engineering team, we created a kind of application that allowed you to view your concurrent CodeX sessions. So what that means is that we're allowed to parallelize a lot more development work, which previously would be single-threaded. So that means that, yes, we can write much more code. However, what we've also had to do within F4 deployed is write a large amount of scaffolding to go around it, because that obviously raises the importance of reviewing that code and having the right auditability and traceability framework. So what we typically do in F4 deployed for both our work and for our customers is implement a kind of validation framework around CodeX that we can get that extra throughput, but we also have the right kind of guardrails around it to make sure that we still trust it. And if I make, I'll use a concrete example there, is previously, let's say, on one of our semiconductor customers, we had an application that previously would have been five custom agents written with the agent SDK. Now, today, that would maybe be one CodeX agent with a bunch of skills. Now, previously, we would have had five sets of evals for those five different agents to make sure that we trusted them. Now, we will still have a comprehensive set of evals, but those will be more end-to-end tests where we will trust CodeX to do a lot of the inner decision making, but we have solid checks to make sure that based on this input, we're still getting the expected output. So that control framework is very important to get a reliable advantage from the large amount of code that CodeX allows you to write. When it comes to the way that engineering teams are organized, does the increased usage of AI for code generation mean that there has to be some level of reorganization, or can you keep the structure largely as it was and your productivity is just amplified? This is a really interesting one, because this has affected both how we've organized the forward deployed team, but also how we've seen our customers reorganize as we work more with them. So I'd say the first thing is in forward deployed, it's really pushed to the fore. The knowledge of the business domain that you're working in has become much more important, because the amount of time you spend architecting and building the actual software side is proportionally less. That doesn't mean that we have no out-and-out software engineers in the team, it just means that we have more domain specialists. So a forward deployed squad, for example, we have working on life sciences. Three of those forward deployed engineers are more of a scientist background, and then only one or two of them will be out-and-out software engineers. On the customer side, the interesting thing that we've seen is that rather than have a central IT function that controls all the AI software that comes in, AI software can be a lot more customizable. Because imagine if all the software is is just CodeX with a set of skills, and then you deploy it into different departments, and those departments might have different actual systems that they want the CodeX agent to make use of. So there's a bit more customization out at the edge, in the individual departments. So what that means is one of our largest customers, where we deployed 11 use cases to production over the last nine months, they reorganized their central IT function. They called it productivity engineering. They were these AI engineers. Instead of all sitting centrally, they started putting pods of them into all the departments, so that you'd almost have a central AI software application. Let's say it's like an intelligent debugger that goes and tries to fix problems in your PRs. Then they would deploy it to each department, and those productivity engineers would customize it for that department. So they'd maybe add evals. They would create tools for the specific systems that they work with. For example, one team used GitLab, and the other one used Garrett. So there's some customization required there, but then they would then release it to those departments. So it creates this unusual, I think, what would have been unsustainable in a previous age, which is you have these customized versions of the central product in each of the departments, but it means that it's much more personalized to those individual teams. We've spoken about it so far from a strategic perspective, and obviously that's what we will continue to, and that's important from a listener point of view. But just zooming in on the technology for a second, there is a difference between deploying tools that are kind of user-driven and deploying agents. And obviously for something like Codex, there is quite a heavy agentic element there. How much does that require teams, especially in customer organizations, to kind of get to grips with the new dynamic of work being completed more kind of proactively in the background? Yeah, it's definitely different. It's like a very different operating model. In fact, we had a moment over the last six months where I'll frame what we did, and then I'll talk about how it changed the behavior. But basically about six months ago on this project that I talked about with 11 use cases, one of the engineers bet on Codex early and basically designed around it. And in February of this year, their productivity kind of took off relative to the other developers on the team. So we then created a kind of bootcamp session to retrain the whole team to design with Codex first and almost like, what are the things you need to sort of step back and let Codex do? And how do you design the application to be effective around that? So I'd say the key bits are if I can reduce them to a set of primitives that you need to trust an agent as a developer, and then I'll talk about the user side. As a developer, we talk about grounding, of having the right principles or constraints that you're giving this Codex agent. There's a set of skills, which are the set of tasks that you carry out. But these will be consistent, even if the tools underneath change, the skill should still be a relatively durable thing, which you can rely on. And we've seen some IT organizations start to almost publish approved skills for doing things like unit test generation and this kind of thing. And then you have the tools, which as you say, these are more fluid. And I think depending on the set of end users, you'll need like the tools to be different. I think this is an area where we're actually working on something internally at OpenAI that we're calling like the context layer, which is almost like a layer in between the skills and the tools where if you could imagine, I could hook up like SharePoint and Databricks. And then we would try and like almost comb those and create a like translation layer to explain those to the Codex agent so that it could just use skills much more fluidly to like, and almost like reduce this amount of customization of tools that's required as you like scale these systems. But I think that's like where we want to get to. But I think for now, there is a bit of like, there's a bit of customization required on the user side to like add that grounding, add that context so that the Codex agent knows how to make use of like your specific context. And one interesting way that we saw that happen is we tried to implement this as a kind of like memory where like, as you interacted, as you wrote PRs, this kind of thing interacted with the model, it would like write facts down about your department and how you how you describe things and the terminology you use and how you calculate things. And then what we found is when we tried to expand it to another department, they got like worst results because they just worked in a very different way. So there's like almost the learning there is like context is very specific. And I think what people have to keep in mind is when you develop a piece of AI software, you do have to consider this like personalization element for each subgroup of users that are going to make use of it. Bearing all of that in mind, how mature is customer adoption of tools such as Codex and AI cogeneration generally in your experience so far? I would say it depends on the like the user base because interestingly, like within our team, because we're very confident Codex users, we've started to use Codex for everything. Like we use it for knowledge work. I used it the other day. I turned on the computer use mode and asked it to clean my LinkedIn inbox and categorize all messages and like tell me which ones I need to respond to and which ones are inbound sales and this kind of thing. I think like Codex can do a ton of things. I think what we see is that several of the customers that we've seen need to be convinced first if they're not software engineers. I think software engineers very quickly like see the value and just adopt quickly. I think with others like with this life sciences company we were working with, interestingly, they didn't initially like adopt Codex. But when they saw we built this document offering application that would like review regulatory documents and give suggestions on how to like tighten up the wording and how to like refresh the data and this kind of thing when a different when changes came through. And we saw a corresponding like big adoption spike of Codex in the business. When they saw that Codex was capable of doing these very difficult tasks, they were like oh well maybe for all of like my day-to-day stuff I'll just start using Codex. So I think the taking a step back the general question is like we're seeing people a lot of people adopt Codex. I think our weekly active users is as high as it's ever been like we're and there is like an exponential rise in the amount of people using Codex. But I think that is definitely led by like software engineers right now and we're what we're like hoping to see and what we're seeing internally at OpenAI is like that's starting to extend into knowledge work as well. So people are kind of using Codex for everything. In general terms, would you expect customers to use Codex to the same extent that OpenAI does or would that be unachievable? Obviously you've got clusters and rate limits that are in excess of what a typical customer might come across and obviously you're you're perhaps also a bit more pioneering in this area just by virtue of who you work for. That is actually an interesting question that we discussed a lot internally. Like I think software engineering is a clear case where like you know the amount of tokens you generally the amount of like work you're going to do. I think interestingly we're starting to see that in some other professions. I think the scientists are an interesting one because they have all these papers that they previously just don't have time to comb through and then once Codex makes them accessible then they do start to churn a lot of tokens to make decisions. So I would say that one of the purposes of forward deployed engineering is actually embedding with all these different sort of professions and like understanding you know like what information do they need accessible to them to do their jobs and then what does that what does that look like. In most cases that's resulted in quite a lot of tokens but not in every case. I think in some cases it's more like you do an analysis and then it's a lot more like interacting with people like for example in like a finance team or something. But I would say if I look at like the kind of top three that we're working on right now which is like we're working on a lot of a lot of semiconductor like verification design that's very much like a very token heavy industry. Life sciences as well like drug discovery clinical trial like document authoring that's also very heavy and like token processing and if I look at one of the other ones actually maybe one of the ones that's not. We worked with an agriculture company to do like a customer success operation that was basically combing all of the data that these farmers were processing and then telling them how to use their equipment better. That was maybe a lower token operation because you imagine it's like you process it maybe every like couple days or week and then give them some useful insights and then they change their behavior but that's changing behavior out in the physical world so they almost need to like go and then do a couple days of farming to actually like impact the to require more tokens to be churned. So I think not every industry will be like there will be a correlation between number of tokens to economic output but I think um for like a lot of the more complex uh like knowledge work ones that we've been working on like kind of drug discovery and semiconductor design those those have maintained it it's just required more I think like building to like bring the information to the model in a way that it can then you know do all this processing. Moving away from that a little bit we have seen you know in the press we've reported it other outlets have reported on the idea that some organizations are cutting back on headcount and employing AI instead and you know various people various talking heads have spoken about this. Interestingly this week Gartner released some research where it found that 80 percent of companies deploying automation tools and cutting staff numbers aren't seeing corresponding returns on investment. In fact I believe that those who kept headcount more or less the same and employed agents or other kinds of AI saw better results. Why is that in your opinion? AI being brought in is very much similar to a lot of the other like technological revolutions where we're seeing new jobs get created other jobs sort of start to change a lot and I think what we're seeing among our customers is like the folks who use it to increase the output of their existing specialists and their existing like experienced staff are the folks who are seeing the best output and I think the folks who see it as an excuse to as like sort of a race to the bottom are the people who struggle a bit more because I think a lot of what I've talked about in forward deployed is like the importance is like domain knowledge or the importance is like context and organizational context and I think if you remove that the models are not nearly as effective like it's really like a kind of confluence of human knowledge of their domain plus like the throughput that these models bring and if I give you an example of that at this semiconductor company there's a task called kernel optimization where you have like a design of a kernel and like the sort of the fun part for your like most experienced kernel like chip engineers is designing like the new kernel and coming up with like you know here's the hypothesis let's build it and let's like see whether it has the right set of trade-offs and then there's a process called optimization which is kind of a thankless process of I have some criteria that I have to meet and I have to like kind of twist my design to meet that so that we can ship it and what we did at this customer is we tuned a model that was very good at the optimization stage and so the chip engineers all like loved it because it basically meant that they did the creative side of designing this chip then they like sort of set it off overnight came back in the morning and then they'd have like a 70 speed up of the chip and they'd be able to just ship it 70 is the best one we achieved some of them was like 10 20 but the fact was that it sort of like took the work that they like really didn't like doing and it freed up their time to then think of you know the next five chip designs and I think the way that that company like what they chose to do was not reduce headcount and just like you know let those people produce more and I think we're seeing a really positive like output from them as a result so I think that is like kind of the summary it's like the people who are seeing this as an opportunity to like almost produce more with like with uh with your best people are the people who are seeing the best results from uh what we're what we're seeing in You mentioned there that you kind of fine-tuned that optimization model and earlier you you had said that you know sometimes it takes more work or less work to bring that context to the model in that case is it a case-by-case situation where in some companies models such as codex can pretty much zero shot whatever problem they're looking for but in other cases people like your team in FDE have to have more of a hands-on kind of fine-tuning approach this is definitely the case so like one of the things we say about we say to the FDE team is like our job is to basically raise the floor for the market of like what you can do with the off-the-shelf tools so I think if you can solve 90 of the stuff with codex that is actually the best possible option for us like hopefully you don't you don't need us I think the FDEs do focus on the like I guess uh probably two buckets of tasks one of them is where something is possible and it's repeatable but you need to have a lot of scaffolding around it to like trust it and that is in enterprises a lot of these like kind of workflow automation tasks where like codex could probably zero shot it but if it got it wrong like you know three and ten times you maybe wouldn't trust it enough to use it and like in those cases the FDEs try and build like kind of a platform or like scaffolding so that you can reliably do that and then again we step away and just kind of let you let you do that and hopefully eventually it finds its way into OpenAI's product offering so that next time everyone can just do that easier the other thing that we focus on which this fine tuning touches on is where the models can't do something and like these are like these very difficult tasks like kernel optimization or like drug discovery or something like this and in those cases what we do is we work with our post-training team usually and like we will basically build the application get the model to the stage where we like we know how good it is at the task and then we will work with our post-training team to improve the model so that it can it can solve the task and in so doing hopefully it it solves this customer's task but it also raises the like the general intelligence of the of the model and like that's sort of I guess like in summary where we focus on with with FDE is like you know making like the solved problems repeatable and then making the unsolved problems hopefully solvable. We're seeing an acceleration in the development cycle for frontier models you and your things are increasing you know we cover model releases as they happen and they definitely are speeding up in your conversations with customers is there ever a line between thinking okay well you know we can we can take the time to fine-tune this over the next few months and also knowing well there's a there's another model coming in the pipeline that might be able to zero shot this in the in the same amount of months are those conversations that you're having with customers or do you have to kind of focus on the the currently released always yeah yeah this is definitely something that we need to that we need to consider in FDE I think our preference is that our customers don't need FDE and like they can just they can just solve it with the model and if we do think that it's in like the that is kind of just around the corner I think then we will tell our customers and like and not work with them but I think in the majority of cases that we work on in in FDE they will be things that I think are like specific enough to enterprise that like you know it's likely that we do need to push the model to get there and I'll give you like one example which is sort of nuanced is like we have this real-time model that we released a while back it's like the first like audio in audio out you know very low latency model and a lot of people use it for like conversational use cases but if you use it like maybe a year ago if you tried to use it for something enterprise ready where it's like it requires very accurate instruction following very accurate tool calling it maybe you you wouldn't have the level of reliability that you would want permit and so one of the things that we did was work with our post-training team to give them very representative evaluations that show like the level of consistency that is required for enterprise and then they were able to like sort of pivot the model to get there so there is a little bit of a case of like oh yeah the model is going to overtake what the FDs are doing let's like not bother but in some cases it's actually like post-training a model of this complexity is hard and like sometimes it's it's FDE finding like these cases that like we need to make sure the model is actually the model evolves with what our customers need to do those are probably the ones that we try and focus our time on most of the time Colin another thing that we are hearing about when it comes to AI is what happens to entry-level positions and people just starting out in their careers when it comes to junior developers what steps are you taking to ensure that there is still an incoming talent pipeline this has been a big focus of ours because we actually the the number of like entry-level people has increased in the FDE team over the last three to six months and the reason for that is that we're finding like with the right sort of enablement and like the right like you asked earlier about uh you know how to think about working with agents like if you kind of like bring people up in that like mindset of like I've got agents helping me with everything like how would I do things differently they improve very very fast and they end up being as productive as like some of our more experienced staff like very quickly so what we developed in the FDE team is something we call the FDE boot camp which is like a one-week thing that we run every like six to eight weeks for the team because there's been a lot of growth in the team so we're having to like update it and interestingly we find ourselves updating the curriculum every six to eight weeks because like the meta changes very quickly but um what it means is that we're able to bring on more entry-level folks because we're kind of like we bring them in we kind of like steep them in like the way to build with codecs and then what we quickly find is they actually become like just native natively like agentic engineers versus the rest of us that kind of did it the old way and had to kind of learn it so definitely in FDE we're seeing like if anything an increase in the number of entry level and in fact we're just looking to take on our first couple like interns straight out of uh out of university which we haven't done before and it's it's because of the positive signals we've got from this the entry-level folks that we've been bringing on staying on the topic of talent and employment is great use of ai for coding changing the hiring process how can candidates demonstrate that they have a firm grasp of ai tools or you know based on somewhat on what you just said is that not necessary can you learn when you you enter into the workplace yeah this is a very a very like topical question because we actively change our hiring process for this very reason i think we previously had a like technical test which you know you have to solve a problem and what you could find is that actually like codecs could probably zero shot that problem with like minimal with minimal prompting after a while so what we've tried to do is see like more of a dynamic interview process so we have an interview that we call the decomposition and that's almost like you present the person with a problem and you allow them access to like any tools they want to use if they want to like use jackie or something during the call like they can but they have to like work with you to understand your problem and if they don't ask the right questions and they don't try and like engage with you as a person and understand your problem they're not going to pass and i think that is sort of like more what we look for is like you know once they have like sprint through maybe a first design if they can use ai to do that great if they can't also fine we'll like try and train them but they do need to be like a kind of empathetic person who's able to like connect with a business user like understand their domain to some level and understand what's important to them and then that is probably like the more important thing in uh in the in the in the forward deployed team at least do that we're seeing we've spoken quite a lot about human tasks and um ai or maybe agent tasks as separate up to this point um obviously when it comes to something like debugging versus generation or deployment versus debugging different organizations will have different i guess appetites on this to what extent if at all is it possible to go from completely from zero to 100 with with a tool like codex at the moment and is that desirable for customers or are they looking to involve humans at some point in the process i can speak mainly to enterprises because those are like the majority of the the fde customers and i think in the majority of enterprises it like there is still quite a lot of like human touch required in many of these processes it's almost like we always describe it as like there's like a workflow which has like some determinism and then you have these like agentic kind of stages in that workflow so like in this there's a bunch of work that needs to happen and if i give an example it's some kind of semi high-stakes use case like maybe we're uh maybe we're reviewing uh payments transactions to see if there's like a possibility of uh of fraud or like money laundering or something like if you wanted to involve an agent in that process you definitely couldn't just like trust the agent uh like end to end you would want like some kind of steps of human review so often what we do is we like look at the existing process and then we look at like what would a fully agentic thing look like and then we work with the customer and kind of workshop out like where they're going to introduce gates where like where a human is going to is going to review and in that instance i just gave you it might be that like it maybe gathers the information and comes up with like an initial an initial recommendation with what a human then needs to manually review and then like and then sign off and then maybe there's a second review stage or whatever it might be but this is generally how we how we sort of see it the interesting thing is i think it's the one thing that has happened is it's become much quicker to get started because i think with things like computer use you imagine like previously you couldn't uh you couldn't do something with an agent until you had like apis in place to imitate what a person does whereas right now what i can do is just like record my screen go and do a process feed it into codex codex could probably be like okay this is how i would do that process and then we then reverse engineer that steel thread into like a durable kind of application where we have the right trust layers built in um within open ai um either either in your team or or more broadly is there less human interaction or is there a similar level i may be biased in this because we're like a very i think in fd we're always sort of like we're the closest to like you know domain expertise always need to talk to people this kind of thing but we find we actually we actually interact more as people and and the reason why is because i think i referred to the start about like the application one of my team built to like kind of almost a viewer for all their parallel codex threads like because codex is doing a lot of work for you all the time there's actually that actually frees up time for kind of like human interaction and doing like the kind of the high value like uh things like you know just getting together and brainstorming as a team or like you know working a problem together well codex is like doing a lot of the like grunt work in parallel so um definitely definitely my team i've seen like the level of human interaction is not a has not decreased and in fact has probably increased i can't comment to like the the other teams uh yeah i think that's like generally my observation so far carlin looking forward a little bit now what does the future of software development look like do you think so i've talked a lot about like i guess the skills that are getting more important and i think if we accept that a lot of the like plumbing of software engineering like you know writing tests making sure that everything is like verifiable and reliable are things that we start to just take for granted then it becomes much more about like creating software that is fitted to like the the needs of each like individual set of set of users so when i described like in fde domain expertise was becoming much more important i think that is that is definitely the the trend that i that i'm seeing is like we're not taking software engineering for granted there's still the architecture and like creative and design side from there is still extremely important but you can do so much more with just like a with a detailed knowledge of a domain and ai to kind of help you along so i think um as we move forward i hope that software will will be able to solve like more and more complex problems because you'll be able to like build on us on a more solid foundation like you'll take more of the plumbing for granted and you'll be able to focus more on these really hard problems and giving an example of where that's like where i've seen that physically happen is uh at this semiconductor company like these 11 use cases that we put to production if you looked at them you would recognize all of like almost all of them as being kind of like software engineering adjacent like you know we mentioned debugging earlier this is one of them you know doing smoke tests the unit tests this kind of thing um but if you look at the next generation of use cases we're starting to take on at this company they're like starting to trend more into like physical design of chips which is like this is a really hard problem and uh previously like we there was no way we could do it with ai because there was so much that we had that had to be true for us to like get to the point where we could work on this higher order problem and now like i think what we're finding is now that we've built all this foundation we're now able to move to these like hard problems and i would look at like all the domains where ai still hasn't made as big of an impact like manufacturing like these like physical sort of ai use cases and um i would say like those are going to be like it's going to be easier to build software in those domains because you're going to spend more time on like what is the specific niche user base need and you're going to spend less time on like okay it's going to take us 12 months for this to be like a reliable piece of software so we need to we need to like plan around that that would probably be the biggest impact that i expect to see and also like what i'm starting to see since like january february this year is like just less time is spent on the kind of like productization of the software and more is spent on like what is the right user experience what is the right like set of features for the uh for the users well unfortunately that's all we have time for today but colin thank you so much for joining us thank you so much for having me on it's been an absolute pleasure thank you so much as always you can find links to all of the topics we've spoken about today in the show notes and even more on our website at it pro.com you can also follow us on linkedin as well as subscribe to our daily newsletter don't forget to subscribe to the it pro podcast wherever you find podcasts and if you're enjoying the show let us know by leaving a rating or review i will be back next week with more from the world of it but unfortunately this episode we are saying goodbye to rory rory thank you so much for being my co-host for the past three years it's been an absolute pleasure i hope that the listeners have enjoyed the podcast as much as i've enjoyed making it and i'll be i'll be listening in keeping up with it until next week goodbye goodbye