How to Read and Analyze Medical Research Papers Like an Expert | With Veeral Tolia

If I already know the answer it's hard to evaluate that so I have to kind of understand how the how they arrived at the answer before I can really evaluate whether the answer is useful for me or not. Right. So I like to look at the methods. So all in the abstract or you getting into the nitty-gritty? So that I typically I mean I think we've talked about this before but most people will mostly read the abstract. Yeah. And I will I'm I'm guilty of that as well. Hello I'm Dr. Tala. Welcome to Tala Talks. Thank you. Today I am so excited to be talking with my friend and clinical neonatologist Dr. Viral Toile. He is like I said an excellent neonatologist but he is also a leader in neonatal research. His current role apart from being a neo is that he is the director of the clinical data warehouse of pediatrics research. So he's involved with really a lot of writing and analyzing of data and papers. More personally he is also the person that I always call when I'm having difficulties understanding something in research or if I'm actually reviewing something and I want to get to the bottom of something. He has taught me probably more about research and how to read an article than literally anybody else in my life even though I love you all all you other mentors. But so today what he's going to be talking about is really how to read a paper and how to kind of think about how the paper was done and how to analyze it like talking about the hierarchies of different ways that studies are done. So thank you so much for being here today. Let's dive right in. When you first open a paper so you've opened the journal or opened it up online and you see a paper what is the first thing that you concentrate on. I think it's really easy to go straight to the conclusions. Yeah. But I would caution you against doing that. The conclusions are the ones that's what the author has decided to tell you and that's their interpretation of the day. Okay. What I find much more helpful is starting just at the top of the abstract. So why are they doing the study that typically is contained in the background statement. Some journals will use an objective statement like what was the objective of the study and then before. To me it's a little bit like if I already know the answer it's hard to evaluate that. So I have to kind of understand how the how they arrived at the answer before I can really evaluate whether the answer is useful for me or not. Right. So I like to look at the methods. Okay. And most papers in the abstract will explain in the methods type of study it was the patient population that they looked at kind of the both the methodology of how the comparisons were made but also if there's any specific things that we need to know about about if it's a blinded study if it's a placebo controlled study if it was done in one center versus a bunch of centers all that stuff is generally contained in the methods. Okay. Then once I have a sense of what they were planning on doing then I like to read the results and the conclusion. So all in the abstract or you getting into the integrity. So that I typically I mean I think we've talked about this before but most people will mostly read the abstract. Yeah. And I will I'm I'm guilty of that as well. If if I decide you know seminal papers papers in areas that I'm very interested I'll of course read the whole paper. But if I just kind of want to know I look at the title I say hey this is kind of interesting. My three questions are what question do they intend to answer how do they intend to answer this question and what was the answer that they came up with. Okay. The fourth question that they can't answer but is the most critical question is how can I use this information for the care of my patients at the bedside for supporting my research and those kind of things. Okay. So let's talk about that before we jump into anything else. What makes you decide when you read a study that it is clinically applicable. That's I think the most important game in reading any sort of medical literature. And some people get confused they think that oh if something is statistically significant and we'll talk about that in a second then it's clinically significant. For me the most important thing is how can I apply this finding which happened in a population of infants that does not does not include my baby because I'm taking care of my baby right now. How can I apply that knowledge to help me take care of the baby that I see in front of me. Okay. So the most important thing in terms of that is something called study generalizability. Okay. Okay. And generalizability just refers to this concept of they've done a study. They've done it in a specific population. They've done it with specific interventions and outcomes. How close are those parameters to the patient that I'm looking at. Okay. Great. Go ahead. No. So I was just going to give an example. So for example every single baby in the study that they're doing is like on the jet and on antibiotics and didn't get steroids or whatever and then suddenly you have this completely different population where you've got your babies on CPAP. All the mommies got steroids. You did Lisa. Is the same intervention as generalizable as those babies. To give an example. That's right. And actually it gets even more complex because in some cases most of these studies are done in large volume centers. Right. In urban centers where there is an academic institution and something like that. And a lot of times people are not practicing in those settings. Okay. Or they're practicing in patient populations that don't necessarily reflect the same patient populations that the study was being done. Right. Or here's a really easy example. In order to if you're talking about kind of the most critically sick infants that we take care of it's those 22 week babies. Right. And some of those babies don't make it into the study because they died before they got there. Yeah. Right. And so when you're talking about study generalizability it's all of those things. But then there's additional barriers that stop babies from being involved in study. So in most studies you have to have informed consent before you can enroll the baby in a trial particularly if it's trial that's comparing A versus B. So informed consent where the parents have to sign often before the baby is even born. That's right. That they would agree to getting randomized into either getting a treatment or placebo or whatever. That's right. That's right. So there's already you're saying there's already a bias that a lot of mothers might not get consented in time especially the really sick ones that are going to deliver precipitously or there's a group of patients that just say no. Absolutely. Absolutely. All of those things reduce the generalizability of randomized trials. So when you were saying generalizability I was thinking more like is this still that something that I want to do because sometimes I feel like I read trials where they're like okay if you for example the vitamin A but you know for BPD if we give vitamin A you know three times a week or whatever we used to do like IM three times a week then it will decrease your severe BPD by some small amount and at some point and then obviously when they went and followed those babies up like neurodevelopmentally it wasn't a big difference but at some point you have to make the decision that is this amount of harm or intervention to the baby worth that tiny improvement like is that a clinical significant improvement if you know instead of being on 28 you're on 22 or whatever it is. Right. Is the juice worth the squeeze? Yes. Yes. Okay. Is that a is that a professional term? Yes. Absolutely. Although don't squeeze any babies. Yeah no. So do you always think about that as well? Yes. So there's a couple one of the things that has evolved over time is that there are a couple of concepts that directly address that. The one that many people might have read is the number needed to treat or number needed to harm and what those are are estimates that you have to treat x number of babies in order to see a benefit. Okay. Okay. But those are kind of misleading sometimes. Yeah. So in the example that you gave of a vitamin A for reducing chronic lung disease the number needed to treat which means the number of babies you need to treat to prevent one case of chronic lung disease I think was 16 or 17. Yeah. In the original trial and that seems like that seems like a lot of babies that are getting IM shots three times a week for eight weeks in order to reduce one kid from getting BPD. But actually a number needed to treat of that is actually pretty good. There are some things where the number needed to treat are in the hundreds. Okay. Okay. So even though we have those kind of additional tools to kind of quantify the clinical benefit it's still up to the individual provider to make that differentiation. And you know for you and I it's a little bit easier because we treat a lot of patients right. So we both probably treat a couple of hundred babies that are at risk of BPD every year. Well and that's something else that I want to bring up and the number needed to treat that obviously the more common the disease the more likely it's going to happen. So logistically you would have to treat less babies anyway to prevent one of them. Absolutely. I mean it's something like neck that you know we're all aiming for like you know 2 3 percent but say the average is 7 percent neck then it's obviously a lot rarer than BPD. You are going to have to give probiotics to a lot more babies to prevent one incidence of neck. That's right. The efficacy of whatever it is that you're talking about intervention has to be higher in order to reduce the number needed to treat in small in less frequent diseases. So basically you would accept a higher number needed to treat in a more rare disease. In a more rare disease or if the disease is really devastating. Or if the disease is devastating because we'll do anything to stop that. Right. So death like I would take a you know I would I would take a very high number needed to treat. Yes. Because we're preventing death. Yeah. You're talking about like ear infection progressing to where you might need two rounds of antibiotics. Maybe. Yeah. Maybe the number of aggressive is less. OK. Yeah. All right. So that's amazing that you're looking at that like the clinically applicable aspect of it. Yeah. And that's a big part of it. Like but studies don't normally put the number needed to treat. Yeah. So it's actually usually a high quality study will give you all the information there that is easy to calculate. But one of the really critical subtleties about the number needed to treat is what you've pointed out that it's dependent on the background dependent on the disease incidence. Yeah. And if you're in a population with a different disease incidence then your number needed to treat is going to be different than the trial. Right. So I'll give you a great example. In very preterm babies fungal infection which we're both scared of. Yeah. Is very very bad. But there are some units where the the risk of that is very very high and some units with risk of that is very very low. Yeah. So depending on how closely your unit compares to the original trials in terms of how common fungal invasive fungal infection is you might have it individually a different number needed to treat. Yeah. Then the trial. Yeah. And we've discussed this in the past depending on your underlying fungal rates. You know in preemie babies you might choose to give prophylactic fluconazole or whatever or not. And that isn't just dependent on like the place you're in. It's also dependent at the time. That's right. We all know that there are like waves of getting more fungemias getting more MRSAs or whatever in the hospital. So that's also a really important part of that. It's all part of generalizability. Yeah. OK. When you read a study are there any times that you look at it and you're like like complete red flag. Like this is ridiculous. So there are several red flags that I look for. The first is if the primary outcome is not an important outcome. Like give me an example. So a lot of studies now will look at neurodevelopmental outcomes. Yeah. This is such a tricky one though. Yeah it is. But if you find a neurodevelopmental outcome a lot of times that's measured with something called a Bailey exam where there's a two point difference between intervention and a control. Right. I don't know that I think that that's super helpful. Yeah. OK. Similarly if the outcome is not something that I care about that's like an important red flag. Give me give me a example of a study where you don't really care about like or you don't really care about a primary outcome. So there are some studies where the clinical outcome is not really helpful for the patient. So like the days until a study lab becomes normal. OK. So not relevant at all for when the direct bilirubin gets back to normal if the baby's eating and gaining weight and whatever else anyway. Right. And let me distinguish here that's not a red flag. The study was poorly performed. It just means that it's not really applicable to what I'm interested in. Yeah. What about a poorly performed study. Are there any red flags where you're like. Yeah. So there one of the things with large randomized controlled trials is that nowadays most of those need to be registered online at clinical trials dot gov. It's a Web site where you've registered not just the trial plan but your protocol what the outcomes are supposed to be and how you're going to evaluate those outcomes. And it's a big red flag if the study presents outcomes that are not what you said that they were going to be in the in the registration and clinical trials dot gov. OK. So they've kind of changed like what they're going to publish based on what they found and then they're going to make it look like oh this is what we were looking for all along. We got it. That's right. Brilliant. And we're brilliant. So one of the characteristics of of studies is that they're expensive and they take a lot of manpower and a lot of time to do. So you're going to collect a ton of information. OK. And in most cases in high quality studies you're collecting the information that you really want to know but you're also collecting a lot of other information. Yeah. Right. Because if you're going to spend 10 years and 20 million dollars doing a study you want to make sure you get as much utility out of that as possible. Well if they come out and they publish a study and the primary outcome is very different than what they said it was going to be it may be that one of those secondary outcomes was the one that they found that was statistically significant. And that's a problem. But I do feel like in a lot of those well-constructed studies they are presenting the primary outcome but then they'll also have like four or five other articles on secondary outcomes or other stuff they may or may not have found. Is that. I think that that's a really useful way of generating knowledge from the same trial. OK. So I think I don't discourage the use of doing that. The problem is when is the bait and switch. Yes. The problem is saying I'm going to look at outcome A. Yeah. And then three years later you say oh we found a statistically significant improvement in outcome B. Yeah. Yeah. You know I did and fellowship I really never worked in the lab. So in fellowship I worked in the lab for a couple of years. I was going to completely cure neck forever which didn't go that great. There's still time. And so you know I worked with rats and lactobacillus and all of these other things. And what I'm so glad that I did those two years because I really began to understand a little bit more how to read like scientific papers and pediatric research. And one of the things that I learned is that if you put enough of anything on a sample you can destroy it. You can get free radicals you get apoptosis. So for me the biggest thing when I read like a scientific article which isn't that which isn't that frequently but if something comes up you know especially if somebody writes about it on Twitter or X or whatever and I'm looking through it my first thing is is is the concentration of whatever they're doing in the realm of anything that we would give to a baby. And a lot of times it isn't like it just doesn't mean anything. You flood intestinal cells with like you know more probiotics than there are cells like that's not something that's feasibly going to happen. You can get very different in vitro results. So I always I always think about like how it's very easy to present data to make it sound exactly like what you want it to be. The challenge I think you know that you know there's the old saying you know lying with statistics right. You can lie with data too and I think that that not understanding the purpose of some of these things is is like that a great example of what you're talking about is with is is with aspartame. The nutritional studies that show aspartame causes cancer. Yes if you give rats a high enough dose of aspartame there is an increased risk of but to get the equivalent of that in a human would require drinking something like you know 40 or 50 diet sodas every day for like 20 years to get the same equivalent amount of aspartame. Yeah it's something ridiculous like that. So it was designed to be it was really the studies were designed to how bad could this be if we get massive doses. But when you talk about kind of regular dosing it doesn't come close to that. A lot of people will use those same studies when they're talking about vaccine risks like oh there's aluminum in vaccines and you know there's less aluminum in vaccines than if you eat an apple you know things like that. There's a lot of ways to twist this stuff and if you don't really understand the the precise reason why certain things are done in animal studies and you know this because you've done animal studies you're almost always trying to overload the system so you can find a signal. Right yes you're right you're absolutely right. Okay let's talk about the power of studies. What does that mean? You mentioned it earlier. Yeah so so statistical power is the ability to tell if there is a difference between two interventions. Is the study able to tell you that? And a great analogy of that is is there an actual signal there? Power analysis really only applies to when there's no difference between the two study conditions. So if you have a placebo control intervention so you have the placebo on one hand and you have the intervention on the other hand and you give a group of babies one and you give another group of babies the other and you don't see a signal that's when you ask the question was this study powered to detect a difference. Here's a good example of that that's not statistically related. How difficult is it to hear a whisper in a really crowded room versus in a quiet room? And that's a good example of statistical power. A well-powered study would allow you to hear that whisper because the background noise is so low that you can hear it. Whereas a poorly powered study there's so much variation in the outcome or the treatment that you can't tell that a whisper is happening. So you end up with a study that shows no difference even in reality even though when in reality there was a difference it was just too small to detect with the study that you designed. Okay. Is that helpful? Yes that's very helpful. So you are a lot more concerned about power when you're not seeing a statistical significance. Absolutely you're only concerned about power when you're not seeing it. Okay. When you're designing a study or if you're or if there's no difference in the two intervention groups. But sometimes I feel like you can see a statistical significance and it's a tiny study and I feel like that's a little bit of a red flag because there were only 30 patients in the study. Can you really determine that yes giving exclusive donor brass milk really is better than giving formula or whatever? So that gets into kind of a different bucket of what we call a type one error. Which is if something is actually statistically significant that does not necessarily mean that it's clinically meaningful and it doesn't even necessarily mean that it's true. Statistical significance is designed as a test of how likely it is that if you were to do this over and over again that you'd come out with the same response. Okay. So let's talk about that. Let's talk about the p-value which everybody loves. You're right. And everybody's obsessed about like I feel like everything's gone to the p-value. And that's really deceptive because if you have lots of events and the p-value does not reach statistical significance you generally feel that there's no difference between the groups. Okay. People are obsessed with p-values and I think some somewhat mistakenly so. P-value is a way that we determine statistical significance. Okay. A p-value is essentially asking it's set at a value and we usually by convention use the value of five percent or 0.5. And we want a p-value which is a probability value that's less than that. 0.05. Yes. Did I say 0.05? Yeah. Did I say it again? Yeah. So 0.05 is five percent. 0.05 is five percent. So we want the probability to be less than five percent. What do we want the probability to be? We want that if you were to repeat this study what is the likelihood that you would get the same exact difference between the intervention and the control just by chance. Okay. Give me an example of that. So an example of that would be steroids for reducing mortality and premature birth. Okay. Okay. So prenatal steroids. Prenatal steroids or antenatal steroids. So when you give it to the mom before the baby is born. The first time you do the study you saw that there was a p-value that was less than 0.5. So it was unlikely that if you repeated that study that the result would just due to chance. It was just a chance. So when we see when you were the first people to do that study you're like oh when we gave steroids there was decreased mortality in these babies and the p-value was less than 0.05. So we feel very good that this wasn't luck. We feel very good that this really is good for the babies. That's right. And if we were to do the study again. Yeah. There would be a 19 out of 20 percent chance that we would find the same. Okay. That it wasn't luck. That it wasn't luck. That it was real. Okay. So the p-value is really what it represents is the chance of that study being real. Basically. Yes. It's the results being real. So we we call this the p-value of five percent. There's a five percent risk of making a type one error. Okay. Which is the error that you make when you think that there's a difference but there's not a difference. Okay. Love love those type one type twos. Yeah. Lots of boards. Okay. So we're both type type one people. So type one is when you think say that again. So a type one error is that you come up with a result that shows that there is a difference but that is just due to chance. So for example going back to that steroid one. Right. So it could have been that one time out of the 20 that you got really lucky. That's right. And that would have been a type one error. That would have been type one error. So we've accepted a five percent chance of a type one error in medical literature. Generally speaking. Now there are there are you know if you're doing a bunch of comparisons. You could just say yes. Yes. No tell me your caveat. If you're doing a bunch of comparisons sometimes they'll make the p-value smaller. Right. Because if you're saying that there is a 1 in 20 chance that this finding is due to chance but you compare 30 things then just on the basis of doing any comparisons you're going to have one that yeah that's got a statistically significant thing. So if you're comparing 30 things some people will make the p-value even smaller. Okay. And that's called a correction. Okay. Versus just one or two. Just five percent. Okay. So now we've already kind of mentioned power. If you are setting up a study how do you determine how to power it? So the power is the ability of the study to not make a type two error. So remember type one error is when your study finds a difference but there is not an actual difference in reality. A type two error is when the study finds no difference but in reality there is a difference that you've missed. Okay. So power analysis is all about making sure that you've designed the study so you're not missing an difference between the intervention and the control. So what data do you include when you decide like I want to find out whether this medicine that I came up with I am going to give oral arginine, I'm completely making this up, to baby's guts and see if they can get to full feeds faster. How do I decide whether I how to power that? So first you have to make an estimate. Okay. Okay. I'm going to estimate that if I give this oral arginine then they're going to get to full feeds five days faster. Okay. So some first you have to have a treatment effect estimate. Okay. Okay. Then based on the treatment effect estimate there are calculators and you can google these calculators. This is the treatment effect estimate. How many patients do I need to adequately power to investigate this condition? Okay. So that's really the only number that you need just the treatment effect. The estimate. Yeah. The estimate. So you you're just guessing it. That's right. Well I mean you want to make an educated guess. Yeah. Yes. Yeah. You're not making a but but again that also goes back to the clinical significance right. Like if it's going to decrease the time to full feeds and I'm putting something entirely into the babies I don't really care if we're going to get there six hours earlier. Right. That's right. So you're also setting that up making sure that you actually care about the clinical outcome. Well and I think that that really dovetails nicely into this concept of the difference between statistical significance and clinical significance. Yeah. Right. Because just because you statistically shown that there is a difference between two groups that does not mean that that is a clinically significant or clinically important thing to do. Right. Right. So with vitamin A we're talking about these these shots that the babies had to go you know anytime you're talking about shots you're talking about some pain but then there's also risks of medications and things like that. The greatest example in neonatal literature I feel like is using steroids after the baby is born for babies with BPD. Yeah. We used to this all the time. And what we found was that really high doses of steroids really early in babies that are at risk of developing BPD was really good at getting them off the ventilator but it left them at risk of having developmental problems later on. And so then we're like we shouldn't do this ever. OK. So we stopped doing steroids. Yeah. We took away one of the most important medicines in our arsenal to combat BPD. And no surprise we saw a rise in BPD. Well right. And since that time so that was kind of like early 2000s late 90s steroids everybody 2000s everybody is like steroids are horrible. Right. And then now during that same period since the late 2000s we're having 23 weekers 22 weekers much higher rates of BPD not much higher rates similar rates of BPD but also much better survival. So altogether a lot more BPD. Right. So then we're all realizing OK like you know at some point we have to get these babies off the vent like that's not good for neurodevelopmental outcomes either. It is. Not only is it not I mean it's it's absolutely proven that it is bad to be on the vent right for neurodevelopmental. So so at some point you have to ask yourself is the cure better or worse than the alternative for BPD really early high dose steroids is probably not worth it. But if you're a few weeks in and you're still on very high settings and just the fact that you're still that that ill that puts you at increased risk of having neurodevelopmental problems then suddenly the same medication is now indicated to reduce your risk of bad outcomes. Right. OK let's go quickly through the different types of studies. OK. So tell me the different types of research studies that clinicians should know just a few words. So the gold standard is a multi center randomized placebo controlled clinical trial. Why multi center. Because it increases the generalizability. OK. And why placebo controlled. So you want the intervention to be tested against something that where no one knew that there was a alternative. So if you just give a medication you don't know what happens if you don't give the medication that's only really half the story. Right. OK. Or also you are worried about like a little bit of bias coming in that if everybody knows you know that you're giving a medication if you're not giving a placebo and you're just giving a medication by definition you know that. Yes that's why we want it blinded. So double blinded which some people talk about is that the investigators don't know which in which intervention you got but also the patient doesn't know. Now in most of our cases the patients are not necessarily aware of the intervention but the families are the other health care providers are and stuff like that. So so double blinded means nobody knows who's getting nobody knows. Yes. OK. So and then randomized control. What does that mean. So randomized just means that you are not allowing the different populations to introduce bias into the results. Ideally if you randomize 100 babies to groups of 50 and 50 if they're randomized properly they will look very similar. OK. How do you actually randomize babies at bedside. That's great. There's a bunch of different ways that people do this. A lot of times people randomize before the fact. So whenever a baby is eligible they'll call a central center and they'll say hey what you know what is this baby. You know this is baby number 14 to the study. Is it intervention or control baby. Other times there are then there gets to be the complexities. What if you have twins. A lot of studies now will randomize twins to the same arm. So you don't know if they got whether they got the intervention or they got the control but they both got the same thing. There are also things like cluster randomizations where you randomize in kind of groups of times. There's also other things we randomize by site. So you evenly distribute the number of babies that come from site A versus site B. So I remember being part of like large national trials where they are actually blinded and you're the randomization which is so exciting. So exciting when you actually get somebody involved in the study and you're like yeah I'm doing my part. And then you open the envelope and because it's blinded it's not going to tell you oh intubate this one or put this one on CPAP. It's like the you know obviously you can't randomize intubation or CPAP but like when we were doing the you know the pulse ox studies the low or the high they were you know they were all like clever pulse oxes which you literally didn't know. And so you'd you just get a color. You know this is going to be orange or this is going to be blue or whatever and you don't actually know what you're doing you just know that they're falling into the orange group or the blue group. I mean a lot of a lot of organization goes into that. I mean that's crazy. And the real reason you randomize is you want to reduce any external sources of bias. The highest quality data has the least amount of bias. Okay so obviously this is a prospective trial because it's randomized and you're moving forward. So when you decide that you're going to do the trial you randomize. Hopefully it's double blinded. You don't know what you're doing. The families the patients nobody knows what they're doing. And then after the study is finished how long do you run a study for? How do you determine that? So it really until you have enough babies. Okay so based on the power analysis. Okay so you go through and that also that also I feel like is a big problem in neonatology. There'll be studies run over like eight years and during those eight years we're like you know suddenly not intubating anybody. We're doing Lisa for everybody. You know a lot of variations can happen when studies run over a long period of time. It's a huge problem. It's also a problem experimentally if a study does not finish enrollment. Yeah. So let's say that. That happens so much. It happens a lot in neonatology. Yeah. Let's say a patient you know a study is supposed to have 400 babies. Yeah. It's to be powered to show a difference if there's a difference and you only enroll 200 babies. Yeah. What do you do with that? Well that did happen in the DART study where they did the dexamethasone a randomized trial where they were trying to show that the babies that were given steroids were extubated sooner. And then there was a later study that followed those kids up in that study. They didn't complete enrollment. That's right. But they did find a difference. So even though you were saying that the power is to not find a difference. So can you speak to that statistically. This is going to get into the weeds. I knew it was. I know. That's why I like sometimes I'm just like well. But the idea is that once you design a study if you want to be a purist you should enroll every single baby before you look at the outcomes. There's always a chance that in the first year of the study or the first half of the study you enrolled all the babies that were going to show benefit. And in the second half of the study there may be a harm or there'll be enough babies that don't show benefit that you will find that there was no benefit. Yeah. And so the true answer is kind of if you want to be a rigid idealist you shouldn't report results until you're done. OK. Now there are mechanisms in place to try to prevent that. So a lot of studies now have data safety monitoring boards. So it's an independent organization. It's not the trial investigators. It's not the people doing the research. It's someone totally independent that has an interim analysis where they're making sure that there's not a big harm. Right. Or let's say you've enrolled 300 babies out of a 400 person trial. You get to 300 babies and there's no benefit at that point. Zero difference. There's zero difference. There are some cases where you don't need to enroll the other 100 to prove that there's not going to be any difference. I feel like the safety ones that we have a few examples of those in neonatology where they're like this is not only not making any difference but whatever they're giving also does provide some harm. So right. Right. So we shouldn't. Yeah. And they're completely external because obviously they're not blinded to it. Right. And that is where would that committee be. It's usually independent. So and it depends on who's running the study. I know that for pharmaceutical companies a lot of times I'll find someone that's not related to the study that's not on the study protocol. That's just constantly looking through the data that someone that knows the data. And you know for the new research network in the United States they haven't. I think they have an external data safety monitoring board that I think is actually international that looks at the data and says you know kind of continue. And they do it at general speaking pre-planned either time based intervals or number of patients that have been enrolled. Yeah. OK. So the highest like evidence that we can get from a study is a multi center randomized control trial. Yes. All right. Next level. So the next level is when more than one. I mean ideally if you have more than one of those studies that gives you your highest quality evidence because your risk of bias is the lowest in a single thing. What people have now started doing is they take a study that's evaluating a treatment and a different study that's evaluating the same treatment and they combine them into something called a meta analysis. OK. OK. And in this there are a couple of different ways you can do it. You can do individual patient meta analysis where you actually look you pull all the patients together. That's a bit more complex. It requires everyone's cooperation. And there's also just kind of the traditional meta analysis where you just take the summary information that's already published from one study and the summary information from a second study and pull it together. It seems like that would provide you with higher quality evidence. It would. It would seem that way. The problem is is that for most studies there are enough differences in how the studies were undertaken that combining two groups of patients that may not be similar. We really limit ourselves with the generalizability of those results. OK. So you would prefer one excellently done massive randomized control study versus a meta analysis of. Prefer is a prefer is a really good word. I feel like if I'm trying to make a decision on a baby it is in some ways more informative to me to have a single really well performed randomized clinical trial to look at that data. It's easier to tell how generalizable that is. It's easier to tell. Does that apply to my baby. Yeah. Then this kind of meta analysis. Yeah. OK. Right. So randomized control study. Next one is analysis. OK. Then there is all these kind of newer types of of of randomized structures that are really really interesting. So some of them are cluster randomization. Yeah. Where you randomize rather than by baby randomized by time period or randomized by by center. Those are very helpful because the randomization again tries to reduce the bias that you get from having specific patient populations. OK. Those are kind of the top. And there is like a pretty big difference between what you get to next. OK. Which are kind of retrospective multi-center case controlled studies. OK. So give me examples of those. So those are things like we we looked at. I'll tell you an example of a study that I've done. So there's a there's and you only do these studies if you feel like no one's going to be able to do the randomized right control trial which is a lot. There's a lot. There's a lot not just because of actually getting the babies involved or it's a rare disease or something. But I think a lot of the time it's our fault as well. Like doctors lack a lot of echo poise. Yeah. Even in stuff that we kind of feel like you know we don't know the data. Like I know that there is not a lot of data about giving I know to preemie babies and acute hypoxic failure. But like I'll have a 23 week or an 100 percent. And the mommy is sitting next to me and being like can you please try everything. I'm probably going to put that baby on nitric. Right. That's the art. It is. It is a problem. Yeah. Like that. Even though I know that study needs to be done. Right. You know. So I think that we practice in a space where 70 percent of the medicines we use are off label. Right. Where the patients are sick. They're sick and they're relying on us to do something. And there is no day that I go go through clinically where I can use a randomized trial to answer every question I have. Right. It's not even a couple. Yeah. I'm lucky if one or two decisions are supported by well you know well well well done control trials. Yeah. So then you get into kind of observational data. So what happens if we compare babies who got this treatment versus babies that got a different treatment in the past in the past. Okay. So for example you're like from now on I am going to put the head of the bed at 15 degrees for all babies. Do I have less severe I.V.H. versus a cohort in the past. That's right. So that's an obstacle. Why is that not good. It's not good. As good as RCTs. Partially for the reason you've already said. Are there differences in the management in the past that you're not accounting for when you're making the comparison. Here's a here's a really great. So that would be a great study to do except that now we're taking care of a lot more 22 and 23 week. Right. And their risk of I.V.H. is higher. And if you compare them to what you're doing 10 years ago those kids you weren't taking care of 10 years ago. Right. Right. So that means that it's harder to know if the difference. In fact if you were to do that study you might find that the I.V.H. rates are worse. But it's because you're taking care of a lot more 22 weekers than you are 23 weekers or 24 weekers than you were 10 years ago. And that's what type of study is that considered a retrospective. So that typically people call that a retrospective. I mean that you know prospective is if you collect the data going forward or if the data are already collected. What if you're comparing both. Like I'm from now on I'm going to put the head of the bed at 15 degrees but I'm going to compare it to babies that have already. It's a prospective with a retrospective comparison. Excellent. I'm so happy that I love that phrase. Okay. So that is I feel like we do that quite a lot. And we often do that incorporated in kind of quality improvement projects. Yes. We're trying to do whatever we can. That's right. Which obviously even if it's not getting published is still hopefully better for your unit that we're trying out one thing at a time to improve the outcomes. Also I always say this anytime you study something the rates are going to get better. So we should just study it to improve the rate. It's called the Hawthorne effect. The Hawthorne effect. Yes. Yeah. And that is like that's a real thing. Yeah. Well documented in the NICU as well. Like if you're going to study neck you are less likely to have neck. And with neck I always think that that's because you know we always say that like standardized feeds decreases the risk of neck and like everybody doing the same thing. And if you are in a study everybody's a lot more likely to do the same thing than if they're not in a study. Absolutely. Like it's not even a stretch of the imagination why like that would be the case. So there are a lot of really good there's a lot of really good studies that show that actually just being in a study is a different experience for the baby than not being in a study. Yeah. We should get all like in the he-monk world like all their babies are in studies. Yes. All our babies are in studies. Yeah. Yeah. Okay. So that so you know perspective whatever that phrase that you said what's the next level down. So I find a lot of value in these like observational research. So it's research where you're just looking at what is happening. That's a lot of That's a lot of the stuff. Yeah. Yeah. So talk to me about that. So primarily what we're trying to do is fill a void where there's not a lot of data about either how things are currently being practiced or in very specific cases if there are medications that we want to test out is a medication better than B medication and things like that. So in our data set we'll look at you know kind of multi-center look at a bunch of different centers over a number of years to see if there was a treatment effect from one versus the other. But those studies are really hard because the biggest source of bias is pretty significant. It's not just statistically significant it's clinically significant and that's selection bias. Yeah. So if you have two babies and one of them is sicker than the other baby you might be more aggressive with the baby that's sicker and unless you have some way to control for that that comparison is going to be biased because the two babies didn't start out exactly the same. Yeah. And going back to the INO just to give an example of this I think a lot of that data is if you're looking back and comparing babies that were put on INO versus not put on INO and then you see that all the babies all the 23 weekers that were put on INO died and all the babies that weren't put on INO didn't die. It's very difficult to tease out the fact that you know the babies that were put on INO were probably sicker for somebody to go ahead and do that whether they had similar echo findings or not something made those that group of clinicians say this is what we're going to try out of desperation. That's the fundamental concern with selection bias is that if you're only comparing babies that got a treatment versus those that didn't there's some reason those babies got the treatment and it's probably because they were sicker. Yeah. And if you're looking to study if that treatment was helpful a lot of times you'll find that the treatment was worse for the outcome okay but still that data is so helpful with you know observational data is so helpful anyway with kind of diagnoses and outcomes and prognoses and like you know it doesn't have to be like this one is better than this that still fills like a huge hole in like what we can tell families what we can expect. Yeah so descriptive things I think are easier right like if you're if you're examining a disease and trying to find out the incidence of certain aspects of that disease it doesn't matter if if your sample has some selection bias right I mean there is probably some effect of it but that that is still useful generalizable knowledge. Right so then that's kind of observational what's below that? I tend to put single center stuff really kind of the bottom yeah. Really? Yeah. That's so depressing. I don't think it's depressing I just think that and people could probably argue whether yeah you know retrospective stuff is better than single center even if those single center studies are being incorporated into the meta-analyses that like a no I'm not saying single center randomized perspective you're saying observation I'm saying observation okay sorry yeah yes so like hey we had a problem with neck so we did this thing and now our neck rate is better yes that to me is is helpful it's important that people publish that because that's how we learn yeah but I don't know that that single center having that experience is going to convince me to change how I'm doing something right and then below that case reports right so case reports really at this point in medicine are used for really weird things yes right like unless something really bizarre happens you probably shouldn't be writing about it but they fill a really important niche right like yeah if you're taking care of something that is only seen one in a million times it would be helpful to read someone else's experience of doing right I mean that's kind of where medicine started right it was really just case reports yeah well we just did one on with surgeons on he had kind of a new way of doing of closing an omphalocele a a giant cystic omphalocele he taught kind of through the techniques that he used and it's getting published as a case report obviously that's very helpful for all the surgeons for all the neo is taking care of a baby that this is a different way of getting this baby fixed and feeding you know and that study will be more valuable than a large randomized double-blinded placebo control trial of yes how to fix a giant well how to fix a giant omphalocele maybe not but just how to because that was so different because that baby wouldn't have qualified for that randomized clinical trial so again remember that I think the most important thing is how useful can this information be for the patient that I'm taking care of and and even though there's like a pyramid of evidence um sometimes the case reports are the most applicable evidence that you can find for your baby and I think that that's the bottom we don't have anything below that right I don't think so it's just somebody writing something from experience right just writing to an article and being like in my experience this is what we should do a lot of medicine okay just a couple more things I want to cover what is intention to treat analysis um you talked a little bit before about equipoise um should we talk about that yes so equipoise is this concept that whoever is performing the trial should really not uh have a strong preconceived notion on whether the intervention is better than the than the control the best example of that that I know of is when Barbara Schmidt did the caffeine study she actually did it because she thought caffeine was going to be worse I love that I love that and she did the trial and she found out that caffeine was was better yeah for babies so in that and bpd that was what was so cool about that study yeah and and so that's a classic example of I'm doing the study because I don't really know what the answer is or I feel like there is sufficient uncertainty on whether this is beneficial or harmful yeah uh the oxygen study is the same way like what what oxygen saturation is you should keep babies that that are premature so equipoise is that concept do can you maintain kind of a neutrality that allow you ethically to have a baby either get the intervention or to get the control condition intention to treat is kind of a subset of that is that some babies you intend to give them a treatment but they don't get the treatment so if you're doing a study where you're giving a baby nitric oxide for three weeks and the baby dies in a week the baby didn't get the full treatment okay right but what if they just stopped it at two weeks because somebody decided it wasn't doing anything then that then then you analyze it both with the intention to treat which is the baby originally was randomized to that condition let's include him in that okay analysis a lot of times I'll also do kind of like a what actually happened okay I love the idea of I feel like an intention to treat analysis is actually more generalizable because that is going to happen in life that you decide that you are going to treat that and then the nurses one day like there is no way that we're giving this baby blood I don't care if it is a high the heart rate is low the baby's gaining weight or whatever we don't like this baby doesn't need blood like there are things that are going to change your mind in the unit even if that is the guideline that you're using you can't be so hued to what you're doing that you do it in opposition to what's best for the baby and intention to treat is entirely designed to improve the generalizability of our study findings okay we're starting out with the best intentions but if we decide before treatment is complete per the you know per the trial that it's not the right thing then we can stop it and we still feel like those trial results are generalizable to our patient okay what exactly is bias in a study and there's lots of different types of biases there's like literally there's a whole bias wikipedia that sometimes I get lost in because you know because there's just so many different ways of classifying bias so in the lay population you might think of biases oh I have this you know preconceived notion that something is going to happen that bias exists as well but traditionally in in the medical literature and research we're not talking about bias from preconceived notions biases when there are external factors that are influencing the outcomes that we're measuring okay and those influence the outcomes that we're measuring in ways that we can't control and it makes us less able to believe the results that we get give me examples of bias so selection bias we've kind of already talked about if you're if you're comparing babies now to babies 20 years ago you're you have a selection of babies that tend to be a little bit younger than they were 20 years ago yeah and the way that we practice medicine is different now than it was 20 years ago so those babies are are quite different groups so you've selected out a cohort of babies that are that are different we talked about this a little bit before that if you're looking at babies in a randomized clinical trial setting a lot of those babies you've selected for babies whose parents are willing to sign consent forms yeah yeah so already maybe a different group than the babies that weren't willing to sign consent forms that's right we're born in the centers that we're doing the research right or born too precipitously to even get the consent consent that's right yeah yeah any other types of bias like when you're looking statistically and stuff another bias that we've talked about kind of briefly is is is a bias in the medical literature which is publication bias yes so this is the idea that in order to get something published you have to convince an editor of a journal that it's worth publishing and it's much easier to do that if you find a difference than if you don't find a difference or if you have a quote-unquote negative study yeah and so in general um if we take a p-value of five percent we'd expect the medical literature to include 19 out of 20 studies should be negative studies but that's not actually what happens right most of the studies that are published that's interesting thinking of it that way that's so true are positive studies yeah um and so there's a huge um publication bias away from negative studies yeah so that's crazy i wonder if there really are you know we really are getting well i mean technically you'd say well we're only studying things that we think are likely to be positive yeah but like in you know and and people have taken steps that now most of the top journals they're doing a very good job of publishing um negative studies particularly these large randomized trials that take you know yes millions of dollars to do break my heart by the way when like there's a randomized trial that's being done in multi-centers over like 10 years and they don't find any difference like i almost want to cry when i know but it is important for us i mean it's so important but one of my one of my friends and colleagues just published one of those last year um and uh i remember talking to him and just being i'm so happy that you published this it was so important for you to publish this and he's like yeah but it would have been nice if we had been able to find something that was helpful and i was like yeah this is helpful it is helpful um okay another quick question the difference between primary and secondary outcomes yeah so generally your primary outcome is the one that you've pre-specified and the one that you're powered to detect okay so give me examples of primary outcomes and like you said something that we care about mortality yeah okay so mortality is a primary outcome sometimes the secondary outcome is mortality free of bpd okay and let's talk about that for a second as well this concept of like the importance of not singling out the primary outcome because in neonatology if you are trying to study death and bpd if you are 22 weeks and you die in the first week of life you're not going to get bpd right so you also have to combine those two right so those are competing outcomes in that case you know a great example that you used before is that we're seeing a lot more bpd now yeah um and some of us were taking care of kids that we weren't taking care of 20 years ago yeah and if the only metric is that your bpd rate is going up you may miss the great point which is a lot of those kids are surviving to get bpd that wouldn't have passed away that would have passed away before yeah so let's say that you are um studying bpd outcomes how do you get rid of the babies like the the um bias from whether the babies died or not because if you died you're not going to end up with bpd so usually what you do is your primary outcome will include all of your competing outcomes so in this case death or bpd and then your secondary outcome would be bpd by itself and death by itself and ideally remember you're not looking at just the single outcome you're trying to take the whole totality of what knowledge you you generate with a study so if you see that there was no difference in the overall outcome right death or bpd but the death went down and bpd went up in the secondary outcomes that's still a really important signal yeah even if it wasn't in the primary yeah and then i read recently last thing i'll say i read recently that like it's not enough just combining those two because any parent would take some bpd over death so also statistically how to kind of like grade those so that like the outcome of bpd you'd still prefer it rather than 50 50 bpd or death so keith barrington out of i'm a huge fan as you are of keith barrington's blog yes there is no there's no equivalence between death and bpd they're not equivalent outcomes they compete in a statistical sense but from a clinical sense are totally different and this is again it gets back to clinical significance is different than statistical significance and we i feel like as a field have gotten much better the last five to ten years of trying to incorporate families preferences um one of the ways you can do that is by looking at outcomes as a rank so with death being the worst and maybe tracheostomy is is next to it and then the next one is going home on auction and the next one is no bpd so that suddenly we've opened up our eyes to the true spectrum of what outcomes we can see i love that thank you so much for everything i mean he just literally knows everything about research and it's fascinating chatting with him i really appreciate it if you're interested in more neonatal education then please subscribe to the channel thank you all so much for being here