Digital Literacy, AI and EdTech quality | Natalia Kucirkova | EdTech Taiwan Expo

Thank you very much for hosting me. I'm delighted to join you all the way in Taiwan. I hope you have had good day so far. Um my name is Nataban Kucherov. I work in Norway. We have right now uh the rainy season and the dark season. So I hope that you have some sunshine and uh I can um energize you with my uh presentation. So my role is a professor. I work as professor at the University of Stavanger here in Norway and I also am a director of the international center for edtech impact and I care deeply about the importance of evidence and impact in educational technologies and all that is related to today's topic digital literacy and AI. So let me share with you what I thought would be interesting to talk about and I know that we have also some time for questions and answers. So hopefully the content that I share will be interesting for you and we can discuss it later on. So let's start by first discussing where biblical literacy as a term comes from. You may know that digital literacy is something that has evolved with time. It is um basically a terminology that has been redefined and re-examined as new technologies appeared on the market. First it was about digital literacy and the very basic technologies that were available in 1960s. Then as more and more computers were coming on the market, there have been more and more people saying that we need to change what we mean by digital literacy. It is not only managing your radio or TV, it is also managing your computer. And then it was about well not only managing computers and PCs but about managing the devices that you have on your desks today like smartphones, tablets, iPads, e-readers, smart watches and of course as we are moving more and more with this evolution of different hardware we are seeing that most of these devices are these days equipped with AI. I quite like what has been defined as digital literacy in 2020. This was based on a paper that essentially reviewed all the different understandings that are about digital literacy because what I mean by digital literacy might be very different what you mean by this term. But if we can agree some kind of common definition, then maybe this one by Ready and colleagues gives us a nice way forward. It is about an individual's ability to value information, use this information effectively, but also to communicate and share this information with others. So if you're using your device in a way that benefits you, that's very good, but it also needs to be benefiting others. so that we can see that you are skilled and digitally literate. And I quite find it interesting to be thinking about digital literacy in the age of AI because if you look at the many hallucinations and mistakes that AI makes these days, it gives us a nice understanding of why we need to be discussing it. What does it mean to be using today's technologies in an appropriate way? You see here the prompt was create a pregnant woman and she suddenly got two pairs of hands. Right? So it is quite interesting how we can be negotiating the meanings between our own capability to be using um digital devices and what these devices are producing and how we are sharing that information. So, as we're entering the age of generative AI, that is really generating content based on our prompts, we need to be rethinking and really redefining what we mean by digital literacy. So, if you have your own definition, I'll be very keen to hear it um when we come to uh to the discussion part. Now most um most people who interact with technologies for the purpose of learning will be interacting with edtech. EdTech is essentially the abbreviation for educational technology. So technology that was designed for the purpose or the intent to educate and there are five different types of edtech products that are these days available and that can be uh using AI. Here is a very nice uh sort of overview of what they are uh by the Asian Development Bank where you can see that they uh can be summarized in three categories teaching and learning, education, governance and employability entrepreneurship and I'll be mostly focusing on the first bucket here which is teaching and learning and that is about learning content, learning activities, learning assessment and learning devices. And I presume those are the devices and those are the edtech that you're also using in your own context. So we'll be looking mostly on the teaching and learning type of edtech. Now the problem we see with this type of edge tech is that there is a lot of funding for the development of it but unfortunately the evidence so the proof that it is really improving education is not here. So we have some very important reports from the global agencies like UNESCO, UNICEF, World Bank that are showing that our students, our children globally who are using edtech whether it is in schools or at home are not getting the educational benefit that edtech has promised. And this is a huge problem. This is something that we can say has been driven by the technology development but the educational side has been lacking behind and we really need to develop our digital literacy to ensure that that doesn't happen again that we can have better edtech that is genuinely improving children's learning and teachers teaching. So the need for evidence-based edtech or essentially edtech that is based on solid science and can positively impact learning really depends on having a clear definition and framework for signaling evidence and quality. Understanding what really makes edtech good for learning and teaching. And this is what we call evidence-based edtech. These days there are many different frameworks that you can use to understand what is good quality in hashtag. In fact there was a report by the UK government that found 74 different uh frameworks. Um another research study that specifically look at edtech evaluation frameworks found that there are 65 different ones. So we don't need more definitions and frameworks. We really need to consolidate and harmonize what is out there to understand what is the evidence behind these different tech tools. And I was very lucky to work with a big consortium of researchers, an international consortium where we um collaborated with colleagues across the globe to define this consolidated framework. And that consolidation allows us to now be evaluating ETH products internationally with a very solid understanding of their quality. And that is the kind of digital literacy that we want to bring across to teachers, to parents, to policy makers as well as developers of edtech products. The framework is called the five E. Uh, and each E, the letter E, stands for a different type of quality in adtech. And understanding this is arguably having a solid digital literacy in understanding what works and what doesn't work in edtech development. All these reports have been produced with the researchers uh in different stages and they are published as small books um and freely available. So you can download them from the University of Stavanger website. And here is the five E. So it is something that uh we have been advocating for teachers, policy makers, developers to really ensure that when they are saying that they are improving children's learning and teaching that they actually can have a proof or evidence that they are doing that in one of these different impact areas or indeed a combination of all of them. So efficacy is the first E and it is about knowing whether the ETH solution works in controlled experiments and controlled environments. So when researchers conduct an experiment where they compare one edge techch or one app versus another app that will be the impact on efficacy in terms of effectiveness. Those will be the studies that are conducted with teachers. So when teachers and practitioners in their classrooms are looking at which tool they should select and whether that solution can be used in a way that it actually improves their students learning. Then it is about effectiveness and really understanding that is part of the digital literacy of the teachers because the teachers need to know how to evaluate all these different apps and learning platforms that are available to them. And this framework allows them to use a specific rubric where they can score it according to the criteria that have been proposed by this international consortium. Then of course it's not only about whether and whether it could or does the solution work, but also who it works for. So does it really work for children with special needs, for students who have been left behind? Does it work for the full spectrum of needs and abilities? So that is the equity in the middle. Ethics. The blue column here is very interesting because it is about ethical use of data. And as you would know with AI, there are many issues with privacy, security and really the ethical question in the ways AI has been developed and AI is being implemented these days. And when I say AI, I'm using it as a shortcut for the large language models and the generative AI that is being used these days. And finally, environment. The green column is very often forgotten when we are talking about the quality of digital tools. But really you mustn't forget that as these tools are becoming more and more prevalent in the society as they are being updated all the time with new apps and new updates that you have been downloading. All that costs energy and energy influences the health of our planet and it also is created based on some materials that are not always good for the environment. So understanding technologies and their quality being digitally literate means also asking questions about the environmental impact of these technologies and I know that is maybe quite a big and complex message but these five E are a way for showing and understanding how digitally literate people are when they are talking about the quality of edtech. Here is another way of showing the five E where you can be asking questions on this wheel of okay does the technology address cost and contextual issues? How is it about the economic outcomes? Is it really promoting equal rights? And what about ethics? What happens to users data? So it's the same thing but only presented in a different way. And if you are interested in knowing more about this in more detail, you can also check the UNESCO framework. There is a nice synergy between the UNESCO framework for system transformation and these five E. And it's really a I would say an enhanced way a deeper way for being digitally literate and understanding the quality of what works and what doesn't when we talk about technologies. So when someone says this technology or this app this learning platform is a low quality or is not good it doesn't work then I encourage you to ask them well in what way does it not work? Is it about the ethics? Is it about the equity or is it about learning outcomes? Is it about the efficacy of the technology? So that's a vocabulary or a language that you can adopt to be evaluating technologies that you are using or technologies that you are encouraging others to use. And here of course I'm thinking about teachers and parents who are mediating the use of tech uh for children and learners. The 5 framework has been adopted by the largest certification providers. So if you want a shortcut, if you want a quick way to know whether an edtech solution works, you can check whether technology has been certified. And these certifications are provided by an organization called EDU evidence that is essentially a consolidation or an umbrella of all the different certifications that exist globally to allow for this understanding of quality on this shared international standard. Um I know that learning spark was the uh recent addition to the uh consortium of certification providers and certainly the Asian edtech marketplace also has a lot of certifications that are part of the edu evidence certification scheme. So if you are for example looking for whether the app that you use has been certified for ethics or equity you can check on the edio evidence website whether it is included in the catalog and that is also part of how the global agencies work. So for example, UNICEF that has been supported by the Asian Development Bank and ARM they have a catalog of recommended resources and as you see here in the slide it does say what is the evidence of impact and that is the way how quality is being defined when policy makers and governments want to make solid decisions about whether something works and whether something doesn't work. Here is another way of looking at that. So you can see on the left there are different companies for apps and learning platforms or digital books and ebooks. And then you can see their products and you can see the different buckets the efficacy, effectiveness, ethics, equity and environment where essentially users can check whether technology has been validated or whether it has been certified in the bronze, silver and gold. So the level of its quality so that we can then say that this technology actually works or doesn't work and is of low or high quality. And this is really essential if we are trying to address the problem of a lot of technology on the market. In fact the market is really saturated with many many apps and learning platforms doing essentially the same thing. If we want to understand what is of good quality and what could be of better quality, then we need to have some kind of system for it. >> And this international endeavor that has been ongoing is perhaps the first step in in that direction. So you can see how essentially the different levels work u on this slide uh in terms of understanding what kinds of studies have been done with a specific learning platform to understand its value for students. So these are serious and solid studies that need to be conducted by researchers and scientists to understand whether the app is actually beneficial for learning because if it's not then we don't really need it right I mean what is the purpose of edtech that claims to be educational if in the end when we compare it for how much students have learned and we see no learning gain then it's not really an ad It's perhaps technology but it's not doing anything for education. So we need to have a good way of verifying, validating and improving the educational value of these technologies and understanding that is part of a digital literacy that is so so crucial in this age of AI when many tools are being launched on the pro on the market and don't really have very good quality. So in summary, digital literacy as I started by explaining right at the beginning has been through many different types of definitions. These days we are really looking at understanding where AI is part of influencing the quality of what we see on the market. And we need to know how we can select high quality atte. And what is quality? What do we mean by quality? That kind of selection process should be guided by a comprehensive and holistic understanding of what we mean by quality of evidence. The 5 framework is a good guide because it's been consolidating 65 different frameworks that already have been existing on the market. It has these five E this efficacy, effectiveness, ethics, equity and environment. And they are essentially five different ways of describing quality and certifications such as the ones provided by edu evidence they can give a signal on edge tech quality including AI and I'm highlighting edu evidence because it is using a solid framework these five E for knowing how we define quality and how we can really promote quality if it is missing in the specific products on the market. So I hope that this short presentation shows you that digitally literate users in today's age need to understand what is quality technology and in the adtech space in the education technology space we have a solid framework that can be guiding that understanding and that framework is the five E and I hope that uh the abbreviation is something that can stays with you and we can discuss more whether it is fully capturing what you define as quality or whether you would want to perhaps see other elements included there. >> Thank you. >> Thank you as well. Here are my contact details if people want to follow up on specific uh references and I think we have some time for questions. So let's take them now. Thank you, Natalia. Any questions? I think they're a little shy. You can ask me question. Essentially the question is that She understands that it comes from the 5G frame or comes from UNESCO. But she's very curious whether this will be affected and biased by whichever the funding sources come from if the product developed receive funding from specific or many sources and how will we be able to know um this is fair and just and unbiased. >> Oh that is such a good question. Thank you very much for asking it. It is something that is of concern to many of us because if you think about how big technologies like Google, Microsoft, Open AI and of course the mergers of them right one is trying to buy the other one so that they have the dominance on the market that is of concern if we are going to define quality that is influenced by the definitions that are provided by these big providers. And that is why I believe it's very important that we work with scientists and researchers who are neutral and are not being paid by Google or Microsoft but are paid by public universities. Now, of course, the global agencies like UNESCO and UNICEF, they rely on donations from specific nations, right? UNESCO is funded these days a lot by actually China. Um, in UNICEF, you have a lot of donations coming from specific private actors as well. I mentioned the Asian Development Bank. So we need to be mindful that all the work that is happening has funding from somewhere and the same for the universities. You know many universities are struggling these days to ensure that their costs are met with only student fees. So they're also accepting donations from Google, Microsoft and have you. So to find a very clean way and a really unbiased way of doing something um these days would be difficult without any funding. You do need the funding to come from somewhere but it needs to be in the questions of quality in adtech really something that is collaborative and not dictated by one or two providers. So I believe that when it comes to these frameworks and quality definitions, the more we can ensure that they are based on an international consortium of people such as UNESCO, such as EDU evidence where you have multiple people, multiple groups coming together, the more we can ensure its independence so that it is not one framework recommended by one big technology, but it is by a consortium of funding members. So that is my answer. Try to look for frameworks and definitions that are coming from a consortia that are international and consolidated and are not single cherrypicking funded directly by the industry. >> Thank you so much. Uh I have a question for you as a first on AI and uh education um technology. I we are c because for the first um section we are talking about um the mobile bank on teenager uh and uh actually there are several uh countries has already set up the new regulations and uh b the young generation to use mobile phone at school. So how did you uh as a AI literacy experts how did you see those um banks new regulations on the young generations >> the audio was not always picking up but if I understood correctly you were asking about how the AI regulations are influencing the adoption in schools is that Right. Yes. >> It's about um whether what what you think in your opinions on um adopting like for example regulations for foam bands in schools across different countries. >> Yes. Yes. Thank you. Well, I live as I said at the beginning in Norway and in the Nordic countries, Sweden, Finland, Norway. Um I believe also Denmark recently they have as the ministries of education have been issuing these very strong warnings that young children especially children in kindergartens should not be using digital devices and we have seen that replicated I think it's in 70 different countries where there have been these mobile phone bans you and check on the UNESCO website because the UNESCO gem report GM report has essentially um issued um this warning that if we are not careful enough then there'll be more and more um not addictions but I would say really unhelpful and ineffective uses of technologies. Let me just double check how many countries have been. Uh 79 countries it is at the moment. I've just checked that on the um UNESCO website. So by the end of 2023 there were 60 uh different countries and now it's 79 uh different countries where mobile phones have been banned in schools. Now I don't know how much you have been following the discussion on this in the research circles but as a researcher I am advocating together with many other colleagues for the need of effective training rather than banning. When it comes to primary schools and secondary schools, we really need to ensure that young people know how to use the technology rather than just to remove it. So researchers who I work with and the research that I have done is not promoting a blank ban that we just remove all the technologies but rather that we really have digitally literate people who know how to use the technology who know how to select the right tool for their purpose and their needs. and they need to have their own agency in deciding whether they want or don't want to use the technology. So I'm not a promoter. I'm not a fan of these bans. They just remove that. Um in my opinion it is removing the agency the single right each human being has in terms of deciding uh what are the tools and resources they they want to use. Having said that, when it comes to very young children and by that I mean two, three and four year olds. So very young children who are in the process of developing their brains. They're in the process of developing their social and communication skills. These very young children, they are really not at the stage of knowing which technology is good or bad for them. So for the very young tender sensitive age of infants and babies I am all the advocate for not using technologies with them and in fact I have done a lot of studies where we have seen that when even mothers use technologies with young babies it is not recommended but for older students and for children at the age of primary and secondary schools I believe we need to have better training for the teachers. We need to have more engaging lessons and lectures so that students can be using the technologies purposefully and banning it is not going to be a solution for purposeful and effective use. So that is my opinion on that. >> Thank you Natalia. We have room for one more question. Hello. Hello. I have a question for you. As far as what I know that the five framework is designed for a specific hashtag tools uh for example on the hashtag website most of the tools are are designed for specific purpose right for example or crystal and so on but in terms of generate sate I could be for can also is suitable for deterine determining the impactfulness of the relative the rules. >> Yeah, that's it. >> Yes, if if you can help me with the end of the question. I heard you were saying that Cahoot is on the certification website and that they with the AI. I didn't quite hear the connection there, but it's the audio. Sorry. I think if you Yeah. So essentially that there's tools famously known by people such as Cahoot, but he's curious whether generative AI tools and learning tools um will have their own assessment method and criteria. >> Ah okay. Yes. Oh that's another great question. So the 5 framework was developed in a way to ensure that as we are getting new wave of innovation and right now we have the innovation of generative AI that the framework can stay over the years because we have seen with the evolution of technology that there is always something coming new. Now we have the LLMs and Gen AI but trust me in five years there'll be something new and different. So we need a framework that can stay and that can guide us throughout all these different innovations. So under ethics one of the ease the ethics strand that does include the generative AI questions and cahoot indeed it's a Norwegian company but used globally they have been assessed in terms of their ethics on how they are using generative AI in the different quizzes. So it is embedded. It is part of the framework. But the framework is not having AI as a separate or new category because it really is developed with the belief that we need to have a framework that can accommodate AI today and maybe the next thing in 20 or five or two years time as innovation progresses. So AI is part of the ethics strand of impact in the framework. I hope that answers the question. >> Thank you so much Natalia. And now we're wrapping up the session and we have learned so much from you and we're very happy that you're here to share your knowledge and ethics. Well, thank you for having me and I hope you enjoy the rest of your day and you get some sunshine or fresh air so that uh you know you can uh think about the different ideas shared today and if you want to follow up I'm available on email, LinkedIn or wherever you want to find me online. Take care everyone. Bye bye. >> Thank you. Bye.