ChatGPT and the Future of Education

okay all right so this is the current version and I'm going to get started right away but I'm just going to drop the link in here quickly for anybody that wants to uh to see the link I got people from Good Old Manitoba Canada hey I spent a lot of time there in Sunny Winnipeg I'm sure you're enjoying it my family still lives South okay and Dallas hello hello all right we're not gonna be able to greet everybody that's coming in but you've got the link so I'm going to get going just by sharing our screen and talking a little bit about what this event is I'm going to throw it over to Hassan in just a couple minutes he's going to coordinate and moderate the session today with an exceptional group of panelists so I'll talk a little bit about Grail who we are and why we exist so Global Research Alliance for AI and learning and education we were formed about three years ago or so and our primary intent is to look at a number of things related about taking the idea of AI and moving it from research Labs into practice there's a large number of very active Ai and education communities out there and so our intent is to try and support that translation now we also have a focus on leadership development a number of us were involved early on in the development of learning analytics as a field it's been an exceptionally successful organization solar being the one I'm referring to one of the things that I had hoped we had done more on early is to support leadership development to Foster effective deployment and that's we're trying to address here we also want to look at as a space for Learning and capacity development in general and this is for anyone faculty staff support staff you know just general members of humanity and then we have the desire obviously to be a sort of a global forum around sense making in this area and since making we're referring to the fact we're all part of these conversations happening in our institutions happening across our professional communities and we're trying to make sense of what does this mean right now the current iteration is chat GPT the broader question is generative AI in the long run we may be at some intersection of AI Robotics and who knows what so we want to make sense of that through a conversation the way that Grail works is we are tiered on three levels we have charter members as you see the universities on your screen these are the ones that stepped up and said hey we're willing to kind of support the development of a professional Community to help solve these problems and that's been the primary intent we have about another four universities that we're bringing on uh hopefully in the next few weeks or so and at the end I've got some results if anybody's interested in tracking it then we have our Advisory Board uh where we're looking at different labs and areas that are doing active work in this space and then engagement with PhD students and individuals doing active work here four areas of focus we want to Target the infrastructure of AI the practices around AI the general skills and literacies needed and I think most importantly the leadership Dynamic what's required in helping deploy and accelerate AI engagement we've got an upcoming event that some of you may be interested in I'll drop these links into the conversation or the chat form in just a second uh we've got one coming up that dragon and I with some colleagues will be running next week looking at what does senior leadership need to know about AI literacies for deployment in universities a few activity is going on I'm not going to spend a great deal of time on these just want to flag them we've got a study going on that's been running for for about 18 months now on sense AI tracking AI Trends we've got our fourth installment of the empowering learners for the age of AI happening in person at ASU in December we've got a journal that's targeted on rapid publication our first issues out in March looking at AI learning analytics and learning sciences and we have a number of workshops and there's a few that will be added here this is particularly Charter targeting our charter members and their faculty and staff so just be aware of this for now reach out to uh to Grace for myself on these and these are our workshops really culminating in a broad view of an AI Manifesto in education if you will your University wants to join stuff's on the screen there again I will drop this into the forum as well if there's any follow-up questions and there's a few random things if you want to stay involved this will be dropped in the chat as well uh around newsletter Weekly Newsletter on sense making Ai and learning um we've got a Google group that we just kicked off yesterday to try and Foster some two-way flow of conversation and then we've got email updates related to Grail specifically on your screen without any further delay I'm really pleased to throw it over to some fantastic colleagues and friends that have been really uh prominently involved in just some of the journey over the last three years that we've been sort of more directly involved in the AI education landscape Hassan over to you okay um thank you George pretty warm introduction and to grail for uh hosting the session uh good morning evening afternoon everyone from wherever you're joining us from around the world it's lovely to see the hellos from all over the world coming to us um I'm delighted to be holding this panel discussion on chat GPT and the future of Education I'm Hassan kostravi I'm an associate professor in data science and Ai and education at the University of Queensland um in Queensland Australia and on the panel I'm delighted to have professor Charles yesterday Professor dragon gasovich and Dr Anisha bakaria talking about chat CPT in the future of Education with me I will introduce them more formally as we start talking with them um can I start with a quick acknowledgment of the land and the traditional owners and their custodians and the lat that the presenters we we pay our respects to their ancestors descendants and continue cultural and spiritual connection to Country we recognize their valuable contributions to Australian and Global Society um it comes as no surprise that AI is said to transform the world generating an estimated 22 trillion dollars worth globally by 2030. um and as part of that development recently we've been looking at generated AI as one of the latest developments in in AI world uh which is going to be the focus of our discussion today in particular one of these technologies that has been getting a lot of attention is chat GPT which is an intelligent chatbot created by openai that uses underlying language models to interact in the conversational way um it forms a dialogue which makes it possible for chat CPT to answer follow-up questions it allows it to admit to its mistakes challenging to correct premises and reject inappropriate requests if you have any um on the right we've got a image of my version of what Chachi PT looks like um it's a friendly and happy chat about sitting in a nice place in a world having coffee and answering to all the questions that that are being thrown to it all around the world it is one of the other developments in the AI world where for creating generative AI which perhaps needs its own discussion at a different time I'm currently in early February 2023 as we're having this discussion it's still free to have and create an account for chat GPT uh using the link that I've got on the link chats.openai.com if you haven't done so already I'll encourage you to create an account and play around with it um however there are General thoughts that um soon there'll be subscription models there already are subscription models but the free version might might go away at some point now given that we've got quite a diverse audience joining us today um I thought I'll just put some quick discussions about some of the general use cases of chat GPT that are that have been well studied and quickly go through them just to speed up everybody and have everybody on the same page and then I'm going to open it up a discussion with our panel members and so in terms of common use cases that have been well looked at with chat GPT you can ask it to write essays on almost any topic ask it to write on global warming imitating the style of Shakespeare and it starts with all mankind or without blind to fate not your thing you don't like Shakespeare much you prefer to see it in Donald Trump's voice you can go and see that it's a post and it will write it in Donald Trump's sort of language instead um it's really good at writing lyrics also in any genre you want to see a song on a rap on education in the age of AI in the style of Tupac Shakur and and there you get it in in a matter of seconds um one of the sort of pro productivity um use cases that is being well studied is giving it letters to write so you can give it some initial information and it will draft um for example here a cover letter for an internship position and the fact that it's a conversational discussion means that you'll have an initial draft and you can keep on asking it to be improved you know ask it to change what you like or you don't like and you'll get better versions as you go got a new paper out and you want to promote it on your social media you can spend some time you know drafting that yourself or you can ask chat CPT to create a LinkedIn post promoting your article you can perhaps give it the title and the abstract and it goes away and creates an initial graph to that for you to revise and consider it's also really good at writing debugging and explaining code um here's an example of it writing a Python program that finds the first 10 prime numbers it writes it first and if you want it to to explain it to you it'll have a set of steps in order to explain how the function works I think there was around 12 steps all together I could expect them all on the page um a good use case for for Learners it provides step-by-step solution to questions from many disciplines here we've got a simple math question solving for x and it will not only give you the right answer but it gives you the step-by-step explanation almost like your personal tutor helping you understand how this works as you go and if you've got clarifying questions again you know if you need more information on a step you'll just ask it and it will re-explain itself or provide further details on that um this has been more one of the more creative ways that it's been used to to develop your creative adventure stories um here's a story on a student attending a high-tech University in the future he'll start the story and it will give you different Pathways so pathway a will do this pathway B will do that and again it'll go ahead and explain each of these Pathways for you to consider if you want to write such a story um also just a few um particular use cases for educators and education I'm taking these thoughts and examples uh by Tory trust from University of Massachusetts Amherst creating a lesson plan so if you've got a particular topic or a course it will allow you to create a initial draft level lesson plan you want to design a class syllabus for a course on educational Technologies it will give you the description courses textbooks for you to consider and revise policies for class syllabus you know if you want to policy on the use of AI and writing tools it'll give you an initial draft of that learning objectives for for example ethical use of AI and education and here you go these are some potential first uh draft of learning objectives for you to consider designing quiz test questions um you want 10 multiple trust funds 10 multiple choice questions on your on a topic of your choice and it will give you those questions scripts podcasts or videos um as well as rubrics um here you want a rubric for video projects and again it will give you an initial graph of what that would look like uh another point for productivity if you're sending out a lot of emails to students you can give it a quick overview if you want to send this to Lily who's missed the class and you want them to do two weeks of catch up by doing blog and here you go it will draft the first version of that for you to consider um so I might I might pause the there in terms of some of the main um use cases that have been well studied before we really want to capture your thoughts and perspectives as we go as well as we're walking through this um so essentially we have a padlet board for you to consider I will share the link in a second uh in the chat uh but essentially we want to hear your thoughts about use cases and implications for educators Learners researchers at Tech developers and University leaders if you've got questions for the panel members please also feel free to add them to their padded board and like the questions that you like and we'll keep a close eye on that and work through those questions as time allows and if you've got any helpful links or resources your own or others that you think others would benefit from please add them to that list and uh and everybody can benefit from that foreign there and pass it over in terms of questions that we've got to our panel members um Dragon maybe we'll start with you um so dragon is uh dragon gasovich he's a distinguished professor of learning analytics in the faculty of Information Technology and the director of the Center for Learning analytics at Monash University his research interested learning analytics center around the development of computational and design methods that can advance understanding of self-regulated and Collaborative Learning he has many prestigious Awards recently Dragon received the 2020 ACM distinguished members award for outstanding educational contributions to Computing dragon is uh it's lovely to have you here with us perhaps you can start us off by talking about your thoughts on Main implications for education and Educators and where you see this going in near future thanks so much Hassan Let's uh it's certainly great to talk today to everybody about the issues that are related to generative Ai and chat GPT in education one of the main things and I want to repeat many of the things and many of the lots of excitement about what we are seeing with Chad GPT can do many impressive things but I think we are still in the realm of speculation jet GPT can be good for that chat GPT could be used for this chat GPT could be also beneficial or whatever or can be harmful for whatever what we don't have at this stage is evidence we don't have research evidence that basically tells us whether it's really useful for that or whether it's detrimental for particular types of skills and I would like to discuss maybe three key issues that I'd like to emphasize that we can look forward and the things that require further investigation and potential attention that we all need to engage in to make sure that what we are getting with jet GPT and the underlying language technologies that is to say large language Wallace can be beneficial the first point is I want to say is that fluency is not a bliss for Learning and we know from lots of research in cognitive psychology that having fluent responses or fluent attacks or feeling that something is flown to us is potentially detrimental for our learning there's lots of research showing that Learners typically that they when they feel something that has moved fluent to them they will least likely engage into such context with such a content and therefore that will leave what do I label as lazy metacognition so then the key question for us is to look for development of approaches that can help students identify weaknesses in such beautifully articulated fluent responses and the other thing is also to think about as well how we can potentially use tiny bit of that confusion in such responses so that we can build on something that is already unknown in the literature such as uh productive failure or induction of confusion that's done by the city developing Arc racer perhaps the opportunity is there to think about clever ways to engineer problems that we can ask the chat JP team to pay the weaknesses of chat GPT to spill out lots of rubbish or educational purposes so I think that's really one of the important things we need to pay attention to uh the other thing is obviously feedback is a challenge uh for many of us in education and the key thing is how we can provide effective feedback uh in the last four or five years we've been looking into automated ways of generating feedback as well as potentially analyzing different types of written products including essays or case notes students alone are doing Etc and typically that's done through the analysis of different rhetorical structures trying to provide certain types of visualizations Etc and we've seen also lots of Promise by using deep learning and also large language models mostly in the past we've been building on birth which is our first cousin to uh gpt3 that is underlying GPT and we saw that we can get Fairly reliable outputs of that so then the research also shows that and the question is then if we can do these type of things then the question is can we also leverage Technologies such as chat GPT to provide students the feedback as well we did some preliminary work on that and our students them as the GPT to provide feedback on students written products and compared compare those to the human ones and what we found is that obviously GPT was much chattier than human feedback providers it was also providing which was a bit surprising to me some level of feedback on the process level but not as much of that process level feedback as the humans would be providing and also we don't really know what is the substance of that process level feedback so I think more qualitative research would need to engage into in the future to get to that level and then the other thing which is really not surprising whatsoever is that it was completely incapable for providing any feedback on self-regulation level I mean that's in value completely impossible anyways without having any process type of data and also being able to comprehend what's going on so then what is really to me having this type of uh some of the preliminary results that we submitted for publication already is leading us to the question can we use chat GPT in a way that Learners are engaging in a form of peer feedback the topic Hassan you've been studying as well without necessarily having the other peers so in a way ah this kind of feedback compares to peer review and also us Learners to provide feedback to chat GPT or whatever GPT generates and then potentially then offer some level of analysis automated analysis of their feedback as well as some level of automated analysis so what GPT generated to them so that we can promote deeper engagement with the potential resources that are out there and final point I want to say in this opening statement is the kind of the thing is that emphasis on the product that we are seeing with chat GPT lately is not in my view the most promising one we should always be able to really think about the emphasis on process but the key issue for that is that we need to collect the right type of data so that we can also engage into the process learning Analytics you you could probably expect for me that kind of bias that is coming from learning analytics is I think offering lots of opportunities to collect and then to analyze that type of data what I see as the promising direction to do that is to create pedagogically valuable instrumentation tools that are promoting tagging highlighting or note-taking type of things and then create specialized tasks that are asking students to seek for particular issues or particular types of rhetorical structures information and other features that could be identified that are spelled out by GPT my view is also that we need to look also for additional types of tools that are combining text analysis to encourage Learners deeper engagement with the products that are produced by GPT we really need to make sure that students recognize that chat GPT will generate lots of rubbish that GPT also is built upon the technology that doesn't have the sense of Truth has and will never have it that type of Technology without radical change in the part I will never know what is the truth and so then the key question is then if you are engaging much more into the assessment of that process can you do it reliably and can we do it in a pretty valid way so thanks so much and happy to have further conversation thank you Dragon for that um I can see a lot of actions happening in the chat as well as I'm padlet uh please do feel free to to contribute to both of those as we are holding the discussion and questions for panel members are also being added and I'll get to those in a second um let me move on to Shazia so Professor Shazier Sadiq um professor in computer science at the University of Queensland and director of the Australian research Council industrial transformation training center for information resilience Chelsea is passionate about the the positive impact emerging Technologies from data science machine learning and AI can have on our future to advocates for responsible and ethical technology developments and believe strongly that those developments require transdisciplinary collaborations between research industry government and the community it's great to have you Shazia here with us today maybe following up on some of those discussion points from Dragon uh we've seen a huge heart around the abilities of generative Ai and chat TBT in particular um we've had previous Hypes is the hype realest time and perhaps you see limitations or shortcomings that we should consider and be aware of as we Implement and adopted thank you for that question Hassan actually really important question to be asking um a warm welcome to everyone from me as well and really pleased to be part of this conversation and amazing to see the big audience over here so um yep indeed we we see a huge interest in the technology since its launched last year uh apparently it took uh Tick Tock nine months to get 200 million and it took uh charging Beauty only two um but you know spectacular Heights and Falls are sort of uh no stranger to to AI as many in the audience would recall you know the so-called AI Winters uh there was many many you know Hypes in history and then there was many disappointments around symbolic Ai and expert systems and what that resulted in is a pulling out funding from both public and private sector and you know AI research and Investments they took a huge dive um so I just want to go down to to to the elements that came together to to see the current height um so even in those days of Gloom actually there were some amazing scientists like Jeffrey Hinton you know sort of persevere and continue to make these fundamental advancements in deep learning which we now is know now is behind a lot of the recent successes that we see so it is important to know that the current height or the upsurge is riding on the back of two other major developments in the world of computing one is the availability of large amount of data or Big Data everybody knows that and the other is the access to cheap computation you know so various architecture scale up scale out architectures cost effective business models for cloud computing and so on so combine all of this combine the scientific advancements the Big Data the computational power with an obscene amount of funding injection from Winter and public sources depending on where you're looking in which part of the world and we are sitting now at this height of the hype cycle and of the so-called Foundation models right and GPT 3 is one of them inverted another you know Microsoft invested what 10 billion and just this morning I saw you know Google's response uh into what they are calling bad um supposedly a tragedy driver there's no accident really that um chat CPT and Technology such as this have emerged and you know there is amazing signs massive amount of data huge computational power and a lot of funding behind it so the hype is very much um you know the interest and the hype is very much real um but obviously there's no technology without limitation so if I may just pick up on a couple of points um that I think uh responding to the second part of your question of what we need to be aware of and specifically I'd like to pick up on the question of input data so I asked GPD itself so in its own words the input data like most language models is a large Corpus of Text data that includes books articles websites and other written materials so quote-unquote now as with all language models we know input data is used to train the model so it can learn the patterns now I saw in the chat previously is is a prediction machine basically to predict an expert input data is also used to fine-tune the model you know for specific tasks and I saw some condensation on chat before with regards to code creation so very different from essay writing and obviously the more diverse the data the better the model will perform but typically this data as we know is not ready to be consumed by the models just as it comes and requires actually a hefty set of data preparation tasks before it can be fed into it and and for for text Data these could be things like removing special characters punctuation stemming tokenization and importantly the selection of the Corpus itself so we know from our own research here in data quality management at the center for information resilience here at uq that the data preprocessing and the curation of the data to make it so-called model ready is one of the most time consuming aspects of of AI and data science projects and takes a huge resource um out of of the data scientists job so tedious as they may be we uh you know if these tasks are compromised uh we end up with a garbage in garbage out situation or even like you know um Dragon also mentioned that the data can Inspire us or misleading or even harmful um so you know some of you may have had some interactions uh with chatgpt and as a response to your question it may have said my knowledge cutoff is 2021 you know uh for example I asked it and it says who is the prime minister of New Zealand and it still gives the answer of Jacinto Dan so if anyone wants to use this technology to navigate for example institutional assessment policies or support student advising or any other settings which are highly contextual so the data is specific to the organization or a region or it is highly Dynamic so the real life phenomena that the data represents is fast changing I think we have to be prepared for a huge upfront and ongoing effort in data preparation and I'm particularly intrigued about the size of the effort made by open Ai and chat GPT to conduct the data selection and the preparation tasks that are given this qualitative response you know making us all wide-eyed um so I'll stop there and we can pick up on some other elements later thank you so much thank you Shazia um um let's move on to Anisha and look at it from a different perspective uh Dr Anisha bakaria is a senior manager at learning analytics at the University of Queensland uh here working working with us our primary responsibilities include directing the design development and implementing on learning analytics products and issues also written nine books I'm always impressed by that a programming in web developments um Anisha it's uh great to have you here with us today um you've been involved with various developments of etsec do you have any thoughts about new exciting Ed techs that we might see coming emerging in the age of generative AI or or thoughts beyond that over to you uh you're still muted Anisha okay uh sorry thank you Hassan uh it's certainly an honor to be on uh such a prestigious uh panel in such great company um so my role today is to imagine a future of educational technology tools uh that can build upon these Foundation language models uh such as chat GPT um initially I'm going to leave behind some of the challenges and concerns that I myself have but I'll return to them towards the end um so essentially one of the things that really um impresses me and makes me quite excited is that we actually have a new communication layer with computers uh via what is being known as The Prompt and the prompt is really just a description of what you want uh important is that it's in plain English um I feel more excited about chat gbt than I do about any of the predecessors EG gpd3 and the main reason is because uh with chat jbt not able I'm not only able to instruct and ask for what I want uh but I'm also using a lot more cognitive tasks uh so I can read what come back what comes back I can analyze it I can ask for changes and direct the output um and then finally uh arrive at something that I think is of high quality and I think it's usable and so there's a lot of human skills and a lot of um the relative judgments a lot of critical thinking skills that go on in this process um and very important is it's not just a single prompt that I'm writing a lot of the time um so here's just an example of me going through and what I was trying to do is get some idea generation not only did I want to add to these ideas but I actually wanted it in a special markdown so that I could display it in a chart um and it took a sequence of interactions with chat gbt um including me even teaching it a new markdown format which it was able to learn and then I was able to copy that um markdown that it generated and actually generate the charts with no problem uh so this is capability uh and the sort of looped iteration that you can have with it is quite powerful um but this leads me to the first set of tools in education technology that I think we are going to need and this is very in sync with what dragon was talking about basically is there's all these new thinking models uh that go on some of them are actually quite high level skills and you're actually swapping between them a lot more this is actually one of the so the the diagram here and the paper reference it's one of the first papers that I actually was able to find that was trying to conceptualize this and map it out um and so when we're talking educational technology and working with these products we really have to be thinking about the competencies that people are going to need um and what tools uh instructors might need to help them along that path and so I do see a really big research gap for tools to work with these new modes of thinking uh and tools to help Learners as well as Educators uh scaffold these particular tasks um the second tool idea uh I had and I should say none of this is actually my idea and I'm actually seeing a lot of this currently happening in open source and startups in this particular space at the moment all building onto open ai's apis um so yes chat gbt is limited to 2021 uh but a lot of what's happening at the moment is people trying to combine their own content but still get the benefits of the art the question answering and the reasoning um and uh this is one stitch tool the one I've included here is open source and basically it's documentation but you can actually ask it questions and chat gbt not even not is actually using the content from this particular site itself to formulate the answers and also providing references from of where that actually came from so the link is there I've chosen a free example for you to try um the recipe for this uh in terms of prompting it's actually quite simple it's really that you're taking the user question you're doing you know very doing a semantic search locally finding related information and then the way you're actually sending it to chat GPT is via Ace a text prompt where you're actually saying given the content below can you answer the question that's posed by the user um this basically I see as something that you know if we were if we wanted to actually Implement chat gbt in a lot of uh you know think of it from an instructor trying to bring the functionality that it has to their own course content whether it be lecture recordings um HTML content learning activities um this becomes a very real approach to that but you know imagine a learner being able to interact with this you know ask it to Simply buy things after ask it to explain things in a different way perhaps even translate it to their native language we'll get all of that functionality and I'll take that a step further and actually imagine that this could be available for all of the courses that a student has encountered at a university um so it doesn't matter what course you're currently in if there's material that you need to refresh on from you know your first year it's actually contained within this and this is all possible with current technology using chat GB3 apis or even the chat GPT um three apis thank you um next I really just want you to think about tool creation and now that we have this new form of prompting which is plain text I think the possibilities of you know who is actually the Creator it does not necessarily have to be the programmer or the researcher and even in creating tools you don't necessarily have to ask you to actually write code um so here's an example is really trying to push the limits basically and I was asking chat gbt you know basically I want you to three paragraphs but I also want you to annotate them with spelling errors essentially and provide some feedback and I'm very specific um I always try to be polite I'm not sure why uh very specifically telling it you know I want these tool tips but there's an error and I want it coming back in HTML um and you know I ask you do you understand it says yes give me the paragraph and it gives me back this didn't quite put it in a code editor but um if I render this basically uh what it's given me is from the paragraph which had errors it's able to highlight and when I Mouse over them I can see what the error actually was um It's Made its own basically by one prompt and made a little writing tool that can give students feedback very similar to you know some of the intelligent tutoring tools are but this capability is and the tooling for it is what we really need to be providing our Educators in the future um there's a little bit of creative prompting and you know the field of prompt engineering has definitely emerged um another little example here is um and this is examples I posted on my blog post right as a learning designer and it's not only getting it to generate questions but it's to generate question markdowns so that the tool that I take it in can actually render those questions and automatically mark them so uh yes it is a four-page prompt that I've written here that's quite detailed but I'm actually teaching it to do something new um and it's able to write the questions no which answer is correct and actually Mark that off um so previous examples are really about um essentially what we call zero shot learning so it's a single prompt to get what you want from it these Foundation models such as gpd3 and chat gbt um they have another technique that's available called fine tuning and so fine-tuning is what most of the startups that are using or building on these apis are doing either via custom prompts or custom data sets um and so this is really where I see the role of uh the entire edtech research Community coming in uh Ai and education is not new so we've had many decades of research from the AI Ed the intelligent tutoring system and the learning analytics communities there's been a lot of interactive tutors that have been built that have been very successful have pedagogical value and also have had a lot of research and evaluation being put into that and so the example I've put here basically is the reflective rating tool ACA writer that's from UTS Center for collective intelligence um and that's one such tool and I really think the research community has a huge part to play because from the tools that they've already built and all the evaluations that they've done they actually would know a lot of the custom rules for building these tools and they would have the data sets to actually fine-tune these particular things so I think this could be a really exciting time for educational research where we can use these large language models the chatbot capabilities so the reasoning capabilities and actually build on to it of course you know we're not without challenges um so things like Equity as charazia suggested these models take you know a huge amount of data training them is you know huge Computing requirements um even at the inference level when you're actually talking to it that takes a lot of compute power so they're not going to be free forever the examples I'm giving even if even if you're calling apis and you then have tools for every single student at a university or school that's fast going to become very expensive there's just inherent bias they're trained from data sets predominantly written by humans and so you know you can imagine what bias actually lives in there that we yet to discover there's no transparency about how safety filters are actually built on these large language models um I understand testing of supervised machine learning I have no idea how you would actually test from zero prompt and um few prompts scenarios to make sure that before you put something out is available there but you know these are research questions how can we safely build Educational Tools upon these models um and how can we get educational researchers to actually partner with the creators of these models both open source and Commercial and perhaps I see a huge role for an organization like rail to help in in making those um things come true so I'll leave you at that and stop sharing my screen thank you Anisha I'm very very insightful and a lot of great opportunities for the edtech community to have to utilize generative Ai and how we reimagine education um looking at the question so I'm just going to go to the questions and kind of try to summarize some of them open questions so everyone on the on the panel for response um so a lot of the questions I think are coming around this um this sort of concern so as we gaze into the future it's evident that Ai and Technology will have a crucial impact on the job market and many routine tasks will get automated most universities recognize the significance and feel like they need to be equipped in sort of their students with this Dynamic ever-changing job market and partnering with AI in their in their future sort of works at the same time they're they've got challenges and sort of ethical considerations in terms of how they can reliably assess students and maintain the highest levels of Integrity to ensure that their students just will be recognized sort of for the capabilities that they're being accredited in so this is sort of raising the question of how do we enable our students to benefit from the latest technology and advancements um in the world and in the world of AI while at the same time making sure that they hold the accreditation you know competencies that they should um just your thoughts on that and I think in particular you know tools that detect sort of users chat CPT have been an ongoing discussion um do we use them how do we use them do we ban it do we do we integrate it into our assessment design thoughts and over to panel members uh driving props maybe we'll go back to you yeah no thanks Hassan lots of interesting questions I think I mean I'll start from the from the original House Point as well whether these tools will be embedded I think they'll they will whether they should I don't know I think it's equally controversial as it is like these are the tools for plagiarism detection there will definitely increase the teaching workload that's all what I can guarantee for that we know basically from everything what we are doing as in the last 20 30 years in educational technologies that didn't really reduce the teaching load and I doubt this type of Technology will reduce the teaching mode we for example had a systematic interview a few a couple of years ago on automatic feedback Tools in education and we found yes they can be effective for many types of things but they didn't really do so good at reducing teachers workload so what routes institutions are going to take or not we know many of them are risk adverse so they would like to know some level of it I think it's much more relevant question is how we educate people about these type of things that are out there my view is that we should they should be incorporated because they will become a standard part of a toolkit that people will be your professionals expected to use we also need to educate people about the limits of these type of tools they are not the Magic Bullets this is not the human equivalent intelligence this is really the type of Technology if you are kind of saying yes it's good gives us a good impression and everything but in the 60s the lies I was also giving good Impressions and people were falling in love with Eliza as well so it's not that basically what we are seeing as an impression but what is capable to do and then you know can you trust fully technology that for which two plus two is sometimes four and sometimes can be whatever and so and that will not be faced and we need to incorporate development of the AI types of literacies into the curriculum we need to equip the people to deal with the issues and the gaps and these types of Technologies can do the issue with them is that let's talk about the fluency bias as well I fear that for any of the kind of reliable type of work when you need to have these certain accuracy analysis and information you need to turn to the types of things you understand chassis are looking into the information resilience so we need to look into the tools and equipping people to now even more carefully proofread and conceptualize what is said by chat GPT because you don't know where the mind will be inside of the process and it'll be somewhere on the way you just basically don't know don't know where your reputation will be on the line and therefore I think there are certain professions that are not necessarily particularly upset with these type of Technologies do you want to have a legal advice from chat GP team given this type of inaccuracies and who is accountable for it or do you want to have a medical advice from such a technology obviously it can be very productive I've been enjoying also gmails for the last two or three years or the complete which gp2 can also give you other types of benefits so we need to find the right ways and we should not be running away from those but empowered people to play with these type of Technologies as we live in that age I said can I add to that so so I do of course you know Dragon makes a number of really excellent points but no panel is too much fun if we agree on everything so I just want to go back to the workload uh issue that you mentioned um dragon and uh personally I do see these Technologies as an enabler as a productivity enhancer I think the workload issue definitely is a real one but it's a moving Target because the scale is growing you know what my class sizes used to be and I've been in the fields you know long enough to know that uh and obviously um you know if we can say the benefits but also the burden of digitization so um so I don't see it as a threat I think automation has caused anxiety all through our recent history and Technology Innovation you know if you uh look at you know other examples of the factory floor and where humans were eyeballing products for uh for Falls and then Along Came computer vision and it greatly reduced the set space of where people's attention was needed and so those workers who actually learned and adapted to the technology moved into better working conditions and this is the part I really agree with dragon on in the sense that I think for for Technologies we need to make for for Technologies like gpt3 um and chargpt we need to make the investment either uh institutionally systematically or personally into exploring understanding and selecting use cases where we see the benefit for for Learners it's not a like you know it's not a silver bullet I think Dragon used that term before and Anisha mentioned you know all the pros and cons but I think we need to make the investment there any show any any add-ons from you yeah um so definitely agree with those Dragon uh and Shazia here um for me basically I think uh you know I think not only using the tool but making good use of the tool where it's accepting what it returns it's you know critically analyzing but also directing the output of it actually needs a lot more high level thinking skills there's evaluated judgment there's been able to craft feedback and instruct it in very specific ways to basically take it to the new level and so you know partly it's intrinsic motivation of students that we want to Foster in that particular space but you know to get to those high level of skills there's there's base level understanding that's probably needed you know using a lot in day-to-day programming even a lot more time is spent you know on crop you know writing instructions for what you want and then you know analyzing me debugging it where you go through those Cycles oh and that that paper I showed was a good example of that how you know I I think there's a lot of fear of chat gbt what I'm not hearing a lot of is where in the curriculum do we think is the right spot to start introducing it um you know it might not be first year and it might be some base skills are required and even if we as a return to some you know face-to-face verified assessments but you know maybe a year two in year three it becomes where we start developing that because I definitely think those base level skills are going to be needed uh initially oh thank you um yeah thanks uh thanks all and it's great to see a lot of um active and great discussion happening on chats good to see many colleagues joining us from around the world and Australia in the chat um so a little hard to keep both on top of a few things but I'll definitely go through the chat later for very exciting discussions that are happening there as well um we have a few minutes left um I'll go back to one of the earlier discussions Dragon pointed out in terms of the need for research and actually for us to better understand the impact of you know chat TPT on student learning um it's been it's been quite uh sparse in terms of actually having any research there and one of the challenges is that you've got this tool which is somewhat a black box right controlled prompts at it and it gives you something as output and it's a black box which is also evolving all the time so you don't actually have a control that you can maintain while you're running your experiments so it's it's introduces quite a various challenges you know let's say you want to investigate what sort of problems are effective um you're trying it today and maybe two weeks from now that's no longer the case because there's been an update to the platform and it no longer is using the same props I'm just wondering if anybody has spots around you know effective ways methods how we can do conduct vigorous research on on chat GPT in particular sort of generative AI with these sort of moving discussions and thoughts and ideas around that thank you I think um that's an excellent point to to sort of close on um so the opaqueness of course makes it both um fascinating and scary right but but it is an amazing piece of technology I just demonstrates how far we have come scientifically from you know Alan turing's original vision of of the machine that convincingly imitates a human uh to me from the research point of view the most important thing is that we don't become polarized in our conversations we don't make the far camps and the against camps I think healthy debate is extremely important but especially within the research community in edtech in learning sciences and and learning analytics we know that the real progress comes when cross-disciplinary researchers um work together so personally I think and I think there were some conversations in chat as well they're banning policing will just postpone the inevitable so instead I sincerely hope we can sort of work together and make this technology work to our needs and our values rather than the other way around I can I can add to this um Hassan um there's a lot happening at the moment in open source um in developing tooling for people building on add-on products via apis uh where they're starting to look at you know how we can chain prompts together and then test results uh the most prominent one is called Lang chain uh there's also tools like gbt index as well as a whole lot of ones to help you craft prompts but also test prompts and test output that's that's coming from it and those might be essential uh tooling that's going to be required for researchers building onto these products foreign analytics methods can be quite beneficial in this type of research um not only in terms of basically analyzing what kind of products oh GPT and students are producing and using but also how they are engaging and how they are interacting with chat GPT as well as how we can then study the underlying learning and other processes uh probably we also need to engage much more into the ethnographic type of work as well to study the teachers side as well and the educational side and understand what kind of values and cultural issues these type of Technologies are promoting we don't know we know that some of these existing language models are biased we don't know anything about GPT 3 because it's not as transparent and we usually experimentation with LGBT Technologies not as easy as it is with uh Bert and the other types of Technologies so we need to engage into that type of research and the methods that are are looking into the computational types of bias as well that can be offered and the extent to which that will be possible is something that is still our programs thank you um okay we're just about five four minutes to um to the end of the session so I will maybe end it a few minutes early for colleagues that have other commitments right at the hour can get into it I want to thank you our panelist Dragon Shazia Anisha for spending your time with us today uh thanks to George for opening up and to all of you guys um it is an exciting time ahead that we're going to look at various ways that we're going to use AI I mean Ai and education has been around since the 1980s and I think for the first time the world is getting a little excited about what it actually can do in their own classrooms rather than just in research World um so thank you everyone I hope that we're the ways for us to maintain this discussion going forward I'm George I'll pass it over to you to final remarks aim ending thank you great well thanks everyone for for joining the event uh we had great participation in here fantastic text uh going on um I just want to emphasize again you know what an interesting time where we're trying to have a global sense making conversation around what does this mean and what are the impacts particularly we as a uh I guess as a species haven't had to cognitively interact with an entity that is Superior in a number of domains that we've claimed as our own we could certainly argue that you know nature is intelligent in a way that we might not necessarily be conscious of but in terms of knowledge work and the structure of Science and outputs that we've created we for the first time are confronting something that is approaching some cases exceeding our capacity for uh intellectual work and as universities and organizations in particular that's the core of what we do so we're going to have a number of existential conversations going forward and we have a number of events coming up both in person online um so feel free to join and track those conversations uh you know grail.ai is the website if anyone's interested in tracking or signing up for it I also think we're at an interesting stage where the first structure of interest in chat GPT is now giving away which was what is it what does it do how do we do this is now giving way to the more substantive as Dragon kicked off by talking about what does this mean and what does this do in terms of how we track and engage online and what are the implications Anisha talked about prompt engineering as a fascinating potential Direction on and in her statement about this is a new layer of interacting with technology and that brings right into the uh work that chazi and Hassan have talked about about information resilience and the questions around ethics and fairness and where did this information come from and what's happening with it so there is an enormous amount of activity going on in this landscape and we're hoping to see many of you join us in that broader conversation thanks all Hassan fantastic work moderating

Channel: GRAILE AI

Views: 4,534

Keywords:

Id: m7F6VhOkoYc

Length: 58min 32sec (3512 seconds)

Published: Wed Feb 08 2023