What I have learned from teaching with AI

today I'm going to be talking about what I've learned from teaching with AI before I get into how I've Ed it for teaching I'm just going to do a quick overview on generative AI so that we're all on the same page so generative AI is an artificial intelligence technique and this technique learns patterns from data and then performs some sort of modeling based on it to provide an output so this data can be in the form of text images speech or any sort of data we perform some sort of machine learning out model on it and we get an output this output again can be any of those different forms of data the aim is to mimic human created content and the most common example people are aware of is chat GPT chat GPT was released in November 30th 2022 and it was one of the fastest uptake of technology or app social media that is happened to date so the time to a million users for chat GPT was just 5 days when we compare this to other social media out there platforms Spotify Facebook Twitter and Netflix these are all five months to 3.5 years for Netflix same thing can be said for the time to 1 million users where chat GPT only took two months to get to a 100 million users Tik Tok Instagram Pinterest Spotify these all took many months nine months for Tik Tok to 55 months for Spotify this came with the problems that come with something exploding so quickly we didn't have time to learn about it or think about how we're going to integrate this into our teaching and how it's going to affect how our students learn and so because of all this it created many problems but now we've had a little more time to start to think about chat GPT and its impact on teaching there was a study I want to talk about first which was a study of 1,600 researchers throughout the world um this was published in nature and it looked at what research think of AI specifically generative AI like chat GPT and they asked them how useful do you think AI tools are for researchers in your field and respondents who use AI were saying about 80% said they were between essential to useful this was slightly lower with respondents who don't use AI or above about 50% or so saying that AI tools were either going to be essential very useful or useful and then when we expand this to how useful you think AI tools would be in the coming decade these numbers go up even more we're almost to about 80% of or close to 100 over 90% of researchers who use AI saying that they expect these tools will be useful in the field and if these tools are going to play such an important role within the fields then this is something we need to be thinking about teaching our students to learn about and I don't think there's something we can ignore as if they're going to be so important our students need to know how to deal with them and need to know what to do about them and think about what are the proses of these methods and what are the cons of these methods as well in this same survey they asked the researchers what they thought the benefits of AI or generative AI specifically is so models like chat GPT and the biggest benefit that these researchers saw was that generative AI helps researchers with who are non-native English speakers improve their language so editing or translation of documents other applications that researchers saw were encoding summarizing research writing and improving just general scientific tasks so we're seeing a lot of benefits that these tools may have and when they looked at the positive impacts of these again there was lots of positive impacts highlighted by these researchers ways to look through data so data analysis ways came up comp ation saving time and money getting new types of data process that we weren't able to do before so we're able to highlight a lot of different benefits and these are the types of things we need to be highlighting to our students and getting our students to understand the benefits of these but with the benefit comes the negatives there are negatives of these tools and in the same survey the biggest negative that came up about was the proliferation of misinformation and this is very important fact that we need to get across to our students and be aware of uh chat gp4 was stated to hallucinate information or give incorrect facts 9 to 12% of the time so one in 10 times you're using it you're not getting a factually correct statement and so the danger here is too that chat GPT will always give a grammatically correct answer so although that information may not come is not true it comes across in an authoritative manner in which you think that that information may be true compared to the Google when you Google something maybe the website looks it's a sketchy website you might not trust that you don't get that same feedback from chat GPT um other problems that came up were making plagiarism easier harder to detect and just bringing in this false information into the both into research and into teaching they also looked at the negative impacts of AI and it looks at that lead leads to an Reliance on pattern recognition without understanding so you're recognizing the P patterns in the data but you don't understand what that means you're not not able to synthesize the data the same way and we're entrenching the biases or discrimination in data so all these generative AI models they're trained on a set of data they know what was in that initial data they were given they do not know things outside of that initial set of data and so any yeah any information outside of that area the models don't know about and so this is partly what comes to that misinformation the model doesn't know about it it can't present that information so one big thing that comes about with AI is plagiarism cheating and that led to the creation of AI detection software and so openi which is the company that created chat GPT they released their own plagiarism detection software it doesn't work it was able to successfully identify AI generated text 26% of the time one and four chance that it was correctly identifying this AI generated text that its own code generated as Ai and then it also misclassified human ritten Texas AI on 9% of the time um so these are pretty poor stats for a tool that we're using to detect if it's AI generated or human generated it's going to go further and there was a study that tested nine different AI detection tools and for this they fed through real TOEFL essays so these are essays written by non-native English speakers and when they fed through these real essays the tools average said that 61% of these really written essays were AI generated the took a data set of real us 8th grade essays fed it through the tools now only 5% were AI generated and so this is showing a bias towards non native English speakers that are going to be very problematic in the use of these AI detection softwares they're not something we can reliable use furthermore you can give prompts to AI so if you take these real tol essays tell AI or chat GPT enhance the word choices you do that now 11% of those essays are determined as AI written significantly reduced how many are thought to be written by Ai and then we can do the opposite we can take those eighth grade essays we can tell the AI please simplify the word choices if it does that now it identifies 56% as AI generated so they are these prompts that you can use to modify how the AI detection tools would respond to these these are all over the Internet so if we can find them pretty simply by a Google search so can our students and our students would qu very quickly learn what prompts they have to give these tools in order to misp pass detection and not be targeted by this so there's no way we're going to be able to use these AI detection software so what's the path forward I think we need to learn to work with AI we need to teach our students about these tools we need to to teach them about the pros and cons and the type of information that you are going to get from these different tools and so in preparing for this presentation I asked my colleagues in the biochemistry Department who was using AI Tools in their teaching whether it be an assignment or just a class discussion and I had a number of my colleagues come back to me that over the last year and a half or so that they've integrated an AI tool within their class so we have seven classes within the last year who've done this and so in next I'm going to go through some examples of how these tools have been used in teaching these include both examples of my own teaching as well as examples that my colleagues have shared with me in other classes to give a more broad um introduction to the different ways these tools are being integrated one other thing though when we're using the your teaching about AI tools I know I'm doing this and I think um many other colleagues in the department are is we're telling our students that while we are being integrating these AI tools and we are allowing them to use for their classwork not everyone is and they can't apply what we have taught them or our rules we have stated saying they can use them to every other class that for each class you need to talk with your professor and learn if using an AI tool is appropriate for that class and this is a case-by case basis so although we're teaching it we're also telling the students to be aware that there are limitations and not everyone is okay with using these tools so the first example I'm going to talk about is in a thirdy year course within the department this is physical biochemistry course that I'm teaching this semester um so what I did first is I lectured about Ai and biochemistry and Healthcare we went through examples of how AI goes from data to generate a prediction so we could understand the basis of how AI works we then uh the students are doing an assignment these are do this week so I can't tell you how well the assignment worked yet but what they I can tell you about the process so far of working through the students and answering their questions on it um as a class we asked chat GPT how is AI used in Biochemistry and Healthcare we asked for a list of 50 items there are 44 students in the class each student's going to get a topic and they have to write an assignment based on how AI is used in that specific area the students had to then find ources they need to go do a scientific literature search on any of the databases that we use so we talked about using PubMed or web of science uh and Google Scholar scopist so different websites you can use to find real um scientific sources and they have to use these sources to back up the claims made by chat GPT and they have to write this assignment for a general audience so we talked about one of the biggest hurdles right now in applying Ai and health Healthcare is that people don't trust Ai and they don't trust it because they don't understand it so part of this assignment was to get the students to be able to communicate to the general public whether AI was a good tool for moving forward or if it had too many problems and that we shouldn't be trusting it moving forward um the students were allowed to use AI to help with the assignment writing chat GPT if they wanted to because the main focus I wanted to do for this assignment was Finding verified information and reading the literature I wasn't marking them on how well they wrote and so but if they use AI to help they have to acknowledge it they have to provide an acknowledgement statement in this of what chat GPT did or what AI did versus what they do and give that appropriate credit we talked about ethical use of these tools that we can't get AI to do everything that's the same as that buying an essay off someone this is not an ethical use and so they needed to make sure whatever that acknowledgement they said that AI did fit within the ethics of using that Tool uh Dr Sher Christian did a very similar type assignment in Immunology where she did prompts from chat GPT on immune system Health the students then had to go and find sources and look at the reliability of this information so both of these assignments are really focusing on getting the students to identify reliable information and compare that to to with what they're getting from chat GPT and determine is that information reliable how do we find Reliable information what is the value of and what is the value of scientific research um Dr Zar farac also used the tools in uh the sports and exercise nutrition course this third year course and she got the students to compare the information they get from chat GPT to the scientific literature so they had to pick a sports supplement on the market market and identify its health and performance claims understand the mechanism of action and validate the claims so they compared the two methods and gave an oral presentation saying what they thought each method was good at um most said chat GPT was really quick good short summaries of information didn't give them much depth but it LO it let them get the hang of that idea before they jumped into that scientific literature gave them a starting point to go off of um they said chat GPT didn't really give them the details on the mechanism of action and it wasn't giving them much for citations well and chat GPT won't give you citations PB Med they said it gave them robust scientific information they were really able to identify the difference in information you get from these two sources another interesting part of this is uh Dr farx surveyed her students before and after it on their opinions on chap GPT so before the assignment she asked if they' used it most of them are using generative AI with the most common tool they're using is chat GPT um and they use it for checking out anything new or unfamiliar to get that really quick summary of that information they find it's quicker than doing a Google search if they just want a quick summary at the end of the assignment she asked are you still going to use chat GPT now that you know its limits and they said yeah we know its limits but it's still useful in giving us that quick information and that we need to use it in combination of those scientific sources like using Pub to get information they really see that value in how quickly chat GPT can give them information it was they saw it as that initial step but then they could go validate it with scientific literature and uh proteins course is fourth year biochemistry course Dr Valerie boo did another assignment using chat GPT and in her she got the students to use chat GPT for the writing and so she asked the the students each had a topic and the first thing they had is they used chat ask chat GPT to write 300 words on that topic they then had to go to the scientific literature and find references for that information add references to what chat GPT wrote and if something wasn't wasn't correct they had to fix it they had to identify that correct information where they could put the citations and things that were not correct and maybe do some modification and once they had this corrected version of that report they had to feed it back into uh chat GPT and ask it to improve it in some ways they had to identify something through their prompting of what they wanted to improve maybe they wanted it to use more active voice maybe they wanted um less flamboyant language chat GPT like C lots of adjectives maybe they wanted to remove that so they had to identify something they wanted fixed within that writing and then at the end they had to reflect on it talking things about what chat GPT did well what prompts worked and what is the importance of that human oversight so through these assignments we're really focusing on identifying the benefits of these tools and ident what works well what doesn't work well and getting to the students to understand their limitations uh moving on from looking at an assignment an interesting discussion that came out of asking my colleagues about how they used chat GPT was with Dr Amy Todd who look didn't have an assignment based on chat GPT but she had open book open web assignment ceser exams in the course and so the first exam they just did it at home as normal an open book exam there was a time constraint on it and then after the midterm she did an Anonymous polling of the class to understand did they use AI when they were writing this exam at home and if they what sort of tools they were using how the tools were used did they use their textbook did they use chat GPT did they Google things did they have a reference sheet and what was most effective and then they talked as a class for developing some class standards going forward of what they thought was acceptable behavior in Asam and the use of these tools um so what she found out a third of the class admitted to using uh AI tools during their exam um this all but one of them used chat GPT and when they compared the tools they found that the chat GPT was just the quickest it allowed them to get information really quickly where a textbook you might have to flip through in order to find what you were looking for um using the a reference sheet their cheat sheet if they made it this was the best method but if they spent the time to make the reference sheet before the exam um and if chat GPT gave something that was unfamiliar that they had to fact check then it was no longer faster so what they liked when they asked chat GPT is they got an answer and they're like oh yeah that's what we learned in class they can answer their question based on that so they wanted it to jog their memory when they got saying I just like I haven't seen that before then they're like okay I should maybe fact check this I'm not sure if it's telling me the truth then it was no longer beneficial on the exam um and they thought it was really useful for the multiple choice questions um which is a bit dangerous that they're using this to answer multiple choice questions but the long answers they said it wasn't really that helpful um that it really that came to where they had to do more fact checking or rewrite it so they didn't find it as useful there and then also notably there was a group of students who lost a significant amount of marks on their exam because they trusted chat GPT and it gave them an incorrect answer for a question so she said she was marking the exam she had a lot of answers that she's like why are all the students answering this question with this incorrect answer so she threw a question into chat GPT like oh they're giving me the answer chat GPT gave me and so that highlighted the discrepan why they were all getting it wrong they didn't have time to fact check that information so what did the class decide on moving forward they decided that they needed to use terms and Concepts that were introduced in the class chat GPT doesn't know what terms you introduced in your lectures and it will often use terms that are more in the scientific literature maybe not something you talked about in terms of a definition in the class and they also said you need to directly and concisely answer a question when you ask chat GPT a question it typically gives you a lot of definition it's really wordy and quite often it doesn't ever actually answer the question it just gives you information related to that question so the students if they got information from chat GPT they needed to rewrite it in their own words and then connect it back to that lecture content with those terms and Concepts and avoiding plagiarism and being mistakenly identified as plagiarism they had to rewrite everything themselves they couldn't take a long answer Direct from chat GPT because there's no way to determine if two students have the same answer both from chat GPT if it's from chat GPT or if they copied from one another so to prevent this they all have to write in their own words even if they're using these tools so I thought it was a really good starting point for going forward of looking at how AI tools are implemented in our exams when we're doing open book exams it's definitely something the students are using and that we need to be aware of and then last example is within the biochemistry honors thesis course so this is our um fourth year students doing their honors thesis projects we have a course in which we talk to them about common things all the students need to know so writing and giving presentations as part of this course I gave them a lecture on using generative AI tools for writing and the pros and cons of different techniques and the pros and cons of generative AI in in general and we did a discussion of PR in when I did this I didn't just limit it to chat jpt but I talked about a variety of different tools these are all freely available in Canada that the students can be using so I gave them some examples that are useful for grammar so grammarly is is a generative AI tool that a lot of people I'd say recommend to students to use to improve their grammar so this is one of the examples um word counter paper raater quillbot they give different information on active voice or repetition of words within their writing so there a way for students to get stats on their writing and understand their common mistakes and then a bunch of General use generative AI tools um chat GPT three fine chat aonic perplexity elicit consensus and po so we went through each of these tools what they are good to be used for and what are some of the cons or things you need to be careful about when using these tools then as a class we did an in-class exercise and discussion about these different tools the students came up with a list of topics they thought should go in their introduction of their thesis they were working on writing their introduction at this point and then they asked multiple of those tools generative AI tools what the tool thought should go in an introduction of a thesis on whatever topic their thesis was on the students then compared these two lists what they came up with and what the generative AI came up with and what they really found is most of what they were working on for their honors research was way too new new for the generative AI to know about it it was not in the database of facts that the um tools are trained on so it really wasn't that helpful it was helpful maybe for those first couple really obvious Concepts that needed to go into their introduction but then beyond that the tools got lost they didn't know what they were um researching on they did find if any of those tools they actually liked find better than chat GPT and why they like this is find actually gives you scientific sources where it gets that information from and so they like that it would say give them a summary of maybe you should talk about topic X in your introduction to learn more about topic X here's a paper to go read so it linked the two together and allowed them to get information from the scientific literature as well um we gave them a statement on using generative AI in their writing we haven't banned them from using it for writing their thesis we base this on elvia's policy for AI and scientific writing and so students aren't done so we're not sure yet how many students are acknowledging or using the generative AI in their writing will be interesting because we have said if they use it they do have to acknowledge it in their thesis and they have to use it in an ethical way um so when we see the honors thesis at the end of the semester it'll be interesting to know of our 21 students if they actually use these tools or not after learning a little bit more about them so in conclusion the generative AI is not going anywhere we need to learn with it this tool is going to become more and more prevalent in our society think I heard last week that Microsoft now has generative AI tools that are going to be integrated into your Microsoft apps Microsoft Word Excel PowerPoint and I think these are going to become just like your spell checker you have Within These tools that you're going to have your generative AI helper or checker for your documents once they become standard in those tools they're even more widely accessible to our students when they're working on it so I think we need to start to teach our students to work with these tools teach them what are the benefits of these tools what are the cons and how to ethically use these generative AI tools um focusing on that reliability of information what information you can get from generative Ai and how to fact check that where we go for Reliable information and through some of these ex I think the students are really starting to understand where that line is drawn for accurate information from generative Ai and using that scientific literature to back that up and really I've been teaching the students throughout the year of don't ask these AI tools to do anything that you wouldn't ask another human to do so it's a really simple way to think about the ethics of it if you want your student uh student wants generative AI to read over their essay and provide them suggestions on how to improve it this is something you could ask your classmate to do therefore I would say this is an ethical use of AI we're just getting some suggestions on getting AI to say you're overusing passive voice maybe consider using more active voice that is a helpful comment to the student that the generative AI can do giving the generative AI the entire assignment question saying do my assignment that's not an ethical use and you wouldn't think it is ethical if you asked your classmate to do your assignment for you so this is a I found is a really quick way for students to have that check of ethics for these tools um yeah so that's all I have for my presentation today uh thank you for attending and thanks for my colleagues for their um discussions in preparing this and learning how people in the department are using these generative AI tools and I'll open up for questions well there is one question by Jillian wcot and that is actually a general question not towards the presenter himself herself but I'm going to read it aloud um does anyone have a statement that they include in their course syi course outline regarding the ethical use of chat GPT so if you have anything like that yeah so I put up on the slide what we've been putting up in our course outline for use of generative AI but I believe citl also has a set of statements for course outlines yeah personally I really struggled with the idea ofing for a few reasons for one I um when students are coming to this University or any University they often don't know about citation referencing anyway really have to teach them a lot and even by the time they finish their degree often they don't know um when we have students write papers they cite sources and one of the things I do is I like to go to their source to see if a the information actually came from that source and B if they've tried to analyze information right in their own words as opposed to just copying a paragraph or batim so when we're doing this chat GPT acknowledgement we're saying it came from chat GPT but I don't have that ability to go back and see where this information came from what it looked like so I'm just taking their word for it that they are not just copying swads of data from chat GPT and I mean the information is not sourced there anyway so it's probably not the best source of information but I would say acknowledging chat PT is kind of like saying I got this information from Google search you got it from a website or an academic journal or somewhere and you can point the instructor to that so they can look at it whereas with saying you got it from chat gbt there's nothing to to do that with so for me if I were going to allow that there's ability in chat gbt to save the chat and I would tell the students to use that link to the save chat and put that in the reference list so at least you have some ability to go back and look at it as oppose just saying chat GPT yeah so one thing I'll say for acknowledging chat GPT is that we say the students are reli are creditable for any information they put in there and chat GPT is not a source you go to get scientific information it's not to be reused in replace of a reference you still need your reference in there and it's definitely a good way for students to hand in like different drafts of a report so that you can see the progression and I also agree those uh chats with the chat GPT though including those as proof of how they used it it can be a good way moving forward as well uh so thank you Katie as usual for teaching me new stuff uh so having done 40 students having SM 40 students having done a this chat GPT assignment where they started with something that chat GPT generated the thing that I really liked about it was like usually I'm trying to correct for oh I can see this student is you know writing in something that's not their native language and I'm trying to like figure out like what is you know what what are they thinking versus like what is just like a language issue that sort of sounds like confusion but maybe the student you know knows what they're doing and the having them tell me their sort of dialogue with chat GPT and the prompts that they gave it made it possible to um sort of see their thinking process much more clearly in a way where I wasn't penalizing like non-native English speakers it made it really even and really clean that way so I would do it again I found throughout the last year our students have been really appreciative that we talking to them about these tools and that we're not Banning them all outright and that we're being aware that they're using these tools and talking about them how to safely use them and what how to ethically use them because they're all you most of them are using them so but they're just using them blindly so teaching them about these tools and talking to the students after these classes they've been really thankful that they've got to learn a bit more about these tools the question is from Don M and I'm now reading it aloud so you said that the tools are not great for learning in that they don't allow synthesis sythesis just pattern recognition do you discuss this in your assignments slcl class as well um yeah so any class I've done where the students are doing an assignment using um chat GPT I talk about the pros and cons in the 3105 so the physical biochemistry course this semester we did an entire week on AI and talking about how these tools work how it goes from data to an output and the students had a understanding of how the what's going on Behind these tools and so they could understand what's going on and we talked about pros and cons of these tools before the students do those assignments another question by Don M reading it a loud now so if you are not AI experts is there a source for this or do you give guest lectures no um so I've only recently started to learn about how AI works and machine Learning Works in the last a few years and I found it was there's some really great information on YouTube of lectures from other universities about these tools and how these tools work and so I thought those were really useful um for learning about these tools when we are talking about chat GPT like mainly we are like referring to version three right that the student have access what do you think about version four that they need I mean I'm thinking probably they give them more information in terms of validity yeah I've heard good things about chat gp4 I haven't spent too much time looking at it because it's not the freely available version that the students are using but I have heard really good things that it is an significant improvement over chat gpt3 some of those other tools like uh find that gives the sources is really good and that that's a freely available tool um yeah so find gives tools elicit is for uh literature searching so it summarizes a bunch of Articles so that's another one that's really nice for getting uh sources involved consensus you ask it a question and it tells you what is the consensus within the literature so it gives you both sides of the argument and whether the literature or the internet is preferred One Way um so you ask it it um is sugar bad for your health and it'll tell you 90% of the literature says yes or and it gives you the sources to go with each side um so it categorizes it which is quite nice um yeah the PO was a customizable it's a little more complex of a tool to use um forget perplexity and chat aonic I couldn't have been too impressed with them if I can't remember what they do

Channel: Centre for Innovation in Teaching and Learning (CITL)

Views: 2,315

Keywords:

Id: 0FvxMUYxVhU

Length: 35min 37sec (2137 seconds)

Published: Thu Mar 14 2024