AWS re:Invent 2019: [NEW LAUNCH!] Amazon Transcribe Medical: Transforming Healthcare w/ AI (AIM210)

hello everybody welcome to our session transforming healthcare with AI my name is Paul L I'm the product manager for Amazon transcribe I'm extremely excited today to have you join us welcome we have a very exciting announcement you may have already heard we announced a new service called Amazon transcribed medical in this particular session I'd like to really walk through what it means to use AI to transform healthcare really take a look at some of the problems that are being faced today by customers by end-users by builders and with me today I have two wonderful guests whom you will hear from Jacob gears from Cerner as well as George Siegen from Amgen I'd like to start off by talking about the challenges of analyzing voice data in healthcare today there's over 1.5 billion hours of medical data audio data that's generated every year and there's these key insights and valuable pieces of information that's locked inside trapped hidden in there because we don't have a great convenient way to really extract that information about 90 plus percent of that information is actually generated by physicians alone so you can imagine just how much audio data medical audio data we have and how much of that we're not able to gain deep insights into and of course there's even more billions of hours of medical audio backlog that's floating around in all of our databases that we can't actually tap into to extract information useful insights from medical audio you're going to need to do two things you're going to take that audio and you're going to transcribe it into text right that's step number one to do that we use automatic speech recognition ASR for short your second step is going to be using natural language processing to run text analytics against that text that's been output by the ASR up stream what I want to do is quickly dive into that first leg and what Amazon transcribed medical can do and also give you a quick overview of what the service is but before I talk about that I want to get into some of the limitations of existing solutions so a lot of you would be familiar with 3p manual transcription services right the limitations here are the fact that there's turnaround time associated is is and can be quite expensive human scribes you may have walked into you know a session with your physician into the hospital in which you weren't just talking with your physician but actually there was a third person there he or she was a human scribe whose job is actually to help physicians or clinicians take notes to capture the conversation that's happening and the limitation here is that it can be awkward for patients right you're talking about intimate things about your body about your health and yet there's another person there who might be listening in and although they are silent it can be awkward and can be a distraction also for clinicians who are trying to deliver better patient it become better patient care for providers human scribes are equal to scale imagine managing teams of teams of people who are not always full-time staff at their provider facilities you're probably also aware of front-end cetacean software which is available and commonly used for what we call post encounter dictation sessions that means the physician or clinician will spend extra time after each patient and counselor and engagement explicitly note-taking and so what that means is these physicians and clinicians have to spent this extra effort and there's this burden associated not just with actually delivering good care for patients but actually doing clerical work and that really takes away from their experience and introduces things like burnout moreover a lot of these piece of software that's available today have lock-ins offer in flexible pricing and not very very transparence tracking against actual transcription usage for instance if finally a lot of these types of software require clinicians to work for the software instead of the other way around for instance you might have to use unnatural speech explicitly noting things like commas in punctuation or a period here and period there and that can give you a very very distracting so I'd like to introduce Amazon transcribe medical because there is a better way and we've built it Amazon transcribe Nicole is an automatic speech recognition service that gives developers a really really easy way to add medical speech to text to their applications and there are four major use cases I'd like to talk through very very quickly the first is in clinical documentation I've already alluded to this several times already the notion that a clinician is spending all these burdensome hours and minutes doing documentation instead of actually focusing on the patients so using Amazon transcribe medical you can actually capture those conversations or dictation on the fly reducing the burden on physicians the second one is struck safety or pharmacovigilance in this example or in this use case what you'll find is that there's meaningful insights hidden in contact center calls from either clinicians or patients who want to report potential side-effects or safety events the next use case is a growing one in telemedicine where we're able to use Amazon transcribe medical to generate subtitles for sessions that are happening remotely between a physician and a patient and the fourth is a broader broader set at the contact center level for payers and providers who actually are generating tons and tons of this medical audio data these calls that are being recorded some of the features and benefits of amateur on Amazon transcribe medical are that it's highly accurate it's very very easy to use and we've made it so that it is very affordable so let's blow up each one of these in terms of accuracy today we cover US English and we cover transcription for primary care which includes the following four specialties in OBGYN family methan internal medicine as well as pediatrics we're able to not only transcribe dictated notes we're also able to transcribe conversations between a clinician and a patient for instance transcribe medical is very very easy to use it can be easily integrated with any voice enabled application its device agnostic is a real-time API meaning that you can open a secure web socket connection over which you send us an audio stream and for which you will get in return a text stream in JSON blobs as I mentioned earlier we feature natural speech which means things like punctuation and capitalization are automatically taken care of no more awkward explicit dictation of things like commas and periods semicolons of that nature in the transcripts we also offer word level timestamps this is going to be key if you want to do things like traceability if you want to generate subtitles for that telemedicine encounter for instance and be able to map that against the audio excuse me against the video recording and then also within the transcript we offer word level confidence scores with that you get a glimpse into how confident we feel well we've done our job in transcribing every single word transcribe Medical is very affordable it offers a pay-as-you-go model which means there's no lock ins there's no contracts there's no terms you have the flexibility to pay as you go more importantly we charge by transcription usage so if you want a better transparency into whether this this offering gives you advantages and gives you a value at that year-end you can actually map that back to the usage back to your charge we're offering it today at seven point five cents a minute and everybody who registers will be able to get 60 minutes free tier per month in the first year of registration so very briefly I want to talk about how Amazon transcribe medical works like I said it's device agnostic you simply call the API you start passing your audio stream and then in return you get a stream of text it's basically that simple and Amazon transcribe medical it's HIPAA eligible starting today so we've covered that first leg of that trip taking audio and transcribing it into text and then we have to do something with that text at Amazon we have a service called Amazon comprehend Medical which is our in-house medical NLP service last year the service was launched for the first time and so I'll just briefly revisit what Amazon excuse me Amazon comprehend medical covers for those of you who are not familiar it's a HIPAA eligible service that uses machine learning to extract medical information with very very high accuracy to help reduce both the costs and the time and the effort of processing very very large amounts of information and unstructured text data Amazon comprehend Medical is able to offer you an API so that you can determine key entities and within those entities determine the relationships we also offer the ability to detect protected health information so now what I'd like to do is talk about how Amazon transcribed medical and comprehend medical can be used in conjunction with one another whether it's a conversation between clinician and patients or a medical dictation you pump it through Amazon transcribed medical you get the text in return and then you take that text and you pump it into comprehend medical to run text analysis with your output you're welcome to for instance pump that into any EHR system electronic health record system and for a moment I'd like to show you a quick demo stringing both services together this 55 year old man with known coronary artery disease comes for a follow-up visit today last month he was admitted to our hospital with unstable angina he underwent heart catheterization on November 15th 2007 at that time he was found to have a tight 99% proximal stenosis total occlusion and collateralization of the mid circumflex right coronary artery was normal ventricular graffia was normal and his ejection fraction was 65% he underwent an uncomplicated placement of a ciphered drug-eluting stent to his proximal lesion the attempted coronary intervention of the circumflex was unsuccessful as his lesion cannot be crossed his post procedure was uncomplicated and he was discharged on the day following his intervention he comes today indicating that he is feeling great his current medications include aspirin 325 milligrams daily lipitor 40 milligrams daily and plavix 75 milligrams daily so what you just saw was an actual real-time transcription of of an audio file that I had recorded in advance it's you can see from the partial transcripts finally mapping into the final transcripts that the clinical thinking is happening on the fly as each new word is being introduced into the sentence that's being spoken context is being introduced and we're able to think on the fly we take this text that we've generated using Amazon transcribe medical and then we pump it like I said into Amazon comprehend medical with that we can get insight into key terms such as pH I information we can highlight medical conditions we can discover Anatomy and so forth we can also drill into any of these terms maybe we're interested in looking at what the ejection fraction is here you can see on a right panel in this quick quick demo we've been able to demonstrate the entities and also the relationship to the term 65 and then showcase the unit of measure which is in percentage if you'd like you can even actually show a much more comprehensive view of the entire list a summary of entities under relationships again this is a really really bare bone a quick way to show you how we just put two surfaces together for all the builders out here this will be easy-peasy for you alright so again what you just saw was transcribed medical working in conjunction with comprehend medical but the thing is you can mix and match you don't have to use comprehend medical if you didn't want to you can use it with your own NLP medical NLP of your own choice and with that I'd love to welcome JP gears from Cerner to talk about his use case and also how they're transforming clinician and patient conversations using Amazon transcribe medical thanks Paul appreciate it hello everyone welcome thank you for coming spending a few minutes with me this afternoon I want to start out with a story and this is actually a true story so two weeks ago I had to take my 15 month old daughter to the doctor she had an ear infection a couple of weeks prior to that and it seemed like her conditions were worsening I had to take along my three year old my six year old was in school so I had my hands full and I'm in with the doctor and in with the MA and I'm probably the 20th patient they've seen that day and they probably have 20 more patients to see after me and they examined my daughter she's screaming because she doesn't like things being stuck in her ear and the doctors asking me questions the doctors busy in front of her computer typing in data and looking up data and it's just a very kind of stressful environment so the visit happens we get home and my wife says starts peppering me with questions what happened is our both ears infected still or is it just one ear is does she need a new medication or should we finish out the old antibiotic does she need do we need to get referred to an ENT doc because she's had several ear infections this month so I was not able to answer all of those questions and research backs me up on that that we only retain 20 or 30% of something when we're talking to a professional in which were not skilled in that particular area like something like medicine so that is where we are building at cerner a solution called virtual scribe helping clinicians get through their day and a much more efficient and easy manner and it also helps patients imagine if I were able to take a summary or a transcript of that visit to be able to answer those questions or have my wife review that in more detail so we are Cerner corporation we are a leading global EMR provider we also have population health platform we are 29,000 associates strong we're live in more than 35 countries so we're a huge huge player in the health IT space we chose to expand upon our relationship with AWS we've been working with them since 2009 we signed a major deal with them earlier this year to help us really get to that next level in healthcare so our clients have been putting data in EHRs for 10 20 30 years and they've not really been able to glean the power of the promise of an EHR which is to gain insights from all of that data that we have in the EMR most of which is unstructured so AWS is going to help us with a variety of ways but they'll really help us unlock all of that data and allow us to embed that into workflows to help clinicians so our vision is really one where we transform a world today where data in documentation in is both tedious and time-consuming and then that decision-making is also very manual to get data out of a system you have to to report or to get data out you have to hunt and pack and go look for actual data we want to transition into a world where data in is heavily automated from voice device data and consumer entered information and we also want to have decision-making be guided by AI and virtual assistants that are continuously scanning through that record and providing real-time insights all that being said as we focus on reducing cognitive load we have some very important guiding principles one of which is that we are not a black box we are showing all of our work by including context as we demo virtual scribe we will show you right alongside the proposed clinical terms where in the conversation that came from so that clinician as they're seeing 35 or 40 patients per day they don't have to remember did the patients say they had chest pain or did they say they did not have chest pain was it a stabbing pain or a shooting pain virtual scribe is capturing all of that keeping a scratch pad of that visit we also make sure that we leave the clinical the medical decision making to the clinician in order for clinicians to trust these new technologies these machine learning models these artificial intelligence pieces they need to be able to trust that we are showing them the value and showing them the work behind it so we need to show before we recommend something why we came up to that recommendation and then finally we're gonna deeply embed that in the workflow we're not gonna make them change screens we're not gonna make them utilize a bunch of Mouse miles and add a bunch of clicks to their workflow we're gonna deeply embed it and it's gonna be right there front and center for them to go on with their day so I'm just gonna really quickly explore some of the challenges of clinical documentation there's a lot of press out there around clinician burnout and physician burnout the studies show that clinicians spent almost double the amount of time in the EHR documenting on their patients than they do actually seeing the patients we want to flip that on its head physicians and clinicians got into medicine because they like caring for patients it's in their DNA so we feel that we should be able to leverage artificial intelligence machine learning natural language processing automatic speech recognition to enable them to actually provide care for the patient so today a lot of time clinicians are not going and entering family history and social history smoking status into those particular sections they're picking up that awkward microphone that Paul talked about and they're saying patient is a nonsmoker or patient smokes three packs per day and then that data is buried inside a note with leveraging virtue with leveraging Cerner's virtual scribe powered by AWS transcribed medical and using Cerner's in LP we can actually codify that and put that into the right spot in the record driving a whole bunch of benefits downstream clinicians want to care for patients and clinicians also want to work at the top of their licensure a doctor does not want to be in front of a computer a doctor wants to be caring for patients and finally providers want their notes created as a byproduct of the visit they do not want to do pajama time which is documenting at the end of their day taking care of all of the notes and all of the regulatory things that they need to go and have done they they want to leave at five o'clock like we all do so they can go watch their their kids play soccer or spend time with their wife or do something with it just for fun so that brings us into leveraging voice so we've had keyboard and mouse we've had touchscreens very successful user interface on our mobile phones and tablets but really I think for healthcare specifically voice is extremely powerful health care is often a very sterile environment where clinicians are using their hands constantly so voice is very natural so across the care continuum whether that be in the o.r suite whether it's in a hospital room whether it's in a doctor's office or whether it's the patient at home there are numerous examples where voice lends itself really well I've mentioned this theme several times about reducing the cognitive load on healthcare teams one of the ways that we're diving diving deep on today is is around voice but you can also use images videos and sensors as different inputs and then you apply the AI transformations to that so I'll use an example if I was talking with my doctor and I said that I had blood in my urine we would take that and we would obviously get the the text of that we would get that that blob of text from Paul's transcribed medical service and then we would apply natural language processing on that and also natural language understanding because we need that context and then we apply ontology mappings and terminologies to that so the blood in the urine is there's a sno-med code for that so that's important blood and urine is also known as hematuria and there's an icd-10 code for that which is needed for providers to do their billing so we take all of that we apply the semantic interoperability to that and then we give that to the clinician in the form of an application deeply embedded in their workflow and I'll show you exactly what that is when we do our demo so with virtual scribe the clinician simply needs to conduct the patient visit and virtual scribe does the rest we give you a conversational transcription generation in real time we codify the conversation using NLP and NLU and we support 17 different healthcare concepts we allow the clinician to directly add things to the HR so things like allergies medications and problems are things we support directly today and discreetly today and then we also allow them to incorporate the transcript directly into their clinical documentation so those allergies that come back with one click they can add those to the record they're codified in that allergy profile so that any clinician and anytime downstream that that needs to be referenced those are then available medications can be added as a brand new prescription to go out to Walgreens or CVS they can also be added as a historical med a lot of times a patient will say yeah I'm taking a multivitamin that needs to be documented that needs to be added to the medication list and that's just a whole med so you can add that as well directly from here and having it kept by the conversation the provider doesn't have to go and search for that they can focus solely on the patient they no longer have to do that context switching that saves them time per patient it results in more complete documentation it reduces clinician burnout and it definitely increases patient and clinician satisfaction I know that I would definitely feel more comfortable talking to my clinic my provider if they're able to look at me and focus on me the entire time instead of transitioning back and forth to and from their computer finding the problem list clicking in the add button starting to type the name of the problem choosing the problem clicking add all of that is eliminated through the use of voice technology so where is this headed ultimately what our providers are telling us is this is fantastic I also want it to generate a note for me so at the end of my visit I want you to propose a note to me and me to say yep that looks good or do a couple of tweaks to it and go on with my next patient so that I can get out of there at the end of my day we can also do all kinds of things like automated patient summaries so those questions that my wife was asking me I would be able to show her a summarization of that or show her the entire transcript of that additionally there's a lot of documentation that that nurses do and amazed do that voice can be leveraged for that we can tie the specific value to the specific form and we can complete form charting as well through voice so how does this work Paul talked about this a little bit I'm gonna kind of share how we're doing it so we clip a microphone on to the laptop that the clinicians use when they're seeing the patients they log in there's a microphone on the toolbar the patient and the provider have their conversation we have our SDK that's sitting locally with Citrix that passes through to the cloud where we have our concept service and our speech service running and those pass those blobs those JSON blobs and those api is back between amazon transcribe medical and then Cerner's you know P and NLU engine so I'm going to play a quick video to illustrate virtual scribe in action on the left side you will see the interaction that occurred between a patient and provider and on the right side you will see the suggestions that get teed up for the provider that scratchpad if you will that they are able to then take action on what brings you in today well you know I've been kind of feeling a little down lately my asthma has been acting up and my albuterol is not really working I just feel kind of depressed let's see what we can do for you so I saw you were already taking wellbutrin 150 milligrams daily what happened with that medication I was getting really dizzy and I started getting really bad dry mouth so I stopped taking it how is your sleep not good I was getting eight hours and now I'm looking to get - well why don't we do some changing on your medications do you have allergies to medicines I should know about I'm allergic to shellfish but I'm pretty careful about it okay let's switch your medications I want you to stop your well featurin start prozac 20 milligrams in the morning and then a bed time for sleep take seroquel 25 milligrams remember that is a sedative so be careful with that and then I'll see you back in about a month and we'll go over your progress if things aren't better we can always incorporate therapy at that time okay so how's that sound that sounds great thanks doctor dude oh great so what you saw returned there on the screen was not only the proposed actions but also a transcript so the you user interface is very very simple we have the medications the problems the allergies section doubt with one click we allow those additions right into the workflow if you don't agree with those or those weren't exactly what you thought or exactly what you wanted or they weren't relevant you can simply ignore them and then we show you the entire transcript that you could highlight utilize take edit and incorporate into your clinical documentation as ice mentioned before one of our guiding principles is that we're going to deeply embedded in into the workflow so on the right side you have the elements of your notes section on the left side you have your chart data and right in the middle sits virtual scribe allowing you to take information add it directly to the chart or take that information drag it drop it into your documentation sections very very easy very very user-friendly again access to both that transcript as well as those proposed actions that are being returned so the tests that we've done show that there is this will result in a significant reduction in the amount of time spent documenting the allergies the medications and the problems it allows your staff to shift to focus on patient care preventative items as well as all of the regulatory burdens that are being asked of clinicians ultimately leading to increased provider satisfaction and increase patient satisfaction reducing that clinician burnout so virtual scribe listens so that I can focus on the patient that's the ultimate benefit that's why I got a new medicine so that I can focus and care for patients so where are we in this journey we are testing this with a couple of our clients we have many more signed up to test with us we're on the journey to augment a note creation so we're collaborating closely with the AWS transcribed medical team to solve for all of these things so speech to text accuracy increasing that making sure that all the medical terms across all of the different specialties are accounted for understanding is it the patient that just spoke that or is it the clinician that just spoke that that's really important for us to return those proposed actions that I showed you in the video and then making sense of the clinical context so if I say to the doctor that my grandma had congestive heart failure today we will return congestive heart failure as a proposed problem it's not really a problem it's a piece of family history so that's where context is extremely important and then we need to take all of that and we need to form it into sentences that a clinician would put into a readable note so it's it's definitely a journey we're working very closely with AWS to make this reality so again just a little bit of a wrap up more and more is being asked of clinicians each and every day documentation is very tedious and time-consuming so we created virtual scribe with Cerner AI that passively listens to healthcare visits captures discrete data allowing clinicians to focus on the patient and not entering tedious data into the EHR Cerner is relentlessly focused on seeking breakthrough innovation that will shape the healthcare of tomorrow because healthcare is absolutely too important to stay the same so with that I will wrap Thank You Jake please um I'll allow me to introduce Jorge Siegen from Amgen hello everybody let me tell you a little bit about Amgen for those of you who don't know who we are we're one of the world's largest biotechnology companies we have a global footprint for research and development a global footprint for biotechnology manufacturing we have we're serving millions of patients in over a hundred countries so I didn't have that up on the screen I guess we're fully integrated biotechnology company and with world-class capabilities in a number of areas due to our history in in biotechnology we have deep expertise in protein engineering and biologics manufacturing we also have a history that gives us a biology first advantage in that we have a number of different biological modalities that we can apply to to look at any given possible therapeutic target and develop the appropriate therapies to address that target in addition our decode genetics facility in Iceland is one of the premier research facilities for human genetics in the world we take much of the information that comes out of that operation and use it to validate the the therapeutic target in the proposed therapies that that we're looking at at Amgen we really focus on Grievous illnesses and trying to find innovative solutions of biotechnology therapies to address those illnesses some of the you know the major therapeutic areas that we are that we focus on is cardiovascular disease bone health inflammation and oncology here are a few of our products you may have heard of heard of some of them things like Umbro for rheumatoid arthritis or Pathak for cardiovascular disease and Aimie big for migraine headaches and what I want to talk to you about today is some work we've been doing using machine learning to help us identify safety events as part of our farm pharmacovigilance programs about a year ago we started a series of experiments to see how well we could do with the latest NLP technologies to develop machine learning tools to identify safety events in the interactions we have with healthcare providers and and with patients and in in addition to that we currently have a large effort or a large staff of trained healthcare professionals that actually interact with patients and healthcare providers and calls in and actually do human curated transcription of the conversations that are going on that we can base our pharmacovigilance efforts upon we also wanted to start looking at using machine translation in this area and seeing how far we could push the technology there as well we started out capturing a large amount of human curated transcription information from our historical systems or out of our existing systems and developed a series of NLP models a whole not one or two but 10 NLP models to help see how well we can do identifying these safety events they employed both traditional linguistic approaches and and also deep learning approaches and we're very encouraged that the top performing models actually performed quite well this is an architectural diagram of the the system we used at Amgen at Amazon all of the model training and all of the model inference was actually done on the Amazon sage maker platform all of our raw data and model artifacts were stored in s3 buckets and here's a little bit of the a few results from what we saw or what we obtained from the different models and it doesn't really show on here but we also used Amazon comprehend in inside a number of the models as a feature generator but the best performing models were we're models that depended on word vector embeddings and different types of deep neural network classification layers and with a couple of the models we were able to achieve overall accuracies precision recall and AEC's up at the 98% level which which was quite gratifying a quite you know quite useful at the same time we were using we initiated a set of studies to use Amazon transcribe to do machine transcription of the audio events themselves in compared to the transcriptions that were that we were also working with - nobody's surprised Amazon transcribe out of the box was not able to to recognize a lot of the specialized medical vocabulary that was contained in our data sets to get around that we actually started building custom medical vocabularies of our own that we could use to supplement the Amazon transcribe vocabularies a number of the well constructed these vocabularies by going to a number of the medical coding systems that are out there that are publicly available things like icd-9 icd-10 CPT pick picks the national drug coding system and even the sno-med ontology and we took the descriptions out of all those coding systems and created a massive medical vocabulary and - as we expected the performance with Amazon transcribed when we added that supplemental vocabulary improved tremendously but there's still room for improvement at that point we got connected with Paul and the medical transcribe team and since that point we've been using medical transcribe and seeing further significant improvements in performance for our our medical transcription of our of our HCP and patient interaction data so where we at currently well we've completed a lot of work with our human curated transcriptions and built a number of models that that perform actually excellently in that area we're right now repeating all of that work with Amazon transcribe medical and actually building some even more advanced models that are transformer based models that have recently become arced that their architectures have recently become available and we're comparing the performance on all three systems and hopefully we'll be very quickly at a point where we can start putting some of these tools into usage and why is this important what are the benefits out of this we are the current manual approach to that that we use it is really difficult to scale and any tool that we can produce here doesn't have to be comprehensive and cover everything if we can simply accurately identify certain amounts of certain items that don't need to be looked at that would be a major win for us in addition many of the safety events that we're looking for are rare and we're looking for them in places like the medical and scientific literature whose volume is increasing tremendously what we're hoping is that some of these automated tools will help us find those needles in those haystacks that are just increasing in size daily and finally there's a huge interest in using some of the new voice enabled technologies that that are becoming available that we're human and the loop just isn't going to work or it would be very difficult or probably won't even work and speaking about things like chatbots we're hoping the the automation tools that we're developing for identifying these events will be very useful in that situation as well thank you very much thank you George so with with Jake you heard how Cerner is using virtual scribe to really transform the way that physician patient conversations are being captured they're going to be analyzed downstream are going to be used either to enrich a patient's understanding of history on healthcare or to really save time and lift a burden off the shoulders of clinicians themselves with George you heard how Amgen is really innovating new ways to tackle the use case for pharmacovigilance not only in identifying safety events but actually identifying things that are not relevant to spend time on right the other side of the coin so I wanted to thank both our guests very much today and thank everybody here for coming Amazon transcribed medical is available across six major AWS regions if you have any other questions the three of us will be available after the session to take some questions and thank you again here are some related sessions that you're welcome to attend we also have another chart we'll keep both on screen to share with you additional health care sessions as well thank you very much [Applause]

Channel: AWS Events

Views: 2,394

Rating: 4.8095236 out of 5

Keywords: re:Invent 2019, Amazon, AWS re:Invent, AIM210, Artificial Intelligence, Machine Learning, 1977

Id: W_fDlbFlf14

Length: 42min 46sec (2566 seconds)

Published: Fri Dec 13 2019