Fighting dyslexia with computer science | Markus Gross | TEDxZurich

I'm a computer scientist and in my daily life I'm developing technologies for the film and entertainment industry but as a matter of fact today I'm not gonna talk about special effects and animation for a change I'm going to talk about me in my role as a father and I really want to start my talk with a little story to tell you this is my son Adrian as a matter of fact this is Adrian back 13 years ago when I started this project it's a funny little boy a vivid and smart he'd talked a little bit too much all the time but other than that was all good except he developed one severe handicap when he joined elementary school he soon developed a significant deficit in reading and writing comprehension and he left behind his classmates very quickly and this went up to the point basically where writing and reading became a true nightmare to him it took years to diagnose him with a condition called dyslexia and as a matter of fact as caretaking parents my wife and I we spent a lot of time and effort and finally money to work on therapies with him and they were all useless they didn't help and this frustrated us a lot now observing him over the years I discovered a bunch of distinct perceptual and cognitive abilities and parents within him for instance his strength in memorizing structural and spatial information now it was actually really his suffering in school that motivated me to embark on a mission and to develop a therapy of my own with weapons of computer science and this is what I'm gonna introduce to you right now now dyslexia is the distinct disability of some people to read fluently and write orthographically correct over time you know and despite of training efforts there is no improvement and this happens in people with average intelligence or even with distinct above average IQs Adrienne shares this condition with five to ten percent of the population which is affected by this and dyslexia appears in various forms and shapes and different levels of severity now the symptoms of dyslexia are actually pretty well understood and they include slow reading and limited writing but oftentimes also letters are put into the wrong order good example is felt versus left oftentimes also dyslexics confused phonetically or crapha Klee similar letters such as B and P D and T G and K I called up my son last night and asked him what is your favorite word as a dyslexic and he said it's Fogell the German word for bird and he would always write it with an F instead of a V or Hagen for driving where there's a silent consonant and H which is not really pronounced and he always emitted so dyslexia also comes along oftentimes with a more general attention deficit disorder but the effect and the devastation of dyslexia is really really significant because it extends to all other disciplines in school or in professional life and it oftentimes leads to under performance and demotivation which sort of can never be caught up again now broadly speaking dyslexia is caused by an inability of dyslexic sprain to build up a map between so-called phonemes and craftiness phonemes are the elementary building blocks of spoken language they are the basic sounds like owl or Chi or I and Kraft Eames are groups of letters such as syllables of fractions of a syllable which build the basic building blocks of written language now neuroscientists have also discovered that you know if you start reading as an untrained reader most of the brain's information processing is happening in the frontal part of the brain there's frontal activity now as we acquire the skill of reading there is a shift happening in information processing and most of the processing happens in the parietal area of the brain and this so-called frontal parietal shift is not being accomplished probably within dyslexics now I teamed up with neuroscientists from the University of Zurich and was a bunch of brilliant PhD students and computer science students from Italy ha one Christian is actually sitting in the audience here and we combined state-of-the-art knowledge from neuroscience and from computer science to develop a new so-called multimodal learning therapy for dyslexia which adapts to the individual capabilities of a child so let me explain this in a moment now multimodal information processing relates to the fact that the human frame is able to process information simultaneously through different perceptual cues or channels and such channels can include color information structural topological information lighting sounds speech shape textures patterns motion and so forth and so on now we all develop over time discrete profile of how our brain develops these individual channels and I for my part might prefer color and shape and you might prefer sounds or motion in combination now the core idea of the learning approach is to reel out the information that is embedded in a word in this example might szerik through different perceptual pathways and this is accomplished by transcoding the word but he had presenting the word into a variety of different codes one is a topological structural code another one is a color code there is also a musical code synth by the system and a shape code this all leads to a multi-modal that presentation addressing as many as possible a perceptual pathways into the brain at the same time and in order to control the information that is associated with the strength coding we use a concept that is well known from information theory and statistics it's called the entropy it measures the disorder if you will now this transcoding is combined with a powerful concept of so-called machine learning and the machine learning component basically develops of a so-called student model a mathematical representation of the child's actual learning capabilities and preferences and strengths and weaknesses and it refines it over time as the child works with a system and this allows us to apply the learning in a personalized and individualized way to the child so it takes the next word in such a way that it's most effective to the child's disability and reduces the error in the best possible mathematical way if you will here is how it looks like the large picture shows actually this multi model had presentation now you see the structural code which is the so-called syllable tree of the word might suruc or of this group of words then you also can see that each letter is associated with a distinct color some letters for instance the capital atom has a different shape than the small cap letters and loads here for Jeremy language were for French would be represented with yet another shape also there is a musical code associated with it the musical code is if each color a distinct note on a pentatonic scale but the pentatonic scale if you look at the piano and you look at all the plaque keys that the pentatonic scale now try it out if you push these keys in a random order and at random speed it will always sound nicely so that's the idea now also what we associate is different instruments to different shapes to better distinct and to dress the multi-modality and finally what we do is length of the note in the composition is basically related to the lengths for syllable to compute the color code which is the mapping of these eight distinct colors to individual groups of letters it's specific for each language and it follows a very complex mathematical optimization procedure but the training is super simple and here's an illustration so you see that the child first of all placed this little memory game and repeats the colors and gets a feedback with the sound and then in a second step you acquaint yourself with a structured topological code by basically redrawing the syllable craft note also the colors and usable feedback and then you train and it shows this might model the presentation its three-dimensional graphics its animated it's cool you can touch it and you can interact with it and you just repeat the word so if you make a mistake then you get an immediate visual feedback like here and you know the system encourages you to retype it and to correct it that's all it does and it represents the words the sequence in which you learn in the best possible mathematical way now the underlying machine intelligence for each in correctly spelled word computes a hypothesis on the nature of the error so for instance here's an example in German the word mood would be misspelled instead of an M in the middle the child types an M and there might be three different sources possible forever or one is a simpler typo on the keyboard because N and M are next to each other a second possible source of error could be an auditory confusion because N and M sound similarly you know and the third one could be the most severe one it's a problem in the crafting phoneme map built up in your praying and this our hypothesis and our categorization of the underlying machine intelligence that's essential to adjust to the child and to build up this mathematical model of the child's learning now we carried out a variety of user studies here in Switzerland and in Germany mostly with children aged 8 to 12 over the years and these proofed the efficiency and the effectiveness of the methods so after three months of training four days a week 15 to 20 minutes we found an improvement in average improvement of the performance in writing in paper and pencil tests which means dictation of more than 30% and what is really remarkable is that the system and the approach generalizes to non trained words that is if you present the child an unknown word which was not in the training database then the average improvement is still over 25% and what was also interested after you stopped the learning most of the effect is remanent and still available after 3 to 6 months this compares to the control group of dyslexics without training which only displayed mild improvement of 6% due to the ordinary school training now what we also found we carried out a study with adults and it works equally well with adults and finally the system is also very effective for non dyslexic children and it works as a vocabulary trainer and in different languages now we use it in a large number of Swiss schools at present primarily in the german-speaking part of switzerland and over the years we trained more than 60,000 children successfully and got a lot of positive feedback from the parents because I know as a parent it is really really emotionally a tearing and requires a lot of effort on the parents side what's also remarkable is ever since we started five years ago we tabulated and stored each and every keystroke of each and every child which ever trained with the system along with a timestamp and this gives us a true treasure it's a huge data repository into which we could dig in with other mathematical methods or big data analytics to discover distinct patterns and effects and phenomena for instance one we discover is that there are groups of children who seem to have similar features that is they seem to have similar difficulties or similar behaviors in learning and this is very valuable to improve the mathematical model underlying the system also we hope that we will be able through the data analytics we are doing to develop predictive power of the system so that the system can diagnose or can indicate that a child probably has dyslexia just by letting it work with the system for some time now motivated by the success in dyslexia we a couple of years ago I ventured into a related condition which is called disc Akua it's less well known but it relates to the disability of children to develop mathematical skills and it's equally devastating and there is an equal number a percentage of the population 57 percent typically affected by it and this is the equivalent for a disc Akula now with all this work over all these years motivated actually by fathers desire to help his son initially we believe that we developed a very simple and more importantly a very effective way of training and attacking dyslexia with the powers of computer science information theory and machine learning and we hope that we can generalize or extend some of these concepts to other fields of learning as well now back to Adrian he is now 25 years old and a student of s computer signs yes you got it thank you very much

Channel: TEDx Talks

Views: 47,727

Keywords: TEDxTalks, English, Switzerland, Technology, Cognitive science, Computers, Health, Language, Science, Speech

Id: 2LK9bC3NUKE

Length: 16min 1sec (961 seconds)

Published: Tue Jan 05 2016