Ep#30: What is an EVS (Event-based vision Sensor) and what are the Applications in industry?

hi so in this episode I would like to talk about uh EVS or even base cameras they are um very interesting um cameras I would say it's a very interesting technology to which is a little bit or completely different from um the normal um camera systems which is which are basically B uh frame based cameras and they have pixels uh unlike them uh EVS or uh Naro nuro foric formic I guess uh it's called or event uh or in marketing EVS uh cameras they are basically um yeah they basically work based on um illuminance uh and dress hols and uh basically um the way that they function uh is dependent uh on the dressold that is defined um for for The Regulators to accept or reject uh the signal so we will go through this um website which belongs to um Sony Sony has recently um launched a product uh which is a EVS cameras and EVS uh sensors uh EVS stands for basically uh uh event based Vision sensors and basically realizes high speed data output with low latency by limiting the output data to luminance changes from each pixel combined with information on coordinate and time with a focus on movement they can be applied in a wide variety of application or Fields so basically there there is plenty or enormous uh application for EVS or event based cameras um they are actually very similar to the way that our eye and or Vision system actually works in here for example you can see um that our human ey has a road cell which is basically con verting uh somehow or gets the light inside and then we have bipolar cells uh for the second stage and then we have retinal gangon cell I hope that I pronounce it uh correctly which is a kind of medium to the to the to the um cortex uh in the brain or visual cortex in the brain um to transfer uh the the signal so exactly the way that our uh eye or human eye is functions EVS also is like that so there is a light receiving unit which basically convert uh energy or Photon energy to voltage basically or current to voltage and then it converts uh in the second stage we have a amplification unit uh and the CT is a compare comparator which basically you will Define a dressold and then if that amount of um let's say U signal uh amplitude reach to that level um over that level it will be a plus event and below that level it it will be a minus uh event and uh in this way you do not basically capture anything everything only those um asynchronous um points or cells that has received amount of value or amount of energy as an input that um basically goes beyond the def def defined value of the dressold will give a positive value so we have either one or zero if it is more than dress ho hold then it will be positive plus and will represent will be represented in the system uh as a plus signal and if it is below that dress hold then we have a minus and that's nothing so nothing appears in the in the in the picture or in the image so this is the this is the general uh format or uh structure of this uh sensor SCE and then um the mechanism is uh as you see in this uh I hope that you can see yeah um so the mechanism as this is uh see imagine that there is a ball U moving front in front of you uh what is this capture is nothing just only ball so any background that the um the sensor is focused and is static has no uh value for the for the sensor only that object for example here as a ball in each pixel will be represented as a DOT and the position of this ball is clear that it changes over the time and it moves from one position XY position to the next position over the time so we have a continuous traction or track or detection of the ball movement over the time without any stop and as I said the beauty of these sensors is that they just only focus on the um motion or movable objects so um here there is a better explanation a bit uh um so yeah so the pixel units B basically attracts or absorb the light and then it will be converted to voltage and the next step we have a Delta modulation which in Delta modulation you you will compare as a reference that you have defined the value uh the reference voltage and the output voltage from the pixel unit so you have to detect or compare this Delta and if this Delta is more or exceeds the uh defined uh dressold then an event plus will be generated if not it will be minus or nothing so the idea or the concept is BAS rly easy but the applications are quite interesting and the next one there is also very interesting um animation as you see you have a high dress hold a lower dressold and a reference voltage so as as as much the the the dress hold for the illumination of this pixel is below the reference the even signal is negative so you will see it here um as a blue uh arrow keys down toward down and when it goes up and reach to the level of reference voltage then the positive signal of uh event will be created it is very fascinating actually so you can actually Define as well the dress hold it depends on your application if you are in a very very maybe uh noisy um environment or there is a huge of exposure might be the reason to change the reference voltage but generally it depends on that um and the next uh another as I said one of one of the benefits of EVS over frame based uh cameras is that frame based camera in exposure light they get blues and it's quite hard to detect any object uh for these cameras um but for for EVS or event based cameras it's not like that as you see it is really it doesn't matter how how much the luminance changes always the output voltage or Delta V is the same and that's a great benefit to detect in any circum any condition any light condition to detect the value of the object and uh that's a great benefit for or is very robust uh I would say without any postprocessing you can get these values and it's very versy so the next um there are some there's a this picture that you see at the left and right so frame base image sensor you can see such a thing uh at the left and EVS sensor you can get uh basically if I if I want to be honest with you is very similar to um yeah the meod of uh Edge detecting by a normal webcam or camera so but the only uh difference is that as as I said objects should be moving in EVS to be detectable otherwise I mean like as I said for frame based image sensors to get such a thing you need to put a lot of high computational power to um to apply methods like laian for example example to get the The Edge detect to to detect the edges but for EVS it's already there without any postprocessing Sony claims that they have um special um Copper to Copper technology to make even the chips uh smaller so basically uh what I have seen is that um in conventional um basically a way of uh chips production of sensor and basically um so it's unlike conventional technology that has the light receiving circuit and luminance detection circuit and the same layer this technology incorporates them on different layers a pixel chip as a upper layer and a logic chip as a lower level um which includes integrated signal processing circuits so they can basically uh reduce the size of the chips with a paral paralyzation of the layers and that has a great benefits for bigger production of the chips with higher um coverage of the scene U basically this uh or field so these chips are stacked and connected using Copper to Copper technology within each pixel as I said at the moment I think that the IND indust smallest pixels is about 4.86 micrometer is integrated with the logic CH chips on a 40 nanometer process resulting in a 1 over 2.5 type sensor with HD resolution of 1,280 in 720 all right so from it it seems that we have much much more compa uh compact design and because of this compact design and because of this um the the the the Instinct the the the base of this technology uh there is a high speed of um computation and also lower latency so because each pixel detects luminance changes as uncurly and it will output event data immediately so there is as I said there is not really needed for any post processing this is also interesting and um buin Hardware event filter sometimes because of H the L LEDs uh flickering in the in the um uh event EVS sensors there is a need of filter so as you see in the left uh picture is taken by even camera um you will see a lot of noise I would say say in the image but if you put on the filter you will see that basically those events uh are much more clear I'm very surprised I don't know if you can see that here but this is this looks like a autoband to me and at the left side if this is the guard rails of the Autobon um I could imagine from the left side I cannot really see anything or detect any object but when I look at the at the right exactly at the same corner I kind of see or notice a car which at the left side I can't see that but at the right side it is there I see kind of like a a car but I mean like it is quite interesting now because uh you can really detect a lot of um movable or useful datas without any kind of computational power for postprocessing um I would like uh to show you an an an application video that that actually belongs to the Sony instead of the pictures so uh as an examples so let's let's review this um video together I hope that you can see that okay yeah um so um this is somehow I guess the applications uh of this I hope that I have chosen the right yeah I guess I have choosing the right video so as I said any movable object can be detected exactly like our eyes Pro um process uh an an an image the even base Vision sors also um process that um so you see that this ball that I talked about in in in in in basically a frame base or conventional frame base you have a lot of pixels uh taken but it's not a continuous it's but at the right at the real time uh or event base you have this continuous uh followup from the position and that will be created as you see uh in this photo that exactly the route or the track the balls goes will be recorded you have the position you have the time you know that what where was the ball at what specific time which in basically frame based you cannot really follow up at what frame or maybe it's uh what was where was the position so it's it's also and I also talked about this chip technology that they have integrated and make it compact everything and parallel par parallel um the the processing units parallel to the detection unit and everything integrated much much more compact from the copper copper technology that they have follow up and makes it possible to aperture ratio without any loss to increase the aperture ratio to the entire sensor and reduction of the circuit and low and create making a lower latency and um so this ball was one example and as I said the incident it doesn't matter how what is the exposure situation in the in the in the in the image um it is always the amount of output voltage doesn't matter what what is the situation it will be it will be the same basically with the camera um so that that was unfortunately the the wrong uh that was the first video I think this one was about the application uh hopefully yes um so this is as I said uh review about that what are the application of even camera um the first one as you see there is the frame base uh camera is a flow of water and at the event base is just representing each single drop of droplets of the water you can exactly detect and position that and that's beautiful and it is very super in a super slow EVS you can see uh the amount of the each droplets the next one is human tracking and the left side you have a frame based camera you see how difficult it is for the frame base camera to detect the person when it is very dark but at the right side you see that it is quite obvious even the position the way it's continuously uh it doesn't it's not really um sensible to the to the um darkness of the colors or whatever so it's it was in metal processing for example monitoring the Sparks is also another uh application or example you see at the right side all these small Sparks can be detected and followed up and the amount of or the number of these Sparks can also be summed up this also very interesting application um you see that uh there is a band or conveying layer there is one object moving up usually could be very dark there is a laser which is perpendicular on that and there is a camera installed on the left side so camera look at the um um basically um the box and the temporal information and also uh the laser line which is on the band and also on the um so you see uh on the band and also the um again I would say just yeah you see that you see that at the right side the laser there is a difference between the the the base line and and the Bosque and um you you can see that you can probably measure the height differences and it not only about the height but also visualization of a 3D model of this box that's the beauty because um you kind of get estimate of the height of the object but from this the other side the EVS can create a temporal cloud of points related to this uh box and it's not only about the height difference but also about the the shape the 3D Shape of that box as you see the right side you can get exactly the form the format of the box and that will quite help a lot to later with machine learning tools to detect uh the real object here it is also very interesting the left side there is a frame base the the platform actually is uh vibrating doesn't you didn't see anything different it's very hard but at the right side application out output you can change the frequency from low frequency to high from 30 HS to any anything so this is this is very very very interesting I would like go back again um this it is basically with uh frame base it's not really easy to detect but with the right it's you completely can see that the the the there is actually a a vibration happening and um this was really cool I like that and vibration is actually for a lot of uh maintenance in the Machinery a big factor which using such a camera can actually um trigger if the the construction something is not right or the structure is uh basically under pressure or something like that yeah so um this this was some application uh overview of these uh frame I I honestly I don't think there are too many uh um event based um cameras available in the market I've seen this one from Sony uh EVS sensors the other one was from propes or something like that uh I don't uh remember but not as I said not many um companies and models of uh EVS at the moment exist in the market but I think they will be uh get common quite well in the in the future so there is also another examples like for example [Music] um um I would say uh application in Industrial Automation especially vision for detection of Falls um and errors in construction or product can also visual inspection basically contamin contamination of detection inspection uh this is also another uh field that can be um applied if the material has been changed or if there is any damage this is also so there are small data size and comp compatibility with machine learning also notable features with the RGB camera the way scratches are perceived will different dependent on the way color appears so the data volume will become larger comparatively EVS detects differences in luminance for each pixel and only use a differential data for output after combining it with coordinate and temporal information so the data is small in size and high in output speed so the difference is here is that if you use a normal frame based RGB camera lots of information comes and you need to do postprocessing in order to detect the cracks or what whatever that is somehow making a problem here but the EVS goes directly to those differences and shows you the um cracks directly without any postprocessing uh in mashing abnormality detection also exactly the same uh so you have you can detect the abnormality by the the output data which is a small inside and non latency and detect the abnormalities in robotic of course the robots moves and of course it's much EAS iier to DET to detect the edges and when it is in motion of course that's easier for the robot to detect and um especially in logistics or transport or home delivery I don't know M drones for example uh a lot of things can be processed without with a with a very very little computational power that takes by normal frame based cameras and uh yeah so for examp example for um monitoring the um people and crowd usually with frame base you have to take a lot of data you you probably save a lot of data from the face which is sometimes not uh suitable uh because of the data privacy so if you need just the location of the people you can use EVs and just without getting much information from the picture appearance of the person um it is it can be data privacy has a bit better data PR privacy in scientific measurement of course in investigation EVS is applicable to Scientific measurement that it can detect highe speed moving object in a field of customer Behavior investigation this is actually something that I personally would like to to try it but unfor fortunately I do not know anyone who can lend me an EVS camera or it's actually it is very expensive as I said I take a look it's about 5,000 Euro for a small um EVS camera which I cannot affer it at the moment so but if I could that uh I would definitely try to measure or repeat the these experiment that are are done by Young because of this interference of um or quantal behavior of photons by Young uh experiments and use these EVS uh cameras to detect the motion um of the photons when they are crossing from these two uh slides uh to see how these photons actually interference with each other because it's very close to how our eyes can understand or perceive uh how to say analyze this uh scene so if you have a detector unlike the frame based detectors this could be somehow have an interesting result I'm not sure just guessing and the last is that the UI uxix development of course there are a lot of tools available in open source Community or software that you can actually use your guesture to have um conversation or connection with machines of course and EVS is a very nice platform to ease this process for the human to get connected with the uh op um machines so this is also another potential application for EVS all right I hope that you guys enjoyed this episode uh if I get in you if I can get in future and an event based camera uh I will definitely make an episode about that and but I hope that you enjoyed it all right have a good weekend bye

Channel: Dynamic Machines Podcast

Views: 40

Keywords: Event based Vision sensor, Event Camera, Neuromorphic camera, vision system, motion detection, dynamic machines, podcast, Event-based vision Sensor

Id: uZ7ac9HpJeo

Length: 28min 14sec (1694 seconds)

Published: Sun Apr 07 2024