Calibration and Stacking in PixInsight 1.8

so this video will show how to take your raw data bias these darks flats and lights and work with them to arrive at a single stacked calibrated frame of your Galaxy or your nebula that you can then do processing on and come to a final image so what we're going to be working with here is a step by step thing of how to do your biases your darks your flats for a few reasons one it's good to see how it happens at every step of the way and really understand the process that's going on behind the scenes - aside from the fact that I like to do it and that's how I do it it's really useful because if there is something wrong or you do have data that's giving you issues or the programs not interpreting things correctly doing it step by step you will immediately see the step where something goes wrong and that's really important because then you can correct it right away instead of wasting your time waiting for an automated process to do the work for you and then just see a result that you didn't expect additionally there can be some things that go wrong you know an odd-looking darker and odd-looking bias because of some settings that you didn't take care of and it doesn't actually make such a huge impact that to your eye there's something wrong at the very end so you'll just go on processing your final image as if nothing went wrong when you're actually not using the best image that you could so it's always good to check if you're interested in seeing what the alternative is if you go up to script and batch pre-processing there is a batch pre-processing script where you could enter all the calibration frames and all the settings for them by starix planets even your lights and have it run through takes a while does everything for you I tend to avoid this up to you so the first thing is to make sure that you have all your files all set so you can see I have for NGC 45-65 over the needle galaxy of biases darks flashlights and this folder called processing which you can ignore right now it's empty we'll get to that for this tutorial I'm only going to be using ten of each type of frame just to save time so 10 by is these 10 darks 10 flats 10 bytes but of course as with everything else you want as many of each individual one as you can when you're actually out imaging just to show you what one of the images looks like this is the galaxy we're going to be working on it doesn't look like much but if you stretch it by hitting this button here or clicking ctrl a it'll show you all kinds of details that are there and there'll be all kinds of green and other things here you can see it's not all that pretty so that's okay the whole point of calibration frames is to get rid of all that grain and to try and bring out as much of the original object as you can before actually doing any processing so we'll close the image and we'll get to work the first thing we want to do is we want to combine our bias frames so an individual bias frame I'll show you looks like this which again is nothing but if you stretch it so you can see at the very shadow end of the image looks like a bunch of noise it doesn't look like much keep that off to the side so I can show you later so we go to process we go to image integration and we go to image integration if this is your first time using pix insight don't worry about all the stuff up here and all the stuff in here we're just going to focus on very specific tasks for now also if this is your first time with fix insight you might be confused about the bottom bar when we get to it for now all you need to know about is that this button here is reset so if I change all kinds of stuff on here and I want to go back to whatever the defaults where I hit reset happens and this circle button here is the okay button or go so what we want to do first work from the top down we'll go we'll add our bias frames let's go without all of them there they are format hints ignore now this is assuming you're working with like a Canon DSLR or a Nikon this will also work with fits images which will come off most si CDs so for combination this is how do you want to combine the images will do average average literally means for every pixel in you know this image we have ten images they all share you know they're all same size so for each particular pixel like maybe you know that one there this image it has some value in the next image it has a different value go across ten images what is the average value so we see average you can do median min/max we want average normalization we don't want normalization what normalization is means if we have this image you know as some average brightness the next image also have some average brightness they're not going to match we don't expect them to match normalization says whoa if the pixel that we're looking at in this image is brighter than the pixel whirling yet in this image how do we know that it's actually brighter or it's not just brighter because on average everything's brighter good question comes in handy later not now we literally want to strict average because we're looking only at very specific sensor data waiting we don't want weights either that basically says you know you look at all the images and maybe this particular pixels value in this image should count for more towards the final than this one many reasons why that should happen there's more noise in this one you know whether you got fogged over or whatever stuff like that again working on just the sensor level which is what your bias is in your darks do we don't need to worry about that and your scale estimator here we will switch to median absolute deviation from the median don't worry about that just take care of it and that's it you can uncheck evaluate noise if you want because we don't need it when you do evaluate noise you usually do it in combination with weights because you know you want your noisier images to be less of a contribution than your nice clean ones if you keep it clicked off it's just going to eat up processing time so don't worry about that so go to pixel rejection so basically if we're adding all these things and getting an average we don't want to include outliers big bright pixels because of ion events you know high-energy things that struck the sensor and made a weird streak we don't want any of that so right now it's set to no rejection so we'll click that if you have a lot of images like 15 20 up then you want to go with Windsor eyes Sigma clipping if you have you know a dozen or so you might go with Sigma clipping I actually always go with Sigma clipping I've found that there can be some artifacts left over with Windsor eyes some Sigma clipping especially as you're stacking your images or their stars and other things Sigma clipping is also what they use when they do professional photometry so you don't need to really worry that there's a big difference there isn't really a big difference but you don't need to worry that you're somehow getting a worse result so we'll just go with that and again no normalization normalization appears again because you can apply it once for the image to figure out what should your pixel value be and then apply it again to make sure that you're rejecting pixels that are too bright because they're too bright not too bright because your image was too bright anyway none of that matters right now so we'll get rid of pixel rejection 1 we'll go to 2 and what's just at the Sigma low and high to 3 each that's basically how many standard deviations from the mean is a pixel have to be before you you reject that value so 3 is pretty average and standard and you're never really going to find the data set that's worth using that's going to need something higher than 3 or lower so we'll stick with that and that's it ok so you see we get all these results so close this for now projection high and low these are individual frames here and the reason why we're interested in these frames it is just to see if anything gets really weird for now when it comes to biases you might not ever need them but if you do start rejecting too much or you start there's some structures that get rejected you know you might see weird lines here then it's worth taking another look usually your rejection hi doesn't look like anything because remember our bias is all in the low end of the region which is why it looks black right now unless you stretch it so our rejection hi is nothing because we're not rejecting any high pixels because we don't actually have any high pixels rejection low looks like a bunch of noise that's good you can see there's kind of a structure happening here so we can get rid of that so now we have here's our old bias and here's our new one so if we stretch the new one suddenly you'll see looks first of all a lot cleaner this looks like noise this that looks like you can see some banding occurring here if you actually have a lot more biases that banding will be much more prominent the noise seems to have dropped back in the background if I actually open up here a master bias that I use it as about 40 bias frames ten of which I've used you can see you get a lot more banding a lot clearer if you go all the way in your noise level here is much more subdued from here of course your singl bias as the worst so we'll just stick with our new bias and we'll save it will go into this processing folder that's empty and we'll save it either as X is F if you have the new upgrades to pix insight or Fitz if you don't or if you just prefer Fitz those are really the only two things you should be working in so we'll call this master bias and hit OK whether or not you get your standing in Fitz or X is F there'll be another pop-up that comes up with all kinds of option you can just sit okay and it saves it you can see master by X is F there so we'll close it so now we have our bias so we have to do our dark so again we can open up an individual dark and see what it looks like looks like this grainy some hot pixels it's about a five-minute dark so we've got a process image integration and image integration again biases on darks actually take the exact same settings so we can just select all our biases remove them and add all our darks and just hit go so now you see we get another output close this again rejection hi there's nothing to reject because it's all in the shadows as you can see rejection low just a bunch of crappy noise get rid of that and now we have here our master dark if we stretch it now suddenly you can see instead of noise in the background you have this smoothed out you can kind of make out some of the banding that is from the bias signal the by signal of course is present in every frame you take and you still have these hot pixels so what we want to do is we want to save this as our master dark so now it's time to take into account the flats now the flats are going to be a little different so first we'll go to process image calibration and image calibration and I'll explain what we're doing here you saw we had the bias and the dark and the bias was a picture of the read noise and delay of the electronic signature on your sensor and the dark was a picture of the thermal noise the heat that generates all kinds of hot pixels in your sensor so the flats have these signals in them as well and what the flat is as we know is it supposed to be an image that only contains information about the sensitivity of each pixel so obviously if some pixels are brighter and darker due to heat or bias then we're not getting a true image of what that sensitivity difference actually is so we need to remove the heat and the bias from our flats before we can use their flats to remove pixel variations sensor a pixel sensitivity variations from our galaxy images so that's with the image calibration tool list for so we can go to add files we'll add all our flats and again ignore format hints output files is really important we need to specify an output directory the reason why is if I just hit go and I specify all the stuff down here which will get - it won't error without a directory specified it'll run through this will open up they'll do all kinds of stuff take a bit of time and then it'll finish but you won't have your calibrated flats the reason is when we did image integration we just said which images we want to merge into one so it popped up with an image the one that was the result of the merging here what's happening is it's taking each individual flat it's going to subtract the bias it's going to subtract the bias from the dark scale the dark I'll explain this in a sec and put the subtract that dark from the flat then you end up basically with a slightly modified version of your original flat doesn't overwrite them and that's good you should never have your original data overwritten so what does it do it wants to put it and save it as a new file somewhere so you need to specify where you need that saved so we will click this here we'll go in to our processing folder and just so that this doesn't become really busy as we work later on I actually like to make a new folder and call it flat Khalid and save it there just somewhere that shows you where your calibrated flats have been saved output extension is fit you can change it if you want I usually leave it fits doesn't matter and the postfix just basically means is it going to rename it and if so to what now because we're putting it in a different folder and our original work fits anyway it doesn't matter they're not going to overwrite existing files but it's nice just to have that there so you know these flats have been modified so flat 0 1 CR 2 becomes flat 0 1 C dot fit you can leave the rest alone now what we're going to do is we're going to select our master bias which we generated it would help to pick the right one and then you go and you select your master dark which we also generated and we uncheck master flat because we don't have one that's what we're in the process of making now what happens here is you have your master bias and it gets subtracted from your master dark your master dark includes in this case five minutes worth of heat noise on the sensor the flats were about I'm gonna one-sixth of a second huge difference in heat noise so what this does is it subtracts the bias from the dark because of a dark you can then be scaled down to represent just the dark of a one-sixth exposure then the dark can be subtracted from here to remove the one-sixth exposure heat noise the bias can also be subtracted from here because bias doesn't depend on time or temperature and then you end up with the flat that you want so optimize if that's checked off which it should be means scale the dark to match the flat in terms of temperature and exposure time which is what we want we also hit calibrate what calibrate will do is it's going to subtract the bias from the master dark which is we just explained we want that to happen so calibrate makes it happen now if you have an over scan region and this is only going to apply to see CDs that are a little more expensive I think by then you probably actually wouldn't be watching this tutorial then you can hit calibrate here on your master bias and what it's going to do is it's going to correct for the over scan region and then you're basically going to be left with just your bias after you're over over scan correction so you can set then by a subtract but that's going to sound like a lot of gibberish to most people again probably anyone who be watching this so we're not going to worry about that if you click off calibrate I just want I remember always select calibrate or whatever doesn't matter it's not going to make a difference because you don't have an over over scan chosen lots of talk so we hit okay so now it finished as you can see nothing popped up but if we actually go in to our folder we'll see we have now our new flats are saved if we open them up they actually won't look all that different from our original flats especially not to the I but rest assured they are much improved so now we have our Fox all taken care of we want to go to process image integration and image integration because we want to combine all these corrected flats into one master flat so we'll hit the reset button here because this time we are going to change some things we're going to add not our original flats but the new ones combination is going to be average as usual normalization now we are going to normalize multiplicative because the flats are going to represent differences in sensitivity we don't want a pixel to appear brighter because the image is brighter we want it only to appear brighter if it actually was more sensitive to light so we do want this normalization weights again we don't care ideally we've done our flats so that we're imaging such a bright light source that we don't need to worry about whether or not one has slightly better signal-to-noise than the other we ignore scale estimator and because we're not waiting we can undo evaluate noise now for rejection this is going to depend on the flats that you've done there is a lot out there that you can search about if you have stars and your flats or if you took it of a not perfectly illuminated object and all kinds of stuff honestly I'm just going to assume that you took good flats because not only can bad flats make an image worse than not having done flats in the first place but if you're going to do flats you need to do them correctly and it's much much much more important that you learn to take proper flats than it is to try and correct statistically for improper flats which will never be a substitute I'm just going to mention that if you do have some stars you know you took it against the dawn sky and you thought that all the stars were gone but maybe a bright star wears in the way you can actually go here and go to percentile clipping and then down here you have these percentile low and high and you usually want these to be really really extreme these you know 0.1 is too much you really want to put these really really low so you clip out the highest highest percentile things you know 0.05 or lower and the reason why is because you want to keep all your original data but pretend that the stars aren't in the picture this is never going to do that fully this is going to create worse signal-to-noise but it's better than not doing anything and keeping stars in your flats you would still want to change normalization here to equalize fluxes so what you want to use but because we've taken okay flats we're going to go to Sigma clipping or again if you've decided you're in the winds Erised camp go with that either way Sigma is low and I are both three and then we hit okay so here we go again we don't have anything on the high end we didn't have stars anything weird you can see our low is now not full of noise and we have our master flat so I'm going to do two things here first for comparison I'm going to show you what one of our original flats look like and second see when I stretch this it's going to be really blue and it's really blue because they took it against the dawn sky so that's the atmosphere if you go to process intensity transformations and screen transfer function what you actually see here is the thing that's doing this stretching whenever you hit control a or that button so if we hit reset you'll see everything starts off here and as soon as I stretch it jumps and what it's done is it's treated each channel equally so we get blue because if you stretch them all by the same amount blue is going to dominate because of the sky I want to see this in black and white the color actually doesn't matter when the program is doing its own stuff in the background but it's nicer to see black and white if you unlink it it will do the stretching so that the channels are all equally weighted I do that suddenly you can see it stretched the blue less than the rest and it comes up as black and white so first of all we'll do this to both first of all you can see right off the bat there's a lot more grain and noise here than there is here and that just comes from stacking that's why you want to take as many flats as possible and that's also why you want to do it if a bright source like the pre-dawn sky because if you do it on a dim source then you stack flats for a dim source they might look like this just because there isn't enough light coming in so some feature is here to note it's brighter in the middle than it is towards the edges and that's because optically there was more light coming in and hitting the center of my sensor than there was hitting the edges of it so that means that same defect is going to show up my galaxy images and we want to correct for that now nothing here is going to show up in my darks or biases because those are done with the cap on so no light entered the system you also have these weird circles happening these are actually dust particles or little grime and they could be on any surface in your entire optical track mirrors lenses focal reducers the sensor itself and the reason why the big circles is because there should be a little speck but they're out of focus because you were focusing on the stars this is really important this is why you don't want to change your focus between doing your images and doing the flats for those images or if you are going to change your focus make sure you do flats before changing your focus as you change your focus the size of these change when we use the flats it's going to divide by the normalized version of this image which means if you can imagine the average brightness of any pixel here is one then the dark circles are going to have pixel values less than one the bright parts are going to have pixel values more than one of course if you divide a number by something less than one you get a big number if you divide it by something more than one you get a smaller number in other words it flattens out the entire galaxy image the galaxy image has these dark circles it makes them all smooth and flat along with all of this brings up the corners here darkens down the middle so you end up with a perfectly what this should look like is a perfectly gray flat image or in your galaxy image one that doesn't have this vignette and one that doesn't have these circles so we're going to save this this master flat and then we can close this so now we're ready to do some things to our image our galaxy images we'll go to process image calibration image calibration but now instead of calibrating our flats you can select them all remove them and add our lights we're going to work on our galaxy images and now we still want to leave all this stuff the same as it was before because we're still need to do the darks and the biases out of our galaxy images but we can now check master flat because we have a master flat and add it we don't want to hit calibrate calibrating it takes into account master bias these master darks all that kind of stuff but we did that to our individual flats that make up our master flat so this is just going to perform things that don't need to be done and because we've actually done the best job we can this will make it worse so we don't hit calibrate we're good with just that and of course output files it would be nice to have a separate directory where instead of having calibrated class they're just calibrated lights so we'll change that everything else is the same we hit go so now it's finished when we can get out of this image calibration and again as before we have our new images and calibrated now unlike the flats we can't just go and integrate them because our images have actually shifted over the course of the night if I open up the first and the tenth I will just stretch them so we can see a little bit of what we're looking at they look black and white because screen transfer function remember is not linked so it's aiming for a color neutral image so we'll just go back and do that so if I actually hover this over the other you can see this kind of like weird effect and that's because if I go into full crop it's a little easier to see the images are shifted slightly so if we stack them we'll get a really weird-looking results and it'll be terrible so we need to do what's called registration or alignment and that's going to basically be done with the tool under process image registration and star alignment so what we do here the first thing is we'll go from view to file a view is something that's open already in your project just cleaner if you work with files because you can locate them on your hard drive but it's up to you you can have everything open if you want you can see you can add files or views let's just use file we'll select any file is good actually as a reference file by default I just always choose the first one but you know if if the galaxy was way up in the corner of the image and you didn't realize that and you realized that it image four and then you centered it then you know you want the galaxy to be in the center of your image you could choose that but I mean my first image was good I was happy with my first image the way it was framed my final image looked like that one just with less noise that would be fine so we'll select that as for add files we'll select the rest I actually I like to select them all it will basically compare the first one with the first one find out that there's zero difference and just save it eats a little bit of computation time eats a little bit of hard disk space a little unnecessary but I like it because it converts cr2 to fit it will get all my aligned images into the same folder up to you so they're all here again output directory just like with image calibration if you don't do an output directory you can just hit go it'll waste your time we want an output directory and I make another folder I call it registered and you can see the postfix here is our four registered instead of c4 calibrated now depending on your data set this might actually be good enough and you can just set go but if you want you can toy around with the star detection now this can get as complicated as you want we don't usually have to play around with this but you might find that after you do registration some images weren't aligned or somewhere aligned but are still slightly off in that case you can up your detection scale too so it looks for larger structures you know if you have really big stars instead of small ones if you find your stars are a little distorted or if it tends and your tracking was not perfect so they tend not to be roughly circular you can actually decrease this and you'll be a little more sensitive to what it will count as a star stuff like that but the default values generally look really good in the end so you can hit just go if you're using views instead of files you actually need to hit the square because the square is the okay button or things in your workspace and the circle is the okay button for anything anywhere on your computer so generally you can actually go on to just do the image integration because most of the time these settings work really well and you don't have to worry about it and you don't get a weird image it's also really tedious usually to check every single one of your images but in this case we'll open them all up and take a look so we've got our first image our last image compare them and we'll see that now we don't get a weird issue we can do this along all our images if you find that you have one image that doesn't just shove it off to the side for a bit keep checking your others if you want if you have just one or two that are different than what they should be you can actually do the process manually you can go to process image registration and make alignment and then you click on your reference the one you want things to be transformed to match and then you check on the one that you want to work on then you click on a star not a super dim one but not a super huge one click on a star and it will guess where that star is on your image you can click and drag and move this round to correct it if it's wrong you do this on a bunch of different stars it takes two stars to figure out whether there's rotation and whether there's been shifting or translation what you can do as many stars as you want I mean the more technically you do the more sub pixel accuracy it will give you if you pick the ones that are closest around the edges you'll be a little better off anyway then you can in just hit the checkmark and it will work and it will match this image to this image let's get out of this and then you'll see that they will be matched and then you'll have to go to file save as and save the specific image under register yourself that's if it's one or two images like this is going to get tedious really fast you got to go back to star alignment and just only you know work on the images that aren't working out for you and toy with these settings I found that only a few times I had to toy with these settings reducing maximum distortion and upping detection scales worked like that but that's okay so anyway everything worked out really nicely and now we have registered images now what these are are these have had the dark subtracted the bias subtracted there have been flat corrected so all those dark circles and light parts have been flattened out and they've all been aligned to each other so they're ready to finally stack into one final image so we go back to process image integration and image integration and again if you can't remember what was said or what's not supposed to be set just hit reset we'll add our registered files now here the defaults are really nice we want at average we want combination to be average again we want the normalization to be additive with scaling we want the weights to be noise evaluation which means if we keep noise evaluation checked off which means when it looks at each image it will evaluate noise contribution of each and the ones that are noisiest for whatever reason maybe there was some high-altitude fog or something happened the focus didn't go that great you know because the atmosphere was changing that will be contributed less to the overall image which is good you're reducing the overall noise really well and the scale estimator should be left at default so in terms of defaults your image integration you don't touch pixel rejection again Windsor eyes if you've got fifteen twenty plus and you want to go with what you've heard Sigma doesn't matter I'm going to with Sigma keep that normalization at scale plus zero offset and we'll change these to three and hit go so now we can close image integration and we can see that our rejections both high and low don't really have much going on that's good there's a little bit of action in the rejection height but nothing to be worried about and now we're left with our image and if we stretch it we can see a lot more of our galaxy now we still see a lot of stuff in the background two reasons why not to worry about that first of all if we open up a single image it's going all the way we can see that there is a lot more going on here so actually if I go to process intensity and screen transfer function and I work here and I scale it down so that you can see roughly the same brightness of the galaxy you can see the background has gone away a lot more so they're about as bright here except now there's more galaxy detail going on the fainter stars are much more visible and the background noise has dimmed if I actually apply this here much much greater amount of noise so if you actually click and drag this triangle on to an image it will apply the settings of that window onto that image so I can be sure that both these images are stretched in the exact same way so here you can see even after 10 images a remarkable difference that versus that even at this scale you can see the noise and actually the full version I did with the full data set which was 17 darks 40 bias used 27 flats and 50 light frames and apply this same stretch here we can see a huge difference so first of all at this scale this appears noiseless compared to this and you can see you can pick out the stars immediately that you can barely see here even here at this scale you can see that this looks a lot smoother and if we actually go in this a little bit of noise has almost completely disappeared the galaxy structure there's a lot more you can see much much nicer by comparison here there is no comparison so obviously stacking is the majority of what helps but you do need to take care of all your calibration frames because what happens is when you do want to bring up the really faint parts of the galaxy if there is a lot of heat and bias and pixel sensitivity differences there then where you know the amount that you can bring up towards the edge is less you can bring out more and more faint signal if you've done everything you can to ensure that that is signal and not noise so this is our image and just to show you again after processing this is what the image became so we started this is our stacked image this is our final image much much nicer and of course this was just one image so I'm from here to here to here calibration gets it from here to here processing gets it from here to here and that's it so of course you want to make sure with your original stack file that you save it call it integration or master stack or whatever you want and now you're ready to start doing some processing on your image

Channel: Richard Bloch

Views: 74,995

Keywords: Digital Image Processing (Field Of Study), Astrophotography, PixInsight

Id: zU5jJgjKuQQ

Length: 37min 6sec (2226 seconds)

Published: Tue May 05 2015