Oracle RAC Interview Question - Restore OCR

hello friends today i'll talk about another rack interview question is how to restore ocr as we all know oracle pressure registry is very important component it stores the node information it stores the instance to node mapping it stores the profile and it stores the resources and if the ocr is lost and then probably the cluster wouldn't start or it will crash so it is very important that we know how to restore it and this particular tutorial i'll be doing on oracle 19c on oel 7.5 so i'll be using oracle enterprise linux 7.5 and this will be done on oracle 19c so what we will be doing is like let me show you my cluster before before going to crs ctl stat res t you can see and you can see almost my database is up and running and all my resources looks good so if i do this and if i say grab db then almost everything so i have got two nodes one and two and looks like everything is online apart from one or two resources which we can ignore so my db is also open so looks good and let me show you where is my ocr so my ocr ocr check okay that is in a disk group called ocr so this is the disk group and this is where my ocr so i'm going to copy and you can see that i have only one this group i we can have five these groups but i have kept only one this group and what we will be doing is we will be crashing this this group will be crashing this particular disk group so we will see what exactly happens so let's take a look let's take a look we'll keep a note of this and before crashing we will check whether we have any backup using command ocr config minus show backup so we'll see if we have any ocr backup and if you don't okay so there is a backup on asm data too which was taken two days before and let's find out if we have actually asm cmd lsdg and let's give it a minute and find out if we have okay we don't have asm data too we have asm data one ocr and what this which means we won't be able to use these backups because the the disc group where these backups were taken that this group is no longer present so what we will be doing is we will try to take a flash backup so we'll try to take a flash backup so ocr config show backup to see the backup and ocr config manual backup to take the manual backup so i'm going to take this particular and the beauty of this command is once the backup is taken it will show me the backup name so one backup is on third of february another is on 4th february and let's see let's give it a minute and find out if it okay so it has taken another backup over here which is on 6th february just now so i'm going to take a note of this we can always navigate to this uh once you know if we have issues and if we know where we have configured so we can look at that but let's and you know beauty is it stores the date in date and time so we know when that particular backup was taken so we have this particular backup now and our ocr is in this particular disk group and i don't have any other additional we can configure five but i have only one and i'm going to crash this i'm going to crash this so how am i going to do that so let me show it to you what i'm going to do is ocr okay oracle asm list disk okay so you can see that i have ocr here i will delete this particular disk okay i'll delete particular this particular disk and before deleting i would like to tell you that you know okay so let me delete this particular disk okay that's done now i will do the list to disk one more time and you will see ocr is no longer there and i'll do the same thing on note 2 as well so here if you do the list disk you can still see so let me scan it because we have deleted it probably after scan okay so it has cleaned this particular disk and if i now do list disk you can see that i had three disk and now i don't have ocr because i have just clear the header and now what i'll do is i will go one more step and i will actually delete the partition so this is that particular disk i'm going to delete this particular partition as well the f this particular partition so i'm going to go one more tab and f disk so i'm going to not only delete the header but also i'm going to delete the partition so okay and that fails because i believe the cluster is up and running which is not allowing me to actually uh delete the partition which is fair enough so let me now okay let me now stop my cluster okay i'm going to stop my cluster okay i'll not use this one and i'm i'll be doing this same thing on note 2 as well so i'm doing stopping the cluster on both the nodes okay and then i will try to start my cluster on only one node i will not try to do it on both the nodes so let it let it finish let it finish okay so i've stopped it on both the nodes and has been stopped and let's wait for here as well to get that message and then i'll try to start my cluster okay has been stopped so that's good news so now the same command but instead of stop i'll say start crs and let's see what exactly happens so we i'm trying to start the cluster on one of the nodes so let's give it a minute and see what exactly happens so ps minus ef okay that is good news so now what i'll do is i'll try to say start cluster okay i'm trying to start my cluster and let's see if my asm comes up so same command i'll use watch and my asm is still not up so let's see my asm comes up and remember that ocr my cluster registry was in the asmbs group called ocr so probably if my asm comes up then i should be able to run this particular command also so give it a minute and find out okay so my asm has come online you can see the pimon you can see the asmp man so now i will try to run the same command ocr check and see what exactly happens fail to retrieve so okay so i guess even though asm so asm cmd lsdg try to find out okay i'm trying to find okay so i got two disks asm data one and word disk and i don't have the ocr and that is why my ocr check command is failing if you see and the cluster it is not coming up okay so probably i can wait for the cluster to come up but it's not going to come because i don't have my cluster registry i have just removed that particular disk so now what should we do so only option that we have is we have to restore it so to restore the restore the ocr what i'm going to do is i'm going to start this crs in exclusive mode on only one node but before that you know what i'm going to do is i'm going to make sure that everything is stopped on both the nodes so let's do that on node 2 so that's already been stopped looks good and i will try to do i will try to start the crs in exclusive mode but before that i need to make sure that it is also stopped on this node so this is not working out so let me stop it okay so i'm force forcing the crs to stop okay so and this is already stopped so then what i'll do is i will disable the auto start if you know i don't want so auto start i have disabled i'll disable the auto start on this node as well so that's done now what i will what i'll try to do is i'm waiting for this particular command okay let's see if it succeeds if it doesn't success i'm going to kill the ha process okay so i'm i guess it's not stopping so let me close this and try to say crs still stop crs shutdown is already in progress waiting for it to complete so give it a minute i'll i can keep on waiting so just i'll just try to see if it stops okay okay i guess it's not ready to stop so let's do one thing let's go here and ps minus ef greg css so these are the css processes and aj yes so let me do one thing let me kill all of them together okay so i'm killing all the css and has processes on node one so that's done and command stop failed i knew that would happen so that's fine so okay so two seven eight six six i have not stopped that so let me do that as well okay so that's done so now what i'll do is i will try okay this one will not work because it stopped so now i will try to start my cluster in exclusive mode on only one node so now right now i have stopped the cluster on both the nodes okay so ps minus ef graph uh iman no asm over here and no asm over here so now i'll try to start my crs in exclusive mode on one of the nodes so let's see and if if we are able to you know start a crs in exclusive mode then probably our work is half done that's victory half victory or half win then probably we just have to fix the restore we need to make sure our restore gets successful so let's see if my asm came online okay not at not at my asm has not come online so give it a minute okay so it's starting asm let's see not edge okay so asm came online and we need to make sure that this command complete successfully okay so while it is doing this i think okay we'll check whether my disk groups are mounted okay that is good my asm data one and vote disk is there so now let me try to restore this particular backup okay so i'll try to restore this particular backup using ocr config so i'm going to take this particular backup the same backup name i can take this particular backup and i'll try to restore it okay and let's see what exactly we get so i'm trying to restore okay so error so basically what happened is like the configured ocl locations are not accessible this is very important so we are the reason is our ocr was configured on this particular disk group and we don't have that particular this group the 2d group that we have is asm data 1 and wdisk so we don't have the this group called ocr so and this is very important so what we will be doing is we will be creating exactly so this we can use the different disk and all that stuff so if you have lost original disk we can use different days so that is not a problem what is very important is like we create the disk group with the same name as the original so we have to create the this group with the same name as the original so let's get on with that so come here okay still starting so that's not a problem so what i'll do is i'll come here okay and i'll say fdisk minus l okay i don't want this command so i just want the okay so i'll use the same disk 5gb so i'm going to create a partition on that particular disk f disk okay not minus dev sdf new primary and then just keep pressing enter enter enter enter enter and right that's done and i'm going to take the same command now and you can see that i have got a partition so now what we will do is before we'll create the asymptote oracle asm list disk only two so let me scan make sure that there is nothing stale and all that stuff so this disc only two so using or same command but instead create disk i'll give it a name as ocr and then i will give this and that's done and then i will try to scan this particular disk and then list it and you can see i have ocr so i'll try to do the same thing over here list disk shows nothing and i'll scan it so make sure that that particular disk is accessible on and you can see that ocr appeared and now if i do oracle assemblies disk you can see ocr on node 2 as well so that's good news so this is still i don't know what it is doing but that's fine so what we will do now is we will try to restore our ocr and this is the command that we are going to use ocr config minus restore and this is where we took the ocr backup of 6 february i'm going to use today's backup that was taken at 319 okay so i'm going to take this particular backup and i'm going to restore it i don't care you know i'm not bothered about this i think it's trying to start the other asm disk which i've already removed but that's fine so i'm going to do that so i'm going to use this particular command ocr config and i'm going to try to restore and now okay i'll try to restore it since we have created ocr and probably this command is again going to fail okay so configure ocr location is the same error so what exactly happened we have uh we have created asm disk but we have not created the disk group so we need to do one more thing we need to connect to asm so i'm going to connect as a root grid user my grid user is same as oracle user so sudo su minus oracle i'm going to set the environment variable to plus asm 1 i'm going to connect as cis asm and then i'm going to create the same disk group called ocr same name because my ocr was on plus ocr so i had to do the same thing so i'm going to do that so let's see whether i can create my ocr this group if this gets successful again another victory okay so let's give it a minute and find out okay so that's good news so we have created ocr disk group and then this command that failed this command that failed i'm going to take the same command the configure ocr locations are not accessible i'm going to take the same command and i'm going to paste it insufficient permissions again my mistake oracle user cannot do it only root can do it so let me try to do it using root and looks like i'm able to restore i did not get that error i'm able to restore my ocr so let's wait for this command to complete and okay the restore is completed and let's see what it is doing so it is uh okay it's trying to start the other things but i am really not concerned now because i have done my work i have restored all that i wanted to do was restored my ocr using this ocr config and for that reason i had to start it in exclusive mode but i'm done with that so what i'm going to do now is i'm going to kill this one more time and i'm going to start my crs and everything so let me kill my css and you know so what i'll try to do is i'll see if i can stop my crs using this otherwise and i'll try to stop it so starting shutting down okay so stay filled okay i am not interested so i'm stopping the crs okay that's done and now it is the final step that we need to do is we are going to start back our crs in normal mode on this node as well as on this node so okay i guess we have fixed we have restored our word disk and now our cluster should come back and we should not have any ocr check issues and all this stuff so that's done and that's done and now i will use this particular command crsctl start cluster all to start everything and let's see whether i get the output of crs ctl stat res t not edge not at give it a minute and let's see whether we get ocr check whether should we will not get it because i believe our crs services have not started so let's wait for it to start okay fail to retrieve that's fine okay that's absolutely fine cannot communicate that is also fine give it a minute and let's see what exactly it does so before that i want to make sure that the ocr disk is accessible here yeah it is accessible so ps minus ef grep pmon okay asm is started that is a good news okay and asm has not started on node one that is not a good news but that's fine so now using this particular command let's see okay so my note 2 seems to have come online so let me take a look at ocr check and you can see i don't no longer get the error and okay oracle cluster registry i think that is something which is still starting so let me try one more time crs ctl start rest all so i'm trying to start all the resources so that's what i'm doing okay and let's give it a minute ocr check here so i i have successfully restored my ocr so that is okay i i've got this error so that's fine okay and then what i'm going to do is i am going to check one more time whether my note instance and everything is coming up so looks like db1 everything is started yeah this group is still not on on db1 so it will come online and instance shutdown is okay it is open on node 2 it is not open on note one so let's give it a minute and find out if it is able it is coming up so shut down starting that is also good news and i and the disk group so we have fixed our we have successfully restored our ocr disk it was difficult okay only only key thing here is this this is the key thing you need to start your crs you need to stop the cluster on both the nodes you know you don't need to shut down the nodes if if you can shut down the nodes it is a good thing because then you are sure that other nodes are not doing any intrusive activity and you start the crs in exclusive mode and then you this is another key thing is like you create the this group with the same name where your ocr was stored so that is also key thing and then once that thing is done and when you start when asm is started then you using ocr config restore or this command your cr config restore you restore your ocr backup and then you stop it once again and start it and if i just want to show that everything looks good so let me take that particular command and run this and see okay my database came online and i will do one thing i'll grab this db and find out okay so looks good db db1 everything seems to be online up okay yeah everything seems to be online so this is definitely a good news and we have restored our ocr so if you ever face this question in interview how to restore ocr the key is you shut down the nodes and you start your crs in exclusive mode and another key thing is you create a disk group with the same name as the original ocr location and then you just restore it and again stop it and start it i hope this tutorial was useful thank you for watching and see you in next video and do subscribe if you like my channel and you know do share the video thank you bye

Channel: Subodh Khaire

Views: 787

Keywords:

Id: e0rpZ2aQMfU

Length: 23min 4sec (1384 seconds)

Published: Fri Feb 05 2021