Check Point firewall advanced troubleshooting and Kernel Debug! Slowness, VPN, Logging, Policy issue

hello guys and welcome back to my channel firewall gan today we are going to walk through with troubleshooting scenarios which are most often you face during checkpoint firewall administration if you want to become expert in checkpoint firewall and checkpoint firewall troubleshooting then please watch this video till the end and understand all the scenarios all the scenarios listed here those are derived from and faced from during a production environment so without wasting time let's get started scenario one common troubleshooting scenario scenario one a user is complaining about latency and connectivity issue they have been trying to ping the ftp server who has the ip address 192 168 11.10 but are getting any response i mean they are not getting any sort of response from that ftp server packet capture on the ftp server shows that the pings are not reaching to the destination so this is the number one scenario and the basic scenario we almost all faced or will be facing during our production environment so let's understand how to you know troubleshoot this sort of scenarios what are the debugs we're going to run to you know resolve this issue let's do the troubleshooting step one you have to do this debug from the checkpoint firewall gateways cli from the expert mode step one do or initiate debug the command is fw space ctl space debug 0 this will initiate your debug then assign memory buffer space to this debug so you have to use the command fw space ctl space debug space buffer size give the before size 32 7 6 8 this is the buffer size you need to give at least so that your whatever the output will have you know enough memory space step 3 take a kernel debug so this is the important step here you have to give the actual command where you will put your debug thing so the command syntax is as as you can see on the screen it's fw space ctl space debug space minus or dash small m space fw space plus space con con is nothing but your refer or term for the connection it's in a checkpoint negative language and drop space vm here we are referring the connection which are coming to your gateway from that machine then step four store debug output to a file so you will create a file and whatever the output generate out of this debug will be stored so the syntax is on the screen you can note down the syntax but i will you know just repeat it it's fw space ctl space k debugs space dash uppercase t space dash lowercase f and you have to put that output to the file called your debug file output.dbz after having this debug in a place you have to reproduce your issue so you have to basically try to ping from 192 168 11.10 to the ftp server and once that is done you have to stop the debug and which is using the command written fw space ctl space debug space zero and once that is done you have to take the output file which you have stored read that output file and you will find a possible cause what is the exact issue now let's add something more onto the scenario now the ping is now working but the user has discovered another problem when they connect from their windows host to the ftp server everything is slow and they something i mean sometimes they get disconnected taking a traffic capture on the mtp server shows that not all the packets are reaching so this is the another things which is occurred now let's see how we can troubleshoot and proceed with the debug so if you see the steps every step still the time uh you put the uh your i mean till the time you assign your buffer is the same for example step one initial you need to initialize your debug step two you need to assign the buffer space to the debug steps three is important here is a slight modification what you need to do well you have to give the debug command for the ftp protocol so the syntax will be fw space ctl space debug space minus small m space fw plus connection con drop space vm space ftp this is the important modification for the syntax here step four store the debug output file as you can see on the screen accordingly you need to store your file you have to just copy paste this syntax and step five you have to reproduce the issue for example you need to again try to ping to the you know 192.168.11.10 to ftp server and then stop the debug using the command fw space ctl space debug space 0 and then step 7 you need to read those uh the generated file and you will found the possible cause what is happening with that packet i'm sure this sort of uh debugs are very useful when you you know facing issue when the connections are not working well or there is some slowness in a network for a particular source to the destination so for such uh sort of scenarios you have to run this debug for sure so if you want to note down these things please uh pause the screen pause the video and take down the debug now let's understand the very important thing and very important scenarios with respect to vpns vpns is everyday work what we have to do it's like we have every time we used to face some sort of issues and you know you need to work on uh to resolve those issues in terms of the vpns so under let's understand the vpn scenarios now when you say this is the you know statement when you run to vpn issue and you find side to side vpn is not establishing you need to run below debug to find out issues run all these commands in expert mode for example you have to log into the gateway from where you have a vpn configure and you have to go to the export mode and whichever the upcoming commands i am going to show you those you have to run from the uh expert mode this is one important utility which is called vpn 2 the syntax is vpn space tu you if you are not sure if the vpn is causing the issue you you can use this utility and select whenever you run this utility then you will see you know number of options there you have to select option number seven to check the status of essay which usually gives you if the essay is you know uh present and the vpn is working well so use the option 7 here and now let's go to see how the debug can run for the vpn issues please note vpn issue packet flow and packet you know having the issues on the packets can be seen using the tool called ikey view tool and there is a file called ikey.elg file which you need to generate please don't i have a very important you know topic discussed during my vpn uh videos so just check go back to my you know video list and you know look into that vpn troubleshooting videos i have explained in detail how the vpn troubleshooting done with respect to the ikey tool and ike elg file i have shown each and every packet and there you know how they look and how you need to understand those things for this uh let's jump back to the vpn debug troubleshooting so step number one move your existing ikea.elg file to some other file name give the some other the file name or move to the some other location usually what i do uh i usually create a backup file whichever the ikea file i have existingly with my system i usually create a one more backup file for that and the syntax is for that is as shown below from the export mode you have to run below command so command is mv mv for the move and your file i key dot elg and give the space and the new file name where the all the data will transfer i key dot elg dot b a k b a key is for the backup only then step number two start your vpn debug so vpn debug can be start using the command like uh uh vpn debug trunk or uh or you can use the command i have shown what will work so vpn debug i key on is the command which we're going to run to start the vpn debug once you started your vpn debug then you have to you know run or reproduce the vpn issue i mean uh you have to replicate the issue so the issue can be replicated in a variety of ways but the simple way what we do you have to just ping from your vpn site one vpn site system which is is configured in between the uh between the in in a vpns so from the whichever the encryption traffic you have from there you have to just ping from one side to the another side vpn machine for example you have a gateway a and gateway b and you have a vpn established between them so under the gateway a you have defined some interesting traffic or encryption domain say uh one two three those are the machine name and under the gateway b if the appear site they also have some interesting traffic defined say one two three or say four five six so for example from one you have to start a ping to say any ip address from four five six machine and that way you will reproduce the issue once that is done you have to stop the vpn debug and to stop the vpn debug you have to use the command vpn debug off uh please note uh the all letters should be uh in a lower case so i can see the v is look like a capital over here but don't use that use vpn debug of all at a lower case okay and then you have to take the i key file the ik lg file will generate it and if you want to make sure that file is generated you have to use the below command which i have shown you which is ls space dash la sign space dollar sign a wdr forward slash log forward slash ikey.elg and once that file is generated you have to take that file uh from your gateway use win sap or any uh ftp or sftp uh tool to get the file from your gateway to the uh ik tool so you have to open ik tool and from there you have to open this ik.elg file and you will see series of the packets uh which packet is failed or not uh based on that you have to do the observations right now let's see most common causes uh which you know have uh which you give with the vpn failed issue or vpn is not coming up issue so two most common uh cause for the cytosite vpn issues are below these are the most general causes uh mostly whenever we found that the vpn is not coming up firstly we do check of these two issues and then the rest of the packets i have explained the rest of the packet as well but let's start with the most common so number one is phase one main mode packet number five it gives at the if you open the packet number five there is field called invalid id no i'm so sorry there is a a field called id once you click that id field uh in mm packet 5 and if you verify the information for example what are the details uh written over there uh you will see there is a something called you you will see something over there as invalid id so that invalid id is referring for your ip address so the ip address of your gateway so if you are trying to establish a communication between your source gateway to the peer gateway so i mean the one side get to another side gateway if you go to the id field one you will see your first gateways ip address so this ip address is the external ip address and is external interface ip address usually sometimes what happens you know due to rush or some other reason we put some wrong ip address and from for that we will see this invalid id errors this is also important if you check the id field two this will give you the response uh packet or or i mean this will give you another response packet this will give you the ip address of your peer site gateways external interface ip address which you have used as a tunnel configuration so if there is some issues on that ip address if you not put it the gate peer gateway's ip address correctly then it will show you this invalid id issue and in a phase two quick mode also we see this invalid id thing so if you go to the phase 2 packet 1 and over there if you go to the id field here also you will see sometimes invalid id field but here this id field refers in different sense so this id field is something refers to the wrong subnet which is being negotiated between your site we call it as a checkpoint we can uh vpn encryption domains or or in other word it's uh interesting traffic so uh if you have a defined encryption domain uh for for example for your site 10.0.0.0 and from other side uh say 1 192 168 1.0 for example like that and that is def that you have to define from your to your gateway plus the other side gateway but if you define something else or maybe some classic example you know some sort of super nighting is happening on this gateway this packet packet 2 of a quick mode uh uh it gives you these details like uh invalid i think i i mentioned the packet one of the quick mode at the initial it's not a packet it's a packet too of the quick mode so over here you will see that the encryption domains whichever you know you are sending from your site and receiving from the other side those are matching or not if those are not matching obviously the vpn terminal will not form so these two you know common issues what we face during the uh i mean these two common issues we have or during uh vpn site establishment we used to get all right now let's see the other packet details so whenever you see when you open that i key view tool and you will see uh packet one and two from the main mode so this is what happened if that issue the packet is showing like a fill so here you have to check responders lifetime problems like a live duration set for phase one again you have to check no proposal chosen i mean uh this this the the points i have mentioned those you will see on that i keep it tool okay for example you will see responder lifetime problem uh over there you will see live duration set for the phase one uh whatever it is in a you know uh uh you know seconds or a minute whatever it is if that is not mis that is not matching you will see your packet one and two those are getting filled you will also see there uh i mean uh during the you know uh lock capture when you capture the logs of your vpns uh in a lock from log server uh on the packet capture or sorry on a log capture you will see a specific errors like no proposal chosen or you know invalid cookie something like that or invalid uh exchange type like that so when you see like no proposal chosen so it's something with your encryption strength mitch match so whatever the encryption algorithms you have defined for example uh sha uh aes or whatever it is uh like uh your integrity algorithms like md5 everything whatever you have design if it's not matching you will get this thing like no proposal chosen right uh whenever you will see this invalid exchange type so you have to think about like aggressive mode which is a part of your of phase quick i mean uh main mode you can define your main mode in aggressive aging mode as well so if you have a uh errors uh showing that invalid exchange type it's like a aggregate mode is turned on for one of the peer but the other peer is not uh you know enable the aggressive agent right so this is what you need to consider if you see the invalid exchange type if you see invalid cookies so center gateway has a new object change for example uh you are you you are facing you have the uh say uh coming side to side vpn between your uh one gateway to the rest of the gateway and you have changed something in the encryption domain of your center gateway that time you will see this invalid cookie error so if you see this error you have to think okay something has changed with the center gateway and you have to consider that manage gateway on edge is mismatching the vpn request to the old object the solution is as simple as that whatever you have created as a new object you have to delete your object now let's see uh packet three and packet four if you most of the time you won't be able to see the packet three and four uh you know showing you as a fail but usually this is a weirdness sometimes uh if you've seen so you have to think below certificate unavailable so maybe you will see this error in our log when i run into this uh this sort of issues uh with respect to the firewall you know the h5 walls or the the s5 or certificates if those i got expired if there is some issue with the certificate things and when i renewed such sort of certificate though this sort of uh packets and packet for you know showing error got resolved um you know there was a time where uh i had a wrong gateway as a peer and that was when the other gateway answer was some encryption setting were probably different so i'll uh this is a statement of one of my uh classic example i have traced during my you know uh walking scenarios with the vpns so uh there was a peer vpn gateway uh which i have selected the wrong vpn pl gateway and somehow that the communication got started but you know the encryption setting was mismatched because i've been a wrong gateway i was seeing the packet 3 and 4 as an issue now let's see uh the packet 5 and 6 of main mode so i think we already discussed on uh uh initially uh like uh invalid id so it's the same thing uh if uh anyway there is some issues with your you know ipa addresses bit upper your of your uh both the gateways mismatching you will see this plus if you see the packet fio is fail so these are the you know other reasons one is your authentication fail like your pre-shared key is one of the reason then payload malfunctions say for example you have selected a wrong secret key maybe your invalid certificate is coming up into the picture certificate is invited or your crl cannot be retrieved from the management server or maybe the certificate is unavailable your certificate is no longer exist on the member sending this messages that may be the cause or if the packet six which is coming responder packet from the peer gateway if that is showing fail so obviously it's with respect to the lifetime problem so you have to check what the lifetime uh you have given there so this is all about the uh main mode now let's see the quick mode so the quick mode uh we have already seen uh like the commonly if you have a mismatch in the encryption domain you will get the issue but if again uh uh you have some other setting which is also mismatching so uh when you do the this um uh log uh i mean you will see the packets in a logs uh uh lock server and you will see uh phase two is giving you no proposal chosen sort of error so usually that is also with respect to your subnet mismatch and uh maybe can be your lifetime or a encryption strength uh which you have given for the phase 2 obviously we all know the phase 2 we also have to give the different encryption algorithms and all those things so if those are mismatching you will see this error so other guys this is all about the vpns now let's see the error occurs during the policy installation first on the management server and before going on to that let's understand uh the the errors which we get during the policy installations so uh you know most of the time we have seen uh when you try to install the policy uh the you will face the error or you will get the errors during the policy installation there are you know too many errors uh scenarios uh we may see during the policy installation there may be the you know variety of the reason for that but i have come up with you know a few important steps where you can you know do this common uh debug for the policy installation thing uh policy installation error and you will come to know why that causing that issue okay so let's jump into that so uh the solution is first on the management server get the policy name and the name of the target firewall customer object from the dashboard so you have to take the your the firewall name and the policy package which you're going to install and then log out from your uh management server and run this below command for uh command from the management server itself so this command basically uh fetched the policy uh from your management server to the gateway so you have the syntax is fwm space yeah you have to run this command from the expert mode of your management server so the syntax is fwm space load space dash d the policy name the policy package name which we are installing the target target is your gateway where the policy is getting installing and a file name where you have to uh have this detail to be logged in so this is a syntax so you will see an error if the management server uh you know uh throwing error or here only if not the debug policy on the firewall from the in-person model uh will give you the details so below is the actual you know command you need to run uh with respect to your you know gateway name and everything and this will pull uh it will pull from the policy in the firewall database right so the syntax is fw space dash d fetch local dash d then your dollar sign fwdirs forward slash state forward slash underscore tmp fw1 and the outputs thing so the initial command uh which uh we discussed was to take the policy from the management server which is fwm load and this is actually the fetching your local policy from your uh management server so you can also try to fetch from the management server but most of the time that will never help so you have this command is very important you can you know run this command as well so most of the time the automatic errors which we usually get during the policy compilations they usually happen in two reasons let's see those reasons number one the gateway is heavily overloaded and there were not enough resources example like a memory or a cpu to complete the automatic load process a typical symptom is that it will take several tries to successfully load a policy to a gateway or that the policy load succeeds more often right after the gate very boot number two the compile policy managed to pass verification compilation on the sms security management server and it had to probably sh probably should have not have as the compiled policy containing errors that prevents it from being successfully loaded into the gateway kernel the gateway will perform lots of extra sanity checks on the policy during the atomic load process as pushing a vector sorry corrupt policy into the kernels is likely to crash or significantly impair the gateway the symptoms here is that a policy push to a gateway will fail 100 percent of the time usually after making some significant changes so if you performed all the changes and whatever the steps i have just mentioned before if that is also done and you are not able to you know get rid of the issue so you can run the policy kernel debugs uh there are a lot of x case available how to you run the policy quality books and still if it's not you know give you some right answer you have to log in take it to the checkpoint and get help from the tag now let's see the firewall logging issues what are the issues we get during the firewall login and how we can you know rectify so most of the time when you you know see your firewall is not sending logs to the management server or management server is not receiving the logs of some of the another reason is happening and you know you are not seeing the logs so these are the steps or these are the things you need to consider number one check if the firewall login is locally going to the dollar fwd ir forward slash log and running so you have to see your logs are you know listing into this directory log there are a lot of uh you know syslog messages also there like a message one message two messages files and what are the logs you know we are producing is those are going to this log directory and everything is running there you have to check that and to check that it's simply you have to run this ls space dash l a l is a small case space fw dot log so you will see that fw dot log file is getting generated and you know uh the logs inside that uh are you know processing or producing if you want to check what is inside that ibu.log file you have to just give the command like less less space fw dot log and you will see what is happening you have to check several times over the course of the minute and see if the file size is increasing or not this is the first step uh what you have to do for the firewall logging issue if it's logging locally that means at least the you know your firewall is you know logging the logs and it's not getting send maybe to the management server so second step is like if it is locally then do the cp stop or you have to rename that all the your fw dot log files and then do the cp start then to force it to try to log back to the management server again and once that is done uh you have to run the command fw lock switch so log will switch if that also doesn't work and you are saying uh uh saying that fw dot log is growing it may be that there is a reason it is uh reason that is isn't logging to the management check for the old log that says the log respo repository quota has been exceeded or no file could be deleted sort of things if there is a problem like that remedy the same solution and go back to the step two so if you see the repository quota has been exceeded when you go to the your log messages which are present over there so over there if you see a repository quota has exceeded or something like that again you have to do the steps to what i have just explained more on troubleshooting on the log issue so these are the steps please note down these steps this is very important if the logs are not growing check for the firewall attending to the logs so you have to run the fw monitor uh or the port destination board 257 257 is therefore for the logs availability processes so you have to run this uh fwd monitor and see if the logs are getting uh generated let it run for at least a minute this command if it's taking management then you have to check your seek is also there six status it's communicating then you have to check get the adw monitor uh you know i mean taking your fw monitor uh file which you have just generated from your web monitor command and run into this uh you know wire shark and follow the you know uh tcp stream so it's very simple you have to just get that ability monitor file right click on to that and on the sync packet you have to select the tcp stream and it will show you all those packets then scroll to the right where you will see the certificate and possible the word deny if this is the case the smart center is rejecting the firewall certificates to fix this install database to the lock server and management server so the management server can update itself to the trustee firewall certificates next would be the you know the reboot of the firewall this will force a fresh third exchange between the firewall and the management server and maybe till here the issue will fix and uh if it's still not fixed then there is a time you have to you know open a ticket with attack and proceed there is a old uh ways like the steps i have just mentioned that from 12 onwards check your dollar fw dr forward sash con forward slash master file on the firewall and add the ip to the smart center master file is helpful on you know the the old appliances like one one zero zeros to change which ip they are sending out logs check master file order to apply to make sure that is done right otherwise trying to resetting the sc uh seek will you know fix the issue could also be the likelihood of the things which we can do like debug your fwd on a firewall and fwm on the management server check your desk spaces reboot the management server create a dummy lock server define a firewall lock to it push the policy change it back to the management server and push again delete the dummy lock server check time so guys uh i have tried to explain you know too many scenarios during my uh what i have faced during my you know overall 10 year of experience with the checkpoint firewall so if you like my video please subscribe my channel and press bell icon so whatever the latest video i am uploading it will come to you if you like this video again i'm saying subscribe it and thanks for watching this video have a good day ahead bye bye you

Channel: Firewall Gyaan

Views: 216

Keywords: Check Point firewall, VPN, VPN troubleshooting, Kernel, debug, Check Point firewall troubleshooting, top Troubleshooting commands, Check Point Basic troubleshooting commands, Check Point firewall Site to Site VPN issue, Check Point Firewall Remote Access VPN issue, Check Point Policy Installation issues, Check Point Loggin Issues, Check Point Firewall showness issues, packet flow, check point firewall labs, mds, vsx

Id: 6eEJppZZekA

Length: 42min 19sec (2539 seconds)

Published: Sun Oct 03 2021