GCP Cloud Logging | Export logs to GCS , Big Query , PubSub for further Analysis or Process

[Music] [Applause] [Music] gcp data engineering so in this video we will see the different options we have to export these logs to different other Google cloud services like Google bigquery or Google cloud storage and also pops up okay the first option what we will do we'll try to create a bucket and then we'll create a scene and we'll try to export this logs okay and then bigquery will create a data set and we'll try to create a sync to export this log in the third case we will try to we have already Pub sub topic created we will try to use that pops up topic to export logs okay so let us go to demo again here we have this locks router click on this locks rotor now I would like to export this log to my GCS bucket okay but here already you can see there are two default buckets already created so these are completely internal to Google Cloud platform by default these are pin created by Google Cloud platform they will try to export log information into these two buckets but as a user if you need to if you would like to have a separate bucket you would like to export this log information by applying some filter you can create a separate bucket all right so we'll do that okay go to Google dot storage right here create a bucket let me give log exports one two three okay now there is in U.S continue let's let's create this bucket now we have this bucket all right now go to our this log router and click on create sync sync name sync GCS okay here sync destination you need to select cloud storage bucket because we are trying to export log 2 our storage bucket so select this so here if you would like to apply some filters right as let me go to my login module operations logging if you like to apply some printer like this by selecting the resource bigquery right then this is one filter you can include this particular filter over here okay right so that you can restrict the logs to be exported to this particular destination but I do not want to include any filter that's why I am not giving any filter condition right next so let us keep the remaining option as it is let's let us create this thing then this will create a sink now we have starting up you can see so we have this sink is created to export log to a GCS packet and also there is one more step is required just whenever you create a thing click on this view sync details there there will be one service account associated with this sync okay you need to give a necessary GCS related IEM permission to this service account in order to export or allow this particular thing to export logs to this disease packet just copy the service account okay now go to IM right so here you need to Grant access to this particular service account okay so it has to write logs to a GCS bucket then it needs to have a Jesus related IM permissions right so storage object created storage object created okay would be enough RL storage object had been also fine right so let us give storage up checked admin it would be better okay then same then it it will start writing those logs into this GCS bucket okay so basically when you are trying to export logs to a bigquery or this is bucket it write those logs in batches maybe it will in every one hour it will not be immediately available ah in the bucket immediately right so you will have to wait for one hour and then see then those locks will be available into a Jesus bucket or bigquery data set but in case of cloud pops up so whenever there is a log immediately it will be published to a pub sub talk almost real time so there is no there is no much delay actually it almost runs in real time right so now we have created a sync to explore Lux to this is bucket in the same way we'll have to create a sink to write this log to a bigquery data set so before doing that first of all let us create a data set in the bigquery to export these logs okay so bigquery to create data set data set name log Sports us multiplication table expected data set okay now I would like to export my logs to this particular data set go to the router createsync same way sync name export BQ okay to be q and then next you need to select bigquery data set select the bigquery data set log export that's it keep past this keep the remaining options Justice and then click on create sync then this should create that sync now go to the sync again now you can see so you can say export BQ this is a scene so click on using details now this also has one more service account associated with this thing you need to give necessary permission to this particular service account to export locks to a bigquery data set so copy this okay now go to IM Grant access right and select the bigquery role so bigquery data editor would enough right click on Save so whenever you create a sync by default there is one service account created by a Google Cloud platform itself and you need to give an essay permission to that service account so if you are trying to write luck to Jesus bracket you need to give GCS specific roles you are trying to export those locks into bigquery data set you need to give that bigquery specific roles okay now we have these things so now you should see these logs after one hour right but for our demo I've already created these two types of syncs already right two days back and exported these locks into Jesus bucket and also bigquery now we'll see how those logs look like okay now go to the bigquery so this is the data set where I exported my logs you can see yeah they will look like this so activity and data axis ok so when you click on this data activity right you can see so you need to specify only data set while creating the sync so automatically whenever this particular sync is trying to export logs to bigquery data so it could create this table this is a default name given by that particular sync the activity and for that access you can see this is an ester table right so click on preview due to some constraint we are not able to see now let us try to query the state table okay so it has almost 42 rows okay query is stable in Utah run okay there is no total let us remove this filter okay now again uh there is some constraint so let us see this result in the Json format you can see it it looked like this only right similar here same way right so this is the log so like this it will export all these locks into bigquery data set in batches so what is the advantage you have once you have this log information into bigquery so you can perform your analysis using SQL right you can write a SQL on top of this table you can try to extract the required information right by using our nested and repeated function SQL function available in the bigquery all right so this is some it will be very mature and also you can build some dashboards to find our insights right from this data using we have already data Studio integrated with bigquery you can you can explore this data using sheets or data Studio at free of cost okay this is a great Advantage we have when you export your audit logs into bigquery Data setup table okay this is one option in the next option we do have GCS packet where do where we have all these locks exported okay go to Google Cloud Storage okay I don't think so we have logs over here it will take some time because it has been recently created go to buckets so I have exploited my locks over here this is the bucket which has been already created and I try to export some logs you can see there are so many logs exported by using DCS sync right you click on anything so we have data flow related compute right so audit logs so we are interested in audit logs right so click on this data access you can see see the complexity the hierarchy of this folder structure right so we have log name over here we have bucket log name all right and then we do have a year folder we have month folder we have date folder right so like that it will create logs so Json fan right click on this click on this thing you can see this lock oh see how big it is so if you have to manually form some analysis it will be very difficult that's why you need to export this logs to the storage services like this and then you can ah further process this locks using our data flow right or else you can use bigquery SQL to perform your analysis okay so I hope you are clear now in the third task her third case will try to export these logs into a pops up topic okay now go to the router right create a sync right so now export pops up okay so next select the service you can see here is a pops up topic so I have already few topics created we'll try to use one of the topic which is already available I don't want to create a new topic so we have this receive data topic right now keep remaining option as it is default now click on create so so that it will export this locks to a pub sub topic over here so its almost real time so it will not be done in batches it will be done in stream almost very near real time okay click on create sync then it will create the sync go to log rotor here we have this pops up so click on details again we do have the service account Associated we need to give necessary permission to export this log to a pub sub topic right now go to the iam Grant access all right go to here we do have Pub sub so just give Pub sub publisher would be enough save okay now we have the necessary permission given to this particular service account 9730-5091 okay 5 0 9 1 yeah now we have all the permission now Whenever there is some event happening in the Google platform like any transaction happening in the Google platform that related information should be our log information should be exported to this Pub sub topic because I haven't included any filter conditions so so whatever the locks are getting created should be exported to a pops up topic now what we will do we will perform some task over here either in a bigquery and then we will immediately try to go to that Pub sub topic and we'll try to print that information or that log information okay so for that what we will do so let me create some compute right virtual machine compute VM so there what what I will do I'll try to deploy my subscriber Python program so basically subscribe to this particular topic where we are trying to export these logs and then we will try to print that information okay so let us go to compute VM environment you can see I've already created one virtual machine for our demo purpose but it has been in a disabled State let me enable it so it has already this subscriber app deployed into this particular machine okay again we do not have to do it because I have already shown how to create this virtual Mission how to deploy our subscriber or publisher application into virtual machine how we can install required dependencies into this virtual machine in my previous video you can go to my previous video where we have a basic video related to Pub Subs publisher and subscriber right using our python client libraries okay now this has been enabled and running so right go to this machine sorry SSH this mission it has access to this Pub sub already click on SSH now we have this already subscriber python script okay you can see it just it has the subscription with this we have this subscription already available okay to this particular topic receive data so go to our Pub sub right this is the topic we have so go to subscription so this is a subscription okay so this subscription is subscribe to this particular topic right now we are trying to export our logs to this particular topic so we have the subscription through that subscription we are trying to receive that log information and we're trying to print using this particular Python program right so right let us try to run this program so let's see what is this painting all right this is listening you can see whatever the logs recently exported to that topic has been printed it is receiving right so but it is in a trimmed it has trimmed that message since each login is very huge it can't print that log that entire log right it can trim that and it can print that okay now you try to run this program so it will wait for there is a sleep time right mention like uh maybe 20 seconds or something then it will wait for the time and then it will come out we don't have any in a trimmed version you can't see each and every log right so that's why you need to process these logs further forever ah further analysis right we can't directly perform any analysis on top of this that we need to extract only related information which would be very helpful for for our further analysis video so thank you thank you very much for watching this video

Channel: Anjan GCP Data Engineering

Views: 8,776

Keywords: GCP, logging, export, big query, storage, PubSub

Id: N19OEDvJB4Q

Length: 20min 2sec (1202 seconds)

Published: Sat Sep 24 2022