Spring Batch Dynamic File Upload Example | Spring Boot | JavaTechie

hi everyone welcome to Java techy in this tutorial we'll discuss how to upload a file dynamically in Spring batch workflow okay all right if you remember already we have discussed the spring batch complete series from the basic if anyone is not aware about it they can go to my YouTube channel and find a playlist called Spring batch so already I uploaded couple of video about the spring batch with different kind of use case so you guys can refer this particular playlist even I'll share the link in video description for your reference okay so to understand how we can perform the dynamic file upload in the spring batch let me walk you through the spring batch default flow with the code then we'll find out what's wrong on it then we'll Deep dive further to find out the solution for it okay so without any further delay let's get started thank you [Music] so to save our time I am using the existing code of the spring batch example which I have discussed in the spring bass playlist so I'll share the GitHub link so that you can directly refer it so to save our time I'm just using the existing code okay so as we know in the spring batch workflow there are the three major component if you go to the spring batch config file you will find three major component one is the reader who will read your file from the source and one is the processor who will process the data and one is the writer who will write your object or file information to the destination which is the DV for us right next if we will observe the configuration the reader writer and processor whatever we have defined we are giving to the step object now at the end to execute that particular step we are just giving that particular step object to the job build Factory okay now if you will go to the controller class I have Define one end point let me open that I have defined one end point slash input data so once you will hit this particular endpoint job launcher will launch a job by taking the job object as a parameter and any job parameter what you are giving So currently you are giving start at that is what something the time so this job parameter is nothing the key and value which you want to pass dynamically or at the wrong time if you if you want to pass something to the job then you can use the job parameters okay so this is what something the end point I have defined and the application is up and running so let me hit this endpoint and then we'll see how the record is getting inserted okay so what is the endpoint input data let me go to the postman this is what the end point so we don't have anything on the body just remove this I'll just send the request okay so we got the status quo two zero zero and if You observe in the console you will find the couple of insert statement here see here we have some insert statement I mean it read from the file by reader it process view of the record then finally it write to the DV this is the default flow of the spring batch so that already explained in my first tutorial you can refer that okay now let's verify in the DV so I will just open this particular table I have the record inserted here and there are couple of table given by the spring batch so if you want to check the job execution you will find the status as a completed if you want to check the step execution you will find the status as completed okay let me Zoom this and if you want to see any parameter what you are giving to the job at runtime so this this is what batch of execution params this is the table you can verify this so if You observe we are giving the start at as a key or this is what the object we are giving as a job parameter at runtime okay so this is where the default flow so if You observe careful in the spring batch config file as part of the reader we are just telling him hey reader please read the file from this particular directory but that is what not something we can Implement in the real time right we never know from what exact path user will try to import the file I cannot hard code the path of the file what I want to perform at as part of the spring batch so we need to tell to this reader don't take the path directly from the hard coded value take it from the file I mean when user will upload a file get that path and just process your reader logic right So to avoid that simply we can remove this path because we are not going to hard code it we want to take the file path dynamically at runtime okay so we don't need to hard code here now then who will give the file part to the reader my controller because controller is the end point I have defined the end point this particular method will receive the file uploaded by user how I can receive the file as you know there is something called multi-part file right multiple file and give the multipart file here I will just annotate I mean I just want to capture the key right so I'll just use request param and the key should be file then value should be the file particular raw binary file okay so now I got the file path now how can I give this particular file which I am giving at runtime to my job so that is how that is what the reason of using this job parameters so the way we are adding the start at here the file which will receive as part of your input just add it to the job parameter then from job parameter we can face from this reader hey Job parameter give me the file which you received okay that's it that is the simple step we need to do so I will just add the file to the job parameter so I will just do Dot add string so I'll just give a key as a anything you can give okay so I will just give full path file name pull path file name and what is the name of your file you will get from this multiple file so what I will do I will just get your string original file name equal to multipart file dot get original file name fine then I will just create a file object which I want to import so I will just create a file file to import equal to new file of original file okay this is what the file I want to import now what next I need to give the absolute path of this particular file in the value of this particular key because this is what the value I mean this was the key and this is what the corresponding value file to import dot get absolute path this you are the value I want to pass to the job parameter so that this job parameter will append this at runtime and will give to my read so that reader can read the path here rather than taking the hard code value so this makes sense right this job parameters I am just appending the file path what user is uploading that's it nothing complex okay so I'll tell you in this approach we will find out I mean we will see some exception then I will tell you how we can overcome that now since this job parameter is holding the full path file name that I need to extract in my spring batch config file right so I just need to use the value annotation to get that particular file path at runtime so what I can do here I will just Define string path to file or something like that then I will get that path to file from job parameter right so I will just use at the rate value annotation and then I have div I need to Define here the dynamic value which I will face from the job parameter so make sure to give the proper name so I will just copy this I will get the value from job parameters okay and the path is nothing whatever the key I am giving pull path file name okay so this is what the value this is what the path to my file which I want to upload and how I can get it since I am binding it to the job parameter at runtime in my controller this is where I am binding as a key and value I can fetch it from this job parameter and that is the key then simple I need to pass the new file here new file path to file okay so it will create a file and reader will read from that particular file now we are not hard coding the path of my file okay that's fine now this step will crack because in item reader we have the argument right so what we can do we will simply copy this argument flat file item return and we'll pass it to the step method okay so just go to the step and just pass it here and just use that item reader as a argument similarly If You observe this tape method is being invoked from the wrong job method so again this will cry right so here also I will pass the item later and I will pass the same item reader to the step method all good right so there is no compilation error now there is again another challenges so what we need to do as you know by default scope of each and every bin is Singleton right read or processor writer everything is the default scope as a Singleton but we need to make this reader scope as a step execution not as a Singleton so what you can do you just need to annotate at the Red State scope okay this tape scope annotation means that spring batch will use the spring container to instantiate a new instance of that component for each step execution okay this is often useful for doing parameter late binding where a parameter may be specified either at the step context or to the job execution context label and this need to be substituted for a placeholder okay the way we're binding the file at runtime so this is what the recommended approach to to use this particular at the red strip scope annotation okay so all good now let's run our code and we'll see how the file is getting imported okay so before we rerun it I'll just delete the record from DV okay so we need to verify whether the record is getting inserted or not so first let me delete it will just verify is there any record exist no all good okay so let me restart the app okay so let's wait it to complete so application started now go to the controller now to call this import data method we need to pass a file and we'll see how the flow is going on so let me call this method and I need to give the file as a key so just go to the postman let's use the same this is the input data go to the body form data and just to choose the key as a file so I'll just name it file then select the file okay I'll just choose from the download fine this is customers.csb let me run it we'll find the exception then I'll tell you what is the problem then I will also tell you how we can overcome it so let me simply run it if you will check in the message here we are getting input resource must exist okay reader is in the let me Zoom this reader is in strict mode foreign so if You observe carefully the and if You observe the this particular line number 51 we are giving full path file name file to import dot get absolute path in this particular scenario here we are just uploading a file but my multivert file is unable to identify what is the location from where the file is uploaded so that is the reason when we are getting the absolute path we are getting see this is what my project directory right from Project directory only it is just searching the customers dot CSV file but in my project directory I don't have any customer.csv file right can you see any file here in this project directory no that is what it is looking for but we are not giving that we are uploading the file from the postman dynamically it could be from my C drive or it can be from my D drive or from download location anything right we we cannot force user to load a file from a specific directory he can upload a file from different directory from his or her machine right so but in my code since we are just getting the file path from the multipad file that is the reason when we are getting the absolute path it is looking in my particular project so what we can do to overcome that first what user is uploading a file keep a clone of it in some directory or store in the DV then get its path and give it to the job parameter this is what some step you can do so Here If You observe we are getting the file right but we don't know the path of it path we don't know from where user uploaded that path we don't know right only what we get we get the file now what you can do copy the file to some storage in your VM since we are trying in our local so I can create in some directory in my in any folder but if you are implementing it in the real time make sure to create a path in your virtual machine where your application is hosted okay if not copy the file to DB then get the file path or similarly copy the file to some storage in your BM get the file path and give that path to this particular place do not give the absolute path get absolute path of the file what you are importing but multi but because multipart file don't know the path from where user uploaded it okay this is the simple Funda so what I can do I will just create a simple folder in my desktop so okay I have something called temp right or I will just create some different one let me remove this new folder let's say um files this is my folder I'll just get a path of it okay so what I'll do I'll make this as a root directory so for that I will simply Define that path as a root directory so I will just create a variable here private uh string I will just declare it final okay final string Temp Storage okay any name you can give 10 storage equal to this fine now I will get the original file from this multiple file but that original file which is the original file name I want to copy it to the Temp Storage so I rather than Define the file name I will just give the path where I want to move that file so I'll just give the Temp Storage okay then I will also add a slice here that makes sense okay so this I want to create a file in this time storage folder now copy the file from the multipart file to the location okay so simply you can use multiple files or multiple file dot transport to the destination destination is the file which I want to import okay so just add the exception fine now here also I don't need to give absolute path because I know where my file will be placed right this is the place where my file will be stored let me copy this and just paste it simple the logic is very simple from the multibod file get the file then create a file object to the root path or the temporary path whatever you define then just transfer the multivert file object to the this this is the destination copy that file to this destination okay what is the destination this is where the destination and then I am just giving the destination details to the full path file name as a value that's it okay now the file will be stored in this particular location right which we just created so go to the desktop this is where it will store I am just giving the path of it and I am just giving the file name now you might be assuming that hey if we'll do such kind of implementation then my BM will be unnecessarily occupied with lot of memory right so you can validate some logic here just you have the execution object with you right you can do something like this if execution dot get status okay Dot um get the code of it or something like that get or you can just do the equals okay or simply you can just do rather than get the status get the exit status okay e x i t yeah exit status then get the code of it which will give you either completed or failed dot equals okay so we can just valid here exit status dot completed then delete the file from that folder delete the file from the 10 storage right again this will increase your lot of i o operation so that you need to think before Implement whether you will keep it in your DB or you will find out some stories where in that particular VM you will not do more i o operation if you do also there will be not impact to your application okay so you can find out such kind of scenario so here you can simply do files okay delete if exist or something like that and give the path what is the path storage path and the file name that that's it right so you can pass this this is a path right so better you can not don't pass like this you can just do paths dot get and give the location you can do like this if the status is completed if not just do the log analysis why the file is not parsed correctly so this is additional step you can add it I am not going to do this now so we just append the correct file path to the job parameter now from the job parameter we are fetching in our reader OK not here go to the reader this is the first bin yeah here we are just fetching from job parameter now all good let me rerun the code so application started will verifying the DB once again select star from the table nothing is there and if you will check the job execution parameter we have started then second time when we got the exception we are giving full path file name but it is taking our project directory right that is what the wrong that is the reason I are getting the exception and again started let's see whether the full file file path is giving the path where we are storing the file or not so simply go to the postman this is the request is ready just send the request can you see here lot of insert statement I mean the file which you upload my reader is able to read that particular file and it is able to process it but before it come to the reader what we did we just copy the file to the different folder on my machine what is this bus batch files and if you open this you can open this directly from this tool and this is what the file my reader will read because that is what we are setting in the job parameters okay and reader will insert the record let's verify it now we can see the record here now if you'll check the job parameters can you see here the pull uh path file name I'll copy it here okay for better visibility this is what the path we are copying the file uploaded by the user that is how this path we are giving to the reader reader is happily read able to read from this particular directory there is no confusion but in real time find out the VM where if you do n number of i o operation it will not impact to your application so based on that you can design eyes notice few of my project we are already storing into the DV and we are converting it to the byte array then we are processing it but in some of the scenario I saw we just make a clone copy of the imported file in the different location then we are processing it the way we just did now okay so the approach can be vary based on your use case or as per your requirement you need to choose the correct decision okay so that's it about the dynamic file upload using the spring batch so do let me know in a comment section if you have any doubts that's all about this particular video guys thanks for watching this video meet you soon with A New Concept

Channel: Java Techie

Views: 26,692

Keywords:

Id: huUdmP5iPM8

Length: 24min 48sec (1488 seconds)

Published: Fri Jan 06 2023