Spring Tips: Spring Batch

hi spring band and since ominous we're going to go ahead and take a look at spring batch and particularly looking at spring batch before because I want to make sure we could cover some of the niceties that were proposed and already included in the first milestone for spring back for that oh so we're going to take a look at a simple example here today we're going to go ahead and go to the facts demo starting with the SPG neutralizer as we always do I'm going to go ahead and use spring book 1.5 rc1 not because you need to be 1.5 per se but because that'll give me the snapshot and mouse and repositories which then I can use to draw the milestone for spring match itself so it's used my sequel driver will use JDBC support here we'll use batch support as well and and that I think will do so go ahead and hit generate then I'll give it some new projects which all start a game generate a new one open this up in my IDE so what we have here is a simple spoon root application as usual in order for us to work with spring batch we need that opt in now we need to activate it so we say at normal batch processing and this sense of the auto configuration is activated on a configuration which in turn sets up all of the spring batch infrastructure because the spring batch is meant to run long-running jobs right from back to the way of modern batch processes batch processes by their very nature are long-running are staple and so you have to keep that state summer spring batch does that in the sequel database for you it will install all the requisite metadata samples for you if you want as well so it'll do that on application startup which is why the first thing that we need to do is that the point to our data source so I'm going to go ahead and just copy the properties here that I've guarded that ran around toss them and I will paste that here there's my setup my local machine and the local historian among people instance and with that I'll have a data source that might seek that a multiple data source of screen back from those who connect you into configure now swing batch is just as any other batch processing system it's very similar to the anything anything you may have worked with in the past including sort of more antiquated as mainframe kind of technology like kick and COBOL and kick so in that world there's very common jars and bracelet very common that sort of a domain knowledge that's required understand and work with batch processing which batch processing is optimized for is dealing with lots of or large amounts of sequential data so in the world of that first thing you have this idea of a job an operator typically runs the job in our case swing boot will run the job for us so it'll look for any jobs in the spring application context when this Java process is started and then just run them you can explicitly kick them off perhaps in your in response to a an event like a web request or a message arriving or something like that but the default behavior with spring boot auto-configuration is to run the job on applications then a job itself is made up of several steps or stages each step takes data from some sort of source reads it optionally then processes it and then writes it out so you you do this in as many successive steps as you need to to arrive at a result so you might take data from a file system and then and then put in a database test a basic detail kind of processing right and then you might analyze that David Davies and then process or result and then write that out to another file this is very very common sort of batch processing and is using Hadoop or : cakes or anything else like that you'll recognize the EM the basic structure of a batch processing job so the first thing we need to do here is to build a job I'm going to build a job using the you know Java configuration and the Java config API so when you use the job builders Factory ABF and the job will have multiple steps so build that using the steps of the factory SBF animal tell spring batch to our created jobs called ETL right and the child will have an increment er unless to say a unique ID will be automatically assigned each job because remember by default by default it's um it's going to turn to look at the parameters that we use to kick off the job and then use that as a way of sort of assigning it a unique primary key so when the spring batch launches a job it looks at the parameters that go into that job and it is use that and store the invaders so if we then try to run the same job with the same in but it looks like this is already running you've already run it or it's already running we can't launch again so for our purposes we're going in rating right now we'll make it just automatically ascribe assign leather and the IDS on each one okay so the first step that we need to do is is to describe a step so I will say step s 1 equals SBS got six I'm going to say we want to read Justina from the file and give us a file here that Pop's in that CSV it has names ages Amina Isis we've got only two thousand ten thousand records so the small file but it allow us to go back the beauty of spring batches that is optimized for large amounts of data so it's not going to do anything naive right it's not going to try and load it's not going to be a select all we've got the on a date still that's tomorrow mr. Kersh through the ropes example here dancer and available anything even if you have a million records to code the storage of a million records and code of for twenty good and level 16 batch has built-in support for paralyzing and defining that work as well when badge has this notion of as chunks whether it seems through the data it goes you know leave a buffer in a large bucket what is called a chunk and then it tries to write that chunk to this to the writer in a read as much as it can and write and read as much of the can then write we can tell it what size chunks we want so we're going to call this file to saving for this file to GD nice and when you tell us they want to create a a application that that reads data from our file system and turned it into a POJO of certain types on this case when i paid it for joke just called person now it's using modern language like cotton or goofy or Scala or something like this then this is one genius code generation that we're doing here is is not something you have to worry about the for our purposes we're just uh orgies in Java and I couldn't use longer long blocks that would then just fine as well might be a little bit at equal to AB data and then just a field and you have been done but whatever we're fine so we have a field here and we're going to take the data from the file come read it and turn it into records of type person and then write it out from records of that person's to the file system such as database so it on the input is personnel different because I'm going to say that when a weed records rather write records in terms of let's say 100 records their time now again you want to fine-tune this this is a a knob and the leaver for you just sort of playlists in order to arrive at the most of vacation systems the goal here is to not overwhelm your gather profit if you're reading a million records it might be finding on 1,000 records at a time another thing to consider when you simplify a chunk size is how many records are you willing to to serve marked as incorrect and move to the five write it the way that spring batch works is if it has different settings different things you can configure in specifying what happens if a particular chunk of data fails to read or parse or to do plot that's perfect if something goes wrong you know during records 999,000 to them you know all the way to the end example to the last million you want to stop and you want to fail the whole job or you want to scale that chunk you want to just note that particular uncle failed and then allow the job to succeed right in general things that you have to take into mind when you describe and build a batch job and thankfully it's easy to do that right so you can mark for example certain chunk is that if a certain chunk failed by that you should just just you should be noted in the metadata tables and an operator can come along and observe that so we're going to say chunk of size 100 and then we need to tell it how to read the data so we are looking for an item leader of some sort of credit parameter here item leader okay Excel and we're going to also write the data so I'm going to use the item writer item writer of some sort so let's create that as well okay and I'll build that now that's our first job will tell Spring bash about it and we've got a first step there we need to actually describe these item design Winer sound so let's go ahead and define a few beans now this is one of the my speech doesn't where we get to look at for a nice features and spring batch for Dino the builders to see spring batch has a very rich collection of item readers and einer meiner and you know they do all sorts of different things they can work with ultra certain things like a long XML documents neo4j result sets jdbc resultset LCIF data TPA JMS matching pet stuff etc right so in a flash file CSV files and messaging hibernate all these kinds of item readers and lighters but there are very common ones especially the domain of - like ABCs and like file system so in this first milestone which is just the mideast last week there's already support for reading and and writing their support for builder 8 guys - expressing a reader and a writer for working with Fox isms and for working with data sources among others and there are more on the way as we march towards a GME but I and somebody loves to imagine works with different years and super happy to see these builders arise because spring Bansha is a fairly old swing PARCC assessment projects go it's been around since 2006 it's co-developed by spring team and I think it's a venture and that that work was born out of you know customers who had use cases related to moving and migrating back chuckles so it's been used by some of the biggest than the best earning in the world we're trying to solve these kind of vast workload problems and so it's very full of perfect spin it's very stable but it also can also the time and the xml configuration format for for spring was prevalent and it's good the only way to go in recent years there's been a java configuration api in terms of the job builder factory in that set builder faculty approaches and our java configuration all that stuff if we added me some ears but one thing that remained very much oriented towards the xml configuration where you would specify lots of different properties on a beam or these various item rated item writers which meant that each time i want to define a matter meter by madam I'd have to say set this set that says that you know etc and it gets a little overwhelming so now it's very nice to have these convenience builders so let's see you the first thing we need is a flat file item reader of type person we're going to call us a file reader and we're going to use the flat file item reader builder and that comes in that comes in spring batch for that oh so we need to specify that we want our spring project to use that version for 900 M 1 and do the default one right so here we're overriding the default spring boot imported spring batch generation right so there we are there's our flat file item reader let's specify a few things the first thing we need to specify the name so it's going to be a file leader and the second thing when you discuss apply as the resource with what data are we reading that you want to read data from the file system a CSV file that you've seen on the console here inside we need to subsidize what that we don't see it I'm going to have it be a parameter of inputs which will be provided into the print your application right so input this is a screen for amic abstraction a resource of spring/summer core ILO resource and so as you know in spring there's all men are different resources and communications for different things so in definitions FAT FAT file system DFS URLs past we have their fun Java instance on so a lot of Java and I know rather lots of different um good options here or didn't matter which one you use you can specify that into a glass of grabbing the screen reader for you so here we've always got a a flat file item we don't want to map it to a CMS audio type person and I want to read the data that is submitted by comma and that has columns like the column bar the first name is H and the email will place first name age and renal cancer and then you should have emanated between our clinic section light office all SSI data set and such so there's our flat file as a meter that'll be able to make it provided or inject it into the screen definition here now we need be anybody writer liver jbp right if I wanna say KBC sash items liner of type person DC writer and this is before when you take advantage of the Builder instead of a lot of configuration itself of cells and so we'll think bring in the person and we're going to tell it to you the data source in this case the data sources so on the passivity book keeps figure for us based on these books five properly so I'm going to connect into my people allies it expect we're going to use a sequence and or less like research into people's at the table epidemic going to create age first name you know values H first name email and the then you need to run tell it once helped swing batches to use the field on the folder that's coming from the reader so whenever the readers going to leave the data from the fascism and turn it into a photo effects person that's going to be then passed to the writer which we going to expensive property because it focuses those slices you map in there and use as a result of primer for the sequencing idea here compared to Satan's planner so there's that we also want to provide a well I can exist assault before need for this that should be enough we're using Athena for so spring batch will create a means of travel or gbq sentence based on that data source for intrinsically although have the option to to override that you want to now we need to plug this all in and make sure it's working it's going to expect a input parameter so for a little prototyping here let's just hard code in there and conduct set properties input and then we'll commit to these users say long sex off in dot CSV let's get up this path and again this is using the special URL context for spring Windows abstraction so it's like that later on we're going to have an output CSU so you should go ahead and specify that here well there's our input notebook and we will turn those into properties that can be injected involved in the values even assign lamentation that should work that should be very simple example of course if you need to make sure that you have data that you have database so let's go ahead to our resources director here and create a new file so we call it C months at sea school and in forgot table of people if exist people we're going to configure data sources or a teacher time of interacting with our database every builder maybe the batch database graduated that bastard let's test the connection here good all right apply now we can interrogate it and goes and see let's not talk table people good and when I want to create a table that is really hit on every one of a table people ID big homes lot smaller on ice Oh and most people this is that as our first feels we want another one for the email in effect email on our two five five not know okay an Olympic age of in lemon Modena even more enough not know on the curse name on our coupon not know okay so there's our simple GDL then you drop the table on every start for this convention is even a sequins if we had option is excited specified of theta dot equal to CC to file we're going to be read by screen 1860 to be super book this is not the simplest ambassadors different convenient vistas because we can have swing install our table persons and it connects to the data store so that means everyone have to worry about that being present so let's see if everything goes to plan we should be able to run this and then visit the accomplice and so that's all from c-section work okay looks like it okay maybe something along the table back really does okay but we don't see the data reflected in the output I wonder why so I've been thinking supply and we're ordering constructor for the person poacher which is what thing is going to map to the record try sir here we go else's on incopy is not sizable Felisa spices better in France again it's about without children let's Lambada nothing like compliments calling it dead simple to get past ok let's give the remote function direct so let's see really hyper okay we've got together for age you know I do this bit has been written to the day wait that's fine now I want to see you know I want to see the distribution I want to see how many people have a certain image sizes ten thousand records so we're going to do a very simple query and then we'll write that resulting data out to another CSV file so we're going to do the same kind of thing in Reverse this is a single step you can have multiple steps and then that is the way you connect this stuff is just an X and then provide another step now of course you can also say on and then apply a pattern that matches the status of the completed previous step of a previous stage myself you've done the content here you can see but the job itself has a completed seven like on the each step also can return step Alex think that's totally expecting to use this sort of quick workflow you can do also it's really cool workflow kind of things our jobs sort of flow decisions and even the ability to provide a custom object that makes the decisions problematic right so but but for our purposes it's simple enough to just leave it like this just have a steps that one before the other another thing instead of having am a step up steps instead of only step up set the rather instead having a subset of has a leader in the writer your steps can also you can also use something called a tactic an attack that is a certain generic fall through mechanisms it's not because you can put in the middle of your clothes immer and it can be used to be something that easy strictly speaking an item leader or not mitre I also should mention that you have a third type of office here called a food centers now a processor is very simple this eventually on this case is where you won the classes in spring batch where you would provide most of the business on discern all that may be very useful with any in fact of default and imposes goes out of the box yeah all of these are just wrappers or adapters these are not similar to things or to the repo this is where you put business logic right so the contract here is that you get the person that came out of the reader and you can do something for that person and then that's what gets sent to the writers just want to do any kind of enrichment augmentation transformations anything you want you can set business logic here maybe you want to go through all the records in a database and do some sort of taxa nation women and then write the results at the Vegas so that could be done as a writer so another day looks fine in this case we don't need to climb right so we've got everything we need let's go ahead and go the second step process f2 SBS park it and then the theorem say CDs to file by sending everything before but in Reverse and what we're going to do is we're going to flip all from the people table and like a count and then we age and improve it by being sure we'll get a and mapping of age to how many people have that me now looking expresses in terms of a map of integer like will be a Kiwi will be the age and the value will be BF yeah count of that age and so we're specifying map integer integer twice chunk I'm going to do the same chunk size there bring in the map type where we are you can specify a reader and the writer 254 and this gets us into a bit of a problem doesn't it we've got an item reader in my item right up here and we haven't really been raised as a lot of type I could have found cast them to the specific type you know wouldn't do all of us do with all that much good because you know eventually in a complicated application you very well may have most instances of a flat file item reader or a BBC match item writer for example so you need types disambiguation alone you think I'm a quite fit you know you could use a qualifier you could say this is a views convention based on the names of the method or the being rather since the name of the method or the name of being okay and whether if it's a sewer step one are one that everyone's but but that old it's a kind of genius one thing I like to do is use sort of configuration classes I like to have a kind of a wrapper configuration class for each step like a public static class step one configuration and this is to say message configuration class that in turn contains two beam definitions and it's a you get a free kind of scoping for this place the benefit is that now instead of me injecting these two beams that I need directions I can say except for one configuration of one config and I can just be reckoned 15 definition so record time so this is a natural thing to do in spring or it is we're saying I want to inject the configuration class itself as IB and then I want to have access to the beans that are created from those configurations provider method but make make them stick we're not actually calling the method reforms going to give us a return value you can do the same thing you know from one beam to another if I want to have my file reader call you know the screen here I decide calls 311 there's no reason I couldn't just say this that food and then use it that feed but said being will go through the lifecycle and all the initialization callbacks and all except that spring guaranteed and it would also be a singleton so I could call this fermented this X is a thousand times and I go only ever have but a false in there like one beep so we're doing the same thing here but we're calling in sea beans so dereferencing your beans on the configuration type of cell so this is going to require a exception it costs of going to show that and you know that didn't myself scoping through a little bit of isolation so now we can define another configuration fast for step two so as configuration public static class status class step three configuration and we are going to describe first and foremost a a babyish reason so let's say item later map of in figure consider Oh KBC meters the turn music jvc cursor item meter but of course we're not going to use the course our leaders are going to use a builder to build that item students I'm going to return data in terms of its meaning a map as a tuple here so we're going to need data source of course and so spring our data source and we'll get a name so I'm going to say up to speed on the then we'll map the data that comes back so we're going to provide a new spring morning's a lot of Rome a part this is very similar if anybody ever use the GBC templates in spring very much properly in Ramsey's container to stay one like the Davis kind of further talking the contract is very simple says when the swing match component the leader may be a record into so if moving so the record is going to ask us to map a result set into something meaningful in this case a map of an in your just life so keep is always a return costume box single can map on the data is going to be a result set shot get in and so the a for the age and we'll set just in C for the challenge now this of course is a nice lambda for the Lambo okay use that we are missing something out when you sold it right after that that build that should work and should give us our data it's going to map it into that and then we need for the reader now we need to wait - did I think that was open data and then write it out to the filesystem Franco and only the sequel thing of course I can certainly the sequel statements arrived at that computer calculated for my face one count aged from our cast HP on page a some people food five ink area so that's that'll be SS Monica to the feed will be an account age a will be a force it will turn that into a map a singleton Mathis as one records one entry so is that it can suck either way in it and that's ours our reader down to the air writers of them click item writer or a map of integer compressed file lighter and same idea we're going to write out to a file confirm solicitors provide an output and if you reference our data studio office Crowder here and there was a new flat file item lighter builder own map all integer integer click build and going to provide this a name those file lighter the name value a leaf we can get these components are going to say to Davis the estate for the progress the progress for this job or which days things on the Davis so one of the checkpoints to go to look at the database until we've got so far right so we need to give a name we can use generic name to go on but it helps me just a little descriptive I'm being serious okay here and what is the file writer when the context of this job is pretty clear but give enough jobs you may want to get some more inventive we do that so we just want to that resource that we just described and we need to provide a line exit now the line aggregators job except this is to figure out how to take the input data that's coming in from the FOSDEM and turn that into a set of Records that's you know looking money or whether it's going to take the data coming in from the jbq reader in turn that into record secondly it's not due to the Foxton as our mind so it's our job to turn these three objects instead of strings alternate right but you don't have to do most of that work that you lose this delimited line I was going so we need to apply this limiter I think the default is is comma or another yeah comment anyway but never never to be explicit and then we can provide a new field extract ensign or field extractor and the contract here is clearly just going forward like even a map which is the input from the reader it turns and a an array of values that can be done using to dampen space right so let's replace that will say integer mops max and reset all in rid of that next it was only one value ever this is a static space candle and what we're going to do let's say I'm next on get key let's it's valid right so I want to write the the Vanessa column will be VL the temp the right values right column wzl rather than less value will be doing age like fine will make us account and I think not ought to do it if I leave is enough here you can so fine coding the header for example had a manager work with the header should things we deleted the XMP if it exists and all this kind of stuff should we append to the file of course of writing out for that new file so all this is such a good fly but I think for our purposes it will be fine to leave things out they are so here we go we found our step two configurations event TV series in a file writer and we can do the same damages here to configuration and you know that impacting eknoll again these don't contribute to the equation of the objects they're just telling spring to column method it doesn't care about the parameters that we pass in because it's already got that feed and so this is it's all just you don't have to provide the parameters we're going to provided whether its dependence injection container itself later on or tops earlier on in this case because they're singleton still have already been initialized alright so that I think should do it elite if now if everything goes correctly you should be able to see on the faucets in here outlook those are out Sofia let's go ahead and run this it's alright LC and there we are unless we have the age and on the right graphics the number of observed instances okay so it says here that the age 99 was you know under day time in that in that in good 5 or index.php breakfast 99 for you cm NFL under length in racket so it lines up with what would expect now this is a simple example we we another swing batch will do the right thing in terms of chunking it will go through the data and leave so much and then write out in a chunk that is that data to there could be out the writer and this is this means that they are code will work it won't it won't load to updated members more in Atlanta and all the item readers animators are geared to want to file by the contract for an item writer is to take a list of n items which is an only child size or less and then write them out matchbox them is by definition one of the things that you don't want to have to do yourself is it then the art of seeing works large amounts of data such that if there's a bug in your processing code you won't find it and so maybe our so I'm fast jobs could run for 24 hours there's no reason a batch job couldn't take hours and hours and hours or all night for some time fine so you want to use many of these sort of batteries that are included spring that as possible being young any important that we've just shown here leave the building a simple job is just the beginning right in an earlier spring tip we look at twink lab data flow you can actually orchestrate batch jobs you can clean contact on stink cloud dataflow so that is the subject for another video I think but this should give you a quick introduction quick look at swing batch what you should appreciate is an instance use the handle sort of finite workload or clothes that have a beginning in them end and a lot of data is still very much fit in that world but a lot of things are also infinite weather is a never-ending for this there's that the capability that you find in springtime data flow so if you look at when conveyance or it is a amalgamation or a mix of both the batch puts things of support here and spring match and the event-driven sort of infinite and never-ending stream of events support that you get from innovation it brings and brings together both worlds and makes it as easy as possible to describe solutions no matter what the nature of that solution all right well this has been a quick look at spring batch we'll see you in next video thanks for watching you

Channel: SpringDeveloper

Views: 58,182

Rating: 4.8916826 out of 5

Keywords: Web Development (Interest), spring, pivotal, Web Application (Industry) Web Application Framework (Software Genre), Java (Programming Language), Spring Framework, Software Developer (Project Role), Java (Software), Weblogic, IBM WebSphere Application Server (Software), IBM WebSphere (Software), WildFly (Software), JBoss (Venture Funded Company), cloud foundry, spring boot, spring cloud, spring batch

Id: x4nBNLoizOc

Length: 39min 20sec (2360 seconds)

Published: Tue Jan 31 2017