AWS Step Functions: JSONPath data processing

alright hey everybody its Rob thanks for those of you who stuck around from Eric's stream I am just watching to make sure that we get started here and it looks like that's me so if you're joining us just now thanks for coming if you're sticking around from happy little api's thanks for staying again today this is AWS step functions and we're gonna be covering JSON path data processing so I'm gonna talk a little bit about JSON path and where it came from why we need it what we do with it how we use it in AWS step functions I'm going to talk about how it impacts your input processing both input to your state machine execution itself and also the input to each of those individual states as you go through I'm gonna talk a little bit about output processing how information is returned from individual States and then how you pass that information on to the next state we're gonna talk a little bit about the context object if you were on App 20-25 last Thursday we talked about this in the context of long-running workflows and the callback pattern or the wait for task token parameter that comes from context we're gonna talk about handling nulls because null is a special value in the pass and then we're gonna build examples of all of these this is a little more I'm gonna say this this is a little more in-depth and it can be a little more abstract than some of the other things we've covered so I'm gonna have some images here don't worry it's not a slideshow like we are gonna build but I think it can just help to see some simple cases before we go right into building so some housekeeping things as always if you hit Docs in the chat you'll get that link for the docs if you hit links you'll get a whole bunch of useful links I hope you find them useful and if you hit repo or github you'll get the link to the github repo for this for you to star or clone hello or hoonjae Dhokla thanks for joining us for the first time welcome to the AWS channel welcome to AWS step functions and hello twitch Ganesh yourself so without further ado let's get into it and before we get busy with the console we could talk about paths and day some paths right so two of the links that I gave you are a JSON path evaluator which just works out to JSON path calm and this is a place where you can take some sample input of your own and input some expressions up here and you can test and see what the results are so that's what that JSON path evaluator is if you want to see what's kind of the canonical definition of this that's the other link for JSON path definition and it's if you're familiar with XPath from the good old days of XML it's a way of describing parsing your JSON data structure so it's a way of bringing structure to what is essentially an unstructured format I'll leave I'm not gonna like work through this all the time right but it's essentially your dollar sign starts at your top-level object and then you use dot notation to walk through in in a format that will be familiar to you if you're familiar with JavaScript or Java so you can use brackets for arrays you use dots for maps and sort of sub objects right and we'll go over some examples of all this so don't worry let's see here input processing I said we'd talk about let me throw this up here so if you look on the left if you pass that input or no input at all to a step functions workflow execution then the state gets defined as an empty JSON object is what you see on the right similarly if you pass in a JSON object with one key name and a value of world then your state looks like that on the right as well you have a JSON object with one name and a value of world and in this example if somewhere in a parameter when you're passing it to a state you reference that top-level dot name key you'll get the value of world and this is pretty straightforward I think is hopefully what you would expect to see from notation like this so let's go right into building this let's see where is my code and we are going to create a new step function state machine every time a demo I get that output that's okay hold on y'all I'm sorry I got a thing I gotta change my screen sizes here you go away you come back thank you alright so what we're gonna do here is just get a single state that's a passed state right and there's there's like nothing interesting about this at all right so we'll create one I've already pre-staged the role for this in the AWS management console under I am will name this our JSON path example and you can see we published this we go over here we have this will start an execution I'm gonna set this up on the right let me close this and as promised we'll just pass it an empty object right so you see we're not doing anything with data here in our definition we're not adding anything in the result state this is pretty much our plain vanilla example so from here we need to go back here to the step functions console and we see we had one successful execution of our state machine here that's good it should be successful when it's not doing anything and we look and we see the input is as we'd expect and the output is as we'd expect right so this is sort of the the base case the trivial zero case twitch Ganesh missed missed happy little ap eyes yes definitely catch up on that one it's how to build the URL shortener that you see me using with all of my links here so you'll definitely want to catch that wizardry later anytime I'm not doing anything I am successful I write my best code when I'm not writing code so I feel you and then if we were to take this example that we gave above and you can probably guess where I'm gonna go with this eventually but if we were to pass this through we go back to the console and we see okay no I didn't execute it did I did I execute it y'all I don't even remember let's see let's go to the console and see if I didn't we'll come back like a squirrel you know yeah I did great we got two executions here and again as advertised in that image right we pass in as input to the entire workflow this single key value pair name in the world and it's carried through because we didn't define anything in that past step all right so now we're gonna make this a little more complicated we're gonna add another task here and it's a a WS lambda function that I've already built and this function exists for the sole purpose of consuming parameters and input and returning output so that you can see how it changes the state and in fact I'll show you the code real quick as I do for all most all of my examples I've written it and go it's a very simple lambda function and go this is a convention I like to do and I know I'm gonna be working with step functions I define my input types explicitly and I define my output types explicitly the go runtime will handle serializing and deserializing your types for you with appropriate hints here right so you got a public member name it's gonna look for that key in the JSON and off you go so all it does is it receives some input and it returns an output in an error in this case we're gonna log that input because we know that we can live tail or logs or get our log information from cloud watch logs so we can see what's happening with the state as we pass it in and we're just going to return a little manipulation of the output right so we get this name key the value of world more-az as I said I know you're going to know where this is going we append or we insert that in this string and return that as our output with the key message so this is all already deployed here but we need to link it up so let's go well first things first let's go to our definition let's give ourselves a little room to work here little room to breathe close that up so here's our definition now we're gonna add a lambda task state right so lambda task state array invoke lambda function will make this be next invoke lambda function array task we need to give it the AR in here and you'll see that all we did was give it this payload where payload will have one key named input and the value will be the entire state now that's not exactly how we've set up our input in the function but we'll get to that okay and then of course here quotes matter and true is anybody hit me up in the chat if you're not familiar with step functions - at least this level we're not really gonna get more complicated with task states or choice states or things like that a little bit on the past State because I do love the past state it's more useful than you might think but other than this this should be pretty clear for everybody right here again we're gonna go ahead and update our example here successfully updated that's great that was kind of silly because I didn't put that AR n in there but we caught it before we executed it so let's go back over here let's look at happy little compute sir sorry Eric for stealing your happy little thing you know what I mean twitch ganesh happy so far super thanks for keeping me honest and we'll go in here and just grab this AR n conveniently a little copy button there Thank You AWS lambda console team for making my life easier appreciate that and then we're gonna drop this in here notice it's not here alright this is the resource that resource is the lamda invoke service the function name is where we're putting our AR n so if you haven't used it before that's where that's gonna go we'll update this one more time with a quick update and now we can governments minimize this in fact let's not minimize that let me show you something them a little my good friend and colleague Eric Johnson likes to show me let's say I already got my profile set it's a little bit of Sam logs and you can do a little bit of Sam logs hopefully I let's see name hello world function that's not gonna be right we got to go back here and look at our template real quick I called this the Jason path lambda function that's a good name that's a good name for this lambda function since it's the JSON path episode and it is a lambda function so we're gonna call it that the stack name is Jason path lambda that's another good name that's a good name for a stack and that - tail on the end means that we're gonna get a live tail of execution of our lambda functions so big props to Eric for showing me how to do this big negative props for me for let's see what did I not do trace back look Lynch point I've done something here to myself pragma no cover or the specified log group does not exist that means I did something silly with my parameters there so let's check cloud formation make sure I get that stack name right jason path lambda you know logical resources jason path lambda function okay well that seems right jason path lambda function jason path - lambda all right let's sit tight on this for a second I think I know what this is I don't I haven't executed this lambda function yet so there grant Lamy getting it right in the chat grant you are absolutely correct it has to end there's Eric invoke once first okay in all fairness Eric got there first but I mean like he's sessions with Sam guy so he should have so I'm gonna reward these five points to grant lamby thank you Eric you can have five points - all right so did I invoke this or not y'all it's there's so many moving parts here let's see reload no I did not so as we did before start this execution we'll give ourselves a little bit of name that's boring twitch do some hello twitch goodness execution started array all right third execution hurray things going to plan succeeded and now let's see what we got here pass input name twitch so far so good output name twitch well we didn't change anything so as expected right invoke lambda function whoa alright so a couple things to notice on this our input is the output from the previous state as it should be right that makes sense that's uh that's logical there but what happened to our name in the input it's not here anymore right and so that's the first thing to notice the reason that happened is because if you don't specify a result path in a state then step functions implicitly adds the dollar sign or the route or the entire state object as the result path so when you look at result paths coming out of operations they are destructive merges so what that means is if you have a field name and you return a field and you assign it to address then you'll have the merge of those into one single map right you'll have a top-level name and a top-level address and I'll show you this in a slide in a second but we didn't do that what we did was we specified nothing so it took the entire results of this lambda function invocation including our payload and put it at the top level overriding what was already there now the second thing that you're going to pick up on we didn't get twitch in our message hello if you remember this is why where I told you that the structure of our input didn't match the structure that we looked for in our lambda function so let's fix these two things together first let's talk about this default output this is basically what I just described to you we started with this name hello world is that actually incorrect that is actually incorrect so I'm gonna hide this slide because it's wrong what we just did was we imply implicitly took the top-level path for our result path and overrode it so that's a bad slide and it's bad and I should feel bad and I do the first thing to fix here then is to change this in our step function definition by adding that result path right so and we see here from the tooltip right let me see if I can not override this a path it's it's got a pretty good explanation here a path that specifies where to place the result relative to the raw input if the raw input has a field at the location addressed by the result path value then in the output that field is discarded and overwritten by the state's result otherwise a new field is created with intervening fields constructed as necessary so I call this a destructive merge maybe that's a real word maybe I made it up you know what I'm gonna I'm gonna claim it if somebody could file a trademark application for me on that I'd appreciate it but in our result path let's just call it dollar sign dot lambda output this is gonna take that out my first patent that's right yeah a method for determining the naming of cool things in step functions an application by Rob Sutter so we're just gonna take this result path we're gonna update this we don't need to redeploy Sam or anything like that tada now we're just going to put all of that data into the lambda output field so what we expect to see is the name field be carried through and a lambda output key be created it's a map with all of that stuff that we just saw below it so this is fix one right so let's go over here did that save my data start execution it does not that's okay that's okay yo name twitch patents will start that execution so this I believe is execution number four yes and again pass output twitch patents and input same thing here input twitch patents as we'd expect and now we see that name is carried in its carried over forward to the next state and we get all of this stuff right this is all the metadata from that service integration which is what this is it's invoking the AWS lambda service and this is all of the information that's returned from it we're gonna deal with this in a minute but that's good and now we can move on to we wanted to see twitch patents here right so we need to go back and fix our function a little what's important is to look at our definition and the way we're passing parameters which is this payload this is essentially the object that we're passing in when we're invoking our function and again the structure is it has a key named input that contains the entire state so that is a key named input with an object with a key named name with a value that's twich patents that's not what we wanted here what we wanted was just a top-level we wanted an object with a top-level key named name so and then from that we pull the value here alright so all we need to do here is that's actually not the right way to do this this now this is a little redundant because just like the result path has dollar sign as the default the input or parameters also have a dollar sign as a default but I'm gonna put this up here so that you can see what it does when it executes also I had promised you some logs earlier so let's see yeah there we go so you see we've got some input here and there's a clue in here in our logs right the value of input is not what we expected because the runtime couldn't deserialize that object into the type that we declared big reason for using type safe language is by the way big fan over here so now we'll see what happens when we invoke after we update successfully updated array will start an execution I already had one over here and our top-level object is name and party people all right cuz everybody likes party people before we do this we'll go over here to our live tail it's gonna take me back execution started there's a little bit of a delay here right because this has to go up it has to execute it writes its logs to CloudWatch logs this isn't meant to be like a real-time messaging system but if we stick around here have a little bit of your favorite warm beverage will see that received input should now be deserialized properly it's decaf coffee by the way I could get too crazy on the streams you know what I mean I'm tired of waiting for this let's go over and see what happened we'll come back to this so this should be execution number 5 execution number 5 failed why let's see what we did to ourselves that's why we weren't getting logs either by the way so alright output party people yeah input party people we won't get output there we'll get this unrecognized token dollar sign was expecting true false or null so again what did I do to myself here I see it just gonna publish this real quick we're just gonna pull that out and let it take the default itself another way that you can do this is giving it the payload with extracting the actual key that you want and putting it in there and I would argue that's a better pattern so we're gonna fix it to that after this I'm very much I asked for exactly what you want and get exactly what you asked for model of programming but I still think it's important to understand the default behavior all right so not so party people now and now we has a sad right because our our function is taking a little too long to get where we wanted it to get to and while you're over there we'll go see if execution number six did what we wanted it to hooray we like green so it succeeded and we can see in this state the input was not so party people the name was there oh well I've done something the payload returned was empty so I'm gonna guess what happened was that it didn't actually pass him iddp his diet dr. pepper is always cold edy J geek JavaScript is a type of language it is a type of language but it is not a strongly typed language and there's an important difference and I am glad that you're dying dr. pepper is always cold but my decaf coffee is always warm so let's go back over here we'll look at our live tale and we see based on the timing here 1822 yeah that's right now received an input that's empty yeah yeah yep so let's go back here don't you know and let's be explicit about what we want to get it what did I say we were gonna give it Jason name hey no what was it expecting I forget this ought to be interesting oh we got to give it the dot dollar sign hello remote thanks for joining us this name dot dollar sign lets us know that it's dynamic and it's being pulled from somewhere in the state so we'll do the same thing here this if you remember if we go back and look at the input to our entire state machine we have this top-level dot name field and we're just gonna pass that in it expects it exactly in that format and this means this is a dynamic field named name that we're making available to the run time and populate that key with the value of a top-level item named name all right and then we'll publish this one again oops I'm gonna need that sidebar back successfully updated start execution I'm starting to accumulate these accumulators let's go with whatever comes to me you know what I mean get rid of these so again down here we hope to see something that says received input accumulators here oh hey that look at that see again further validating my theory of asked for exactly what you want and get exactly what you asked for an important point that I want to note here this sort of messing with the input and the visualization and all of that is easier on standard workflows because it allows you to graphically click into each one I still say that unless you need the features of standard workflows you should be using Express workflows as your default but when you're developing or playing around to see what happens you want to get a better understanding that's a good time to be using these so we'll go back in here again we've proven to ourselves enough time now that the input and output of that pass state are the same but now we see that we've fixed our other problem here hello accumulators hooray how easy to change from standard to Express so easy is always relative there's a couple things to be aware of don't go just changing in production if you're using a framework like the AWS server this application model or Sam then you just need to add a single property to your workflow definition that is uh is it type I'm blanking on the property it's in the docs that's Express all capital letters if you're doing it in the console here you're creating a new state machine then you're creating Express right here again I suggest you're setting you set up logging there's an earlier episode where I show you how to configure login as well and also an app 20:25 we talked about this and why question him from Shenton froid standard allows long workflows eg when I'm waiting for a human approval that's correct Express has very short lifetimes that is also correct Express workflows can execute for up to five minutes so you want these to be short running fully automated tasks maybe you've got an API call to a downstream service provider that can sometimes take a while and you have to do retries as long as you know that you can get your retry logic done inside of five minutes and a complete execution of your workflow you know it's still default to X to Express workflows why first the rate of invocation is so much higher it's a hundred thousand per second implications so for your automated processes you can really blow those out quickly and it tends to be cheaper it's a pricing model that's more like the way that AWS lambda is priced where you have a number of indications and the number of resources that they consume rather than state transitions so good questions from you both let me drop a link in here for you app 20/20 we have two episodes dedicated to Express workflows and standard workflows that are both deep dives into those so really recommend you check those out if you're interested if you do nothing out of the box you're getting standard workflows because they were there first all right so I'm gonna cancel doing this and now let's go back and see if I have a correct slide where we set this result path output and this is what we just did thanks thanks thank you both so this is what we've done here right you see we have that input that's similar our output is coming from our function and we've told it where to put it lambda output in this case instead of output and then we see that that's actually the result that we get of this execution right is the name and the lambda output now what we're really concerned with here is the payload and the message we're not concerned for our purposes with all this other stuff maybe you are but a good example of when you don't want to deal with all of this is when you have the dynamic parallelism state right and you have a lot of concurrent executions and you're starting to run into that limitation of state size because your state can only be 32,768 characters so you don't want for each of your objects to be carrying around all of this information in the state right really all you're concerned with is this so there's a couple ways you can deal with that one is my buddy the past state another is output path so let's look at an example of the output path real quick here we start with that same input name world and we process it through our function which returns that message hello world and we put that inside output this is kind of a contrived example here just think of the information had in our console and that's the example I'm gonna show you if you that's the first thing that happens so it takes your input state and it takes the output of your task or your state and puts it into result path wherever you specify it using that destructive merge the next thing that it does is looks for an output path and just like input path and result path output path if not provided defaults to quote dollar sign quote or the top level object meaning take everything forward output path is what you're taking from this state into the next state as input so it's the output of the state so in this case if we want to cut all of this down to just the payload that we have we can set the output path to be lambda output payload and this will make more sense if you're looking at what I'm looking at so here if we want to cut this down and only take this payload forward we can do that I'm gonna do this real quick and then we're gonna undo this so that I can show you another way so we've got output path again the tooltip helps helps us out here helps us know what we're talking about a path that selects a portion of the state's input to be passed to the state's output and again if it if emitted it has the value dollar sign which designates the entire input so again major shout-out to the AWS step functions team for building this integration with visual studio code beautiful stuff beautiful stuff so in our case we said we wanted to define that output dot payload all right and like everything else we've gotta add this here and this is our end State which is kind of silly because you don't necessarily want to munge your output before passing it on to something else but whatever publish the step functions will update ourselves here give ourselves another invocation name out put patters it's like Panthers but misspelled but actually I spelled it that way on purpose edie J geek agreed I don't know with what but I appreciate the support and then we got our execution started so now this is execution 8 again you're probably bored of seeing this one but again just to show you that it's a new execution output path errs and that output we have the implied output path of dollar sign so it takes everything forward into the next state which is why we get an input here but this time compared to our previous one we see that we've whittled down that entire object that we received with the result path inclusion to only include the lambda output dot payload object or key which consists of in a map of a single key message to a value of hello output Panthers so this is the type of stuff maybe with an execution ID or something like that that you would want to return in that dynamic parallelism state to minimize the amount of state that you're pulling through the application as you go all right now again I'm gonna back this out and show you another thing that I like to do especially when I'm developing the workflows I'm gonna get rid of this output path it is not my friend yeah I mean it is it's a great say there's no problems with it but it's just it's not the way I work again I tend to be overly verbose as you've all probably figured out from watching this series and also really explicit like I want to ask for what I want to get exactly what I asked for and so what I'm gonna do is call this a manipulator no jokes please don't appreciate them little auto-completion there that's good stuff a past state fun fact you can use a past state to just model your workflow make this end and true and we talked about this in app 2025 as well as when we introduced the toolkit for visual studio code you can use a pass - just model your workflow for those conversations with the application owner or the business process owner to say is this actually what you think is supposed to happen in this process and then once you get agreement you go and build out the individual components within it that's one thing another thing you can do with it is mock data when you're at the early stage and you want to test passing that data to and from your function my favorite thing to do with it is you can manipulate the state of your machine because you have a result here but you also have parameters let me give you a little caution here you might think that this is what you want to do to get some item out of state right so you might think that what you want to do is what we had a message in there right let's just say we want to translate this to be a top-level message and we'll call it lambda output dot payload dot message this is this is wrong so if you're watching this video don't do this as the solution to your problem what I'm about to do I'm gonna show you why this doesn't work what this is gonna give you is this as an actual quoted string value but I want to prove it to you never take my word for anything I'm pretty sure edj geek will agree with that one as well always trust but verify you know what I mean so name Barrett fires we start our execution array it's number nine number nine and come over here to this past state and we see that his input we got that whole big thing that we wanted and what we want is to get the thought lambda output dot payload dot me I knew you'd agree with me buddy dot message right so what we want is to get this hello verifiers as our message but again what we're actually going to get here is this result that's string quoted not what we want and then verify again that's right then verify again especially if it's me telling you so how do we actually do this the answer is right here in our lambda function a pass state takes parameters as well the parameters are like a special type of input path or a special type of input object so you can pass parameters into your passed state and manipulate them there and then the passed end state gets returned to your result path I'm gonna remove this because we let's leave it let's call it pass result so that we this is going to be non-destructive then so that whatever we assemble and pass in will then come out into this and we can prove to ourselves that we left everything else alone right so it's not actually the result state that we want what we want are these parameters and we want these parameters to be something like you can you can add I don't know date you can add hard-coded values if you're trying to construct an object for a test or something like that it's 20 2004 28 I don't know it was 18 3800 Zulu that's my made-up date format right there don't use that at home kids and then similarly you can define variable parameters here and this is where we want to go with our existing parameters and so now this is gonna pass in an object with two properties or two it's a map with two keys and two sets of values because it's a passed state it's not doing anything to them other than returning them and it's going to return them to this result path that we specify a passed result and while this updates why would you do this especially for IT automation workflows you may do things with service integrations or lambda function calls and just take the output of that execution back into a map in your state and you may only want bits and pieces of that you can use those to construct the properly shaped inputs for other API calls and service integrations we talked about service integrations last week you can use them for pushing messages into sqs queues or publishing to SNS topics other types of integrations that you may want to do this allows you to shape and structure your data the way that you want Chintan dot ford you have to write message that star else it won't evaluate the dynamic part on the right and it will eval the literal dollar sign whatever is a string that is correct if you don't give it and this is in those let me get the links out for you or the docs actually it's in that Doc's link under data if you don't end those your keys with a dot dollar sign then it doesn't know that it is processing a variable so that is an important point to point out I'm gonna put forgetful in here because again I don't remember if I executed this or not we'll see if this was execution number 10 or number 11 I don't think I did yeah number 10 good so we get here and we've got all this output and what we want is to grab this out of here and some hard-coded field that we have and we said we were going to store it in result path so that we can prove that it's non-destructive right but so here we have this past result then this is that static constant string that we hard-coded in there remember date had no dollar sign message had a dot dollar sign and parsed down through here hello forgetful so out of all of this we've constructed another object that we can use as input to an API call or our next step and again just as a last step if we return here and modify this we remove the result path we're gonna get that destructive overwrite we can run this one more time give it a fun name like destroyer happy times happy times in the AWS step functions episode destroy you're on the loose y'all eleventh execution and now if we go down here to our output we see that we've taken the output from that previous state and without any other executions or tasks we have manipulated it mutated it to the form that we want to pass into a future state right so cool fun stuff there's one more thing that I want to talk about here nulls null is not unspecified right for the Amazon States language you have to spell out null just as you would true for a boolean boolean excuse me so in this case what happens with that it's gonna be interesting and just yeah ignore that in this case what happens with that result your output of message helloworld that object when your result path is null it's discarded so your final state is the state that you began with or that you entered that state with boolean I mean like the little cubes I don't know I don't know what I did to bring maybe I said bullion you never know on this show but let's go back gold bullion not bouillon oh sorry is it a soup yes yeah boolean did I say boy bullion it's hard you know the mouth gets dry I'm going on don't worry about don't worry I'm going on so if you do result path null that's actually gonna throw an error how I'll go ahead and prove this to you I like proving stuff quick update this blah blah there was an error updating state machine if we go into the toolkit logs you will see value is not a reference path reference path didn't start with dollar sign right so when you're giving a reference path it has to begin with the dollar sign that's because we gave it a string but just like a boolean null is provided unquoted yes I did says edj geek well you know that's not the last mistake I'm gonna make today I hate to inform you but I'm aware of it so you know we'll be fine and then if I scroll down here you'll see that we successfully updated the state machine so now if I bring I had this hidden the whole time this was like a very Eric Johnson move of me yeah switch screen that's right I'm just kiddin buddy so what I did was update result path here let me show you what I showed in the logs this was all hidden I originally had null up here in quotes and that we tried to update the state machine and we get failed to update value is not a reference path reference path didn't start with dollar sign at state's invoke lambda function result path so it also helps us figure out where the problem is right so we're it states invoke lambda function result path now we change this back to an unquoted null just looking around to make sure this is actually being streamed yes we change this back to an uncoded null and we publish it and now it's a successful publication so that's hooray here we go okay so at this point we can start this execution with named don't hide the screen Rob and we go back to our console and see what we get and this execution 12 fails the reason why it fails so let me show you name don't hide the screen and Rob and this was in the invoked lambda function where we made this change right you'll notice here this is exactly what we expected to see from this screen we got that output or the message would be hello don't hide the screen Rob but because we gave it a result path of null it was discarded and we wound up with the final state that exactly matched our input state so it gave us what we what we asked for as the input to this state but and you can see this on the execution failure down here our past state no longer has a lambda out dot payload message field to manipulate where it's doing its state mutations so we can either remove that past state or correct the way that we've shaped that data in the return but that was expected and intentional so if we go back over here to get ourselves back to one good place put our result path back to lambda output a manipulator still in place we publish our update start and a start an execution name happy path an execution started and now lucky number 13 failed that's great let's see what I did this time huh did I forget to publish let's see anybody watching closely enough to catch that invoke lambda function oh who sees it who sees it two things happened one I made a mistake too I ignored the error updating the state machine and just went on as if everything was fine so I executed lucky number 13 was exactly the same as lucky number 12 reference paths must begin with a dollar sign this is also told to us here reference path is a path beginning with dollar sign used to identify components so that was my fault that was my fault I mean it's all my fault because I'm the one doing all the work right but that was especially my fault this time we'll go a little slower and look for the confirmation that we wanted there start execution name how about now execution started and now I guess lucky number 14 maybe succeeded so we're back to green right and because we've done this manipulation at the end we're back to this final structure that we wanted so we got 10 minutes there's one other thing I want to show you but I want to stop real quick and just see are there questions on what we've done here questions on input path output input path result path output path I'd encourage you to think of them in that order because I think you will use output path the least often or questions about manipulating state in a passed state no cool arrays lists collections however you want to refer to them they exist so what I want to do here which condition up for now I need to try it out and learn more yes always always always go build I will push up each of these examples that we did as individual state machine definitions to the repo some point later today but definitely go make sure you go check that repo out star that repo subscribe they or follow the AWS channel here follow me on my channel here it's good stuff over in app 2025 but the most important of all of them is to go build so go build when we're done here but I want to show you one more thing let's let's give ourselves a new state machine and again we're gonna start with this give ourselves a little bit of room here we'll have one pass state and this is uh this is an example that you can find in the docs but I find it helps to talk about it this is just going to handle a raise so we'll publish this two step functions that roll array handler enter successfully created that's great so remember we talked about in pass States you have parameters that come in as the input to the state it's not just pass States but that's what we're dealing with here so we're gonna say parameters first item dollar sign important is going to be equal to items will pass in a list 0 i say sub i don't know they're nice weird when you say sub 0 because then it's like mortal kombat please don't sue me capcom and whatever owns the right to to derivative work and then you can do things like last three items because it supports that : notation right so what we're doing here is we take a list and the first item goes into this state as first item and then the last three items go in as last three items and then they'll get returned destructively at the root because we haven't specified a result path right so if we update this and then we invoke it where am i where am I reload all right Handler we called it items right items we'll make it easy for ourselves one two three four five six seven eight nine ten so we'll pass it ten items yes it's zero based arrays so I need to know this syntax for step functions but VTL for api gateway on yes yes unfortunately that Amazon States language is its own thing we pass in this execution execution started there are priority examples for Sub Zero P V loft people off PB loft thank you good looking out appreciate you keeping me from being sued because let's be honest nobody wants to be on the end of that state machines alright that's good we have a succeeded execution we like those and let's see what we have here on our state so in our input yeah that's great it's a list of ten items so what we would expect to see is our first item is one in our last three items will be eight nine ten and in fact ooh well I you know and so what happens when you try to like push stuff in at the end of the stream right I used the wrong syntax I brought that on myself Chintan for it yeah yeah you know what happened here three not the beginning and then three but three from the end that's the syntax for three from the end this attacks has a name if somebody knows the name of this let me know it might be the like like I've seen it with cons lists but I don't think it's like a constant ax I don't know you know I don't know let's let's publish this let's update it real quick so I can prove to you that things actually work this time items where we want 11 12 13 14 15 that's enough to make our point so we expect to see 11 as the first item 13 14 15 as the last three did that start I tried to click the button also at least in this case yeah it started go to output what I asked for was not the last three items what I asked for was all the items from 3 onward so again this is this is like a syntax that you need to be careful with and know what you're doing which I obviously do not today is it considered somebody and off play that's right that's right there are only two hard problems in computer science cache invalidation naming and off-by-one errors let's look I know there's a solution for this so let's read the doc so there it is fouls 3 : last three dot vowels 3 : we expect to see 30 40 50 well that's what we gave it so that's interesting because I tested this the other day and I swear it worked all I know that's not the right syntax put a minus somewhere yeah that's like my general like I couldn't do like a Clea arrow or something who knows I like that's the syntax that I think it should be because the that means 3 from the end and this means starting from the beginning so I don't want to get too hung up on this there is a syntax out there that will let you know that may even be in the JSON path specification but the point is that you can carve up a raise that way but definitely be careful what you're doing and in fact the takeaway from all this might be better to carve up the array inside your function or somewhere else if you can do it -3 if like Python I'm not sure if it's like Python but I got three minutes Johan so I will try it - 3 : all right - 3 : power to the people quick update shooting for it is Python I mean his message is the above is Python I'm not calling you Python which if you're a Python developer would probably not mean anything anyway so let's see uh calamy items I could cheat and just make it like 5 long I'm not gonna do that all right 1 2 3 4 5 6 7 8 9 10 and this will probably be our last execution of the day so let's see if we can get negative 3 to work input as expected output last 3 items 8 9 10 y'all are the best you are the best thank you so much so AWS wishlist default to previous JSON input I know you have my contact information please get with me separately to to discuss that because I'm not we can talk about here for executions - oh yes in the extension keep that input up there also on my wish list so I will +1 my own wish list item for you so you don't have to type in the sample all the time agreed I get it I want to thank you all for joining today I hope this was useful to you I hope that you have a better understanding of how state is passed around both inside the state machine and back and forth between your functions or other integrations that you have I hope you understand when to use input path result path and output path some of the limitations that we have around them and I hope you have fun and again I'm on my channel which is up there on the screen the same time but on Thursdays this week we'll be going over Amazon sqs for building resilience into app 2025 fantastic service over 15 years old so stable you want to come learn that but otherwise I'll see you here next week all right so thanks everybody have a good one

Channel: Serverless Land

Views: 8,065

Keywords:

Id: QpZ6IdKvOdw

Length: 60min 15sec (3615 seconds)

Published: Tue Apr 28 2020