Jedi Values in Akka Streams | Rock the JVM

hello boys and girls this is Daniel for rock the jvm and in this video I'm going to talk about acha streams and a particular feature of Akko streams that I like to call GI values so this video is for the skull programmer who has the very very very basic familiarity with acha streams in the sense that you can use them already made components and glue them together and start a stream but otherwise no knowledge of behind-the-scenes functionality in Akko streams as required if you've never used aqua streams I'll give a brief overview in this video as before I'll recommend that you code alongside me whenever you need to refer back to the concept of materialized values so this is the official name of the concept just refer back to this video and for your convenience this video is also available in written form at rakta jvm comm /blog alright so I'm back in my code editor where I'm going to share some insights so akka streams is an implementation of what is called reactive streams this is a specification that allows high throughput and fault tolerant streams of data by simply plugging in streaming components and aqua streams is the implementation of the specification for the JVM so the components that we can use our sources sinks and flows I'm gonna share some code here shortly so the names are pretty self-explanatory sources are producers of data sinks are consumers of data and flows are transformations that will transform elements along the way so let me show you how aqua streams works so for aqua streams we will need an actor system because aqua streams components run on top of actors so I'm going to use an implicit Val system as an actor system the actor system has embedded of what's called a materializer so that the ARCA streams components can have their own life and I'm going to create some streaming components so I'm going to create for example a source as source and I'm going to import acha streams Scala DSL source so make sure you import the right package over here and the source can wrap any iterable any collection so I'm going to wrap for example the range 1 to 1000 so this source is an aqua streams component that will emit all the elements from 1 to 1000 in succession so this is a producer of data let me implement a flow as flow of int this needs a type argument so I'm going to import flow from auto stream Scala DSL so if you're following along with me just make sure you import the right package so the Scala DSL package and I'm going to use flow of int dot map and for every element that's going in I'm going to emit twice that element so this will mean that whatever element goes into this so-called pipe so you can imagine a pipe that receives elements on one end and emits twice that element on the other end and I'm going to also create a sink as sink for each int print line and I'm going to import sink from the Scout DSL package in just the same way so the source will emit the elements flow will transform the elements along the way and sync will consume the elements these are the streaming components now how do we plug them together where we can say source dot via flow to sync and the result of this expression is what is called a graph now this graph is basically the layout of all the pipes that you are going to use to send and receive data but the graph is not alive until you call graph run so whenever you call run Alco streams will actually spin up the available resources so that all these elements can actually flow through this inert graph so this will actually make the graph come alive and if I run this application if I add a main method for example I can move this line inside and SIGGRAPH run and I should see all the elements 1 to 1000 multiplied by 2 and then printed to the console so print will actually output to the console all the numbers from 2 to 2,000 in steps of 2 so notice we have all these elements printed to the console 2 4 6 8 and so on and so forth that's because the source emitted the first 1000 the flow transformed them by doubling them and all those doubled elements ended up in to the console so this is how we can actually spin up a runnable graph and make it run by calling the run method on it now if you take a look at the types of the streaming elements in question and I'm going to hover over this source for example you will see that the source is of type source of type int and this other type not used if I hover over the flow will have a flow of type int + int and not used and if I hover over sink you will see sink of int and then this type future done now the main question that many ARCA streams programmers especially when they start out with aqua streams might be asking is what on earth are these types these types are confusing especially for a newcomer it's quite understandable if you anticipated a source for example this guy to be for example a source of int this flow to be a flow of type int + int just like a function transforms elements from type a to type B and this sink should be a sink of type int but what are these other types so we have source of int and not used we have flow of int int and a third type called not used and the sink has a second type called future done so what are these now at this point many acha streams programmers especially when they start out either ignore this extra type and try to focus on what they understand or they try to bash their head against the the documentation and may or may not understand what's going on because the third type is very very abstract but it doesn't actually have to be that heart and this extra type is actually pretty damn great so let me explain I will name this extra type GI values here's the thing it's one thing to run a stream plug in some components and you obtain what is called a graph and then you call the run method like I did here in main and the graph now has its own life but what kind of value do you get when you run the graph running a graph by calling this method it's an expression much like anything else in Scala so it must have a type and the value so what's the value of running a graph so what's this value that I've defined over here what's it worth now I'm going to call this value a Jedi value for reasons that I hope will become apparent surely or at least by the end of the video a Jedi value is what you obtain after running a graph so what I call source via flow to sink the Jedi value of this crap this graph is of type not used so this not used as a dedicated type in akka streams which is a bit like unit and not useful for processing this is what I why it's called not used now here's the thing with these GI values all streaming components in Arcis streams have a jedi value when plugged into a living breathing graph so in this code with the source of your flow to sink I have three components so the source the flow and the sink and thus three GI values somewhere however the graph itself this graph can only return one so then the question becomes which Jedi value does the graph return now the rule is like this when you write something like what I've shown here on screen source of your flow to sink the leftmost Jedi value will be chosen so the Jedi value of the source which is not used will be used here in the graph so the graph run call will actually return the Jedi value of the source which is of type not used all right so I hope this makes sense now you can actually combined these Jedi values by calling some special methods so I'm going to create another graph as source via flow and I'm going to call another method to Matt now now this to Matt function has multiple argument lists and the first argument is the actual component that I want to plug in which is the sink and a combination function of the GI values between the GI value of this particular components source via flow which is the Jedi value of the source as I said earlier and the Jedi value of the sink so you'll have a combination function let's call this left Jedi value and right Jedi value and I'm going to return the right Jedi value okay now this left Jedi value is not used but I'm going to leave it here for readability now I have this another graph good now if I try to run this other graph let's call this another Jedi value as another graph that run now in this case another graph dot run is of type future done which is the Jedi value of the sync now I will need to import both future from Scala concurrent and this done from akka streams so I will have future done as the Jedi value of this other graph now I hope you're not laughing too hard at this Jedi value thing it actually does make sense so bear with me here because after I've returned this another Jedi value I can actually use this future this future in particular will tell me when the graph has actually finished running so I can say let's move this up top I can say another Jedi value dot on complete and I can say arrow let's call this print line stream is done now I will need an execution context to run this on complete callback because the oncomplete callback runs on some thread and I will need an execution context to do that and so I will use the actor systems the dispatcher so I'm gonna say import system dot dispatcher which is an execution context you can think of it as a thread pool all right so I'm running this on complete callback and I'm going to print stream is done after this other graph has actually finished running now let me comment this out and I'm going to run this application again which will do the very same thing it will print all these numbers from - mm but this future can actually do something when it's complete so the Jedi value here actually has a very meaningful utility it can allow us to tell when the stream has actually done so notice that in this case the Jedi value has a meaning which is that the future can be monitored for completion and when you can tell when the stream has finished running also notice that this Jedi value has actually no connection whatsoever to the elements being shoved into the stream so this function that picks the right argument from the two Jedi values over here is called keep dot right for the very common pattern of picking a Jedi value either from the left or to the right of using this to Matt or via Matt methods in plugging akka streams components now there are various components in our streams that return various GRI values for example if you want to extract the sum of all the elements that go through a stream you would construct something like a summing sink as sync dot fold and I would pass in some type arguments here int an int much like I fold operator on a collection that starts with a seed with a 0 element and a combination function that takes the current sum and an incoming element as returning current sum plus incoming element so this particular sync is a sync that actually exposes another type of Jedi values the Jedi value exposed by the sink is of type future int so the int here will actually contain the sum of all the elements that go through it for example let's comment this out and let me create let's call this a some future as and I will need a graph and I'm gonna say source dot to Matt and I'm going to pass in the sink the summing sink and I'm going to use this keep-right function that we saw earlier so when I say keep that right it means that the graph this source to Matt summing sink will actually return the Jedi value of the summing sink which is of type future int and then I need to call run on it so I will return a future int in this case and not only will I be able to tell when the future is actually done by monitoring the completion of this future but I can actually inspect its contents so I'm gonna say some future dot for each let's call this print line meaning the sum of all the 1000 elephants that I shove into this particular stream so if I rerun this application I should see the sum of all the 1000 numbers that I've shoved into this stream because the something doesn't print anything anymore it only aggregates the values that are coming into it and so the 500 thousand 500 is the actual sum of all the 1000 numbers that go through this stream so not only will you know when the future is done but you will also know what the sum of the elements was so notice that in this case we can use the elements in the stream to process them in a single value that the stream will return at the end in the form of a Jedi value now why are these Jedi values greater so JP values are pretty awesome because without them once you start a graph your pipes are opaque sealed and irreversible there's no way of controlling the dreams so once you start no turning back there's no way of controlling streams I'll talk another time about how to shove the data into a stream manually or as a reaction to something now there's no getting any information out of the stream without these GI values there's no processing the values aside from flows and the world will be a very very dark and opaque place so various streaming components innaka streams can offer various Jedi values sinks in particular usually offer futures in the style of future done or a future containing a meaningful value so this will often allow you to combine the values inside the streams into one until if or when the stream has finished some flows offer some control mechanisms like kill switches which allow you to stop the stream at any point some sources offer entry points through which you can send the data inside the stream as you wish I call them Jedi values because they are very powerful the aqua streams library calls them materialized values that's because when you plug in components together you have an inert graph so something like this but when you call the run method on at the gram the graph comes alive or is materialized so the Jedi value returned by materializing a graph is called a materialized value the value may or may not be connected to the actual elements that flow through the stream so this is an important point so Jedi values may or may not be connected to the actual elements that go through the graph and materialized values can be of any type which again may or may not be different from the types of elements flowing through the graph all right so I hope this was useful I'm Daniel and you can find this article at ROC the jvm comm /blog in written form and you can follow me on twitter and linkedin with the links in the description attached to this video now I'm dying for feedback so please leave yours in the comments and if you like this video go ahead and subscribe because more videos like this will be coming soon until next time thank you for watching [Music] [Applause] you

Channel: Rock the JVM

Views: 1,349

Rating: 5 out of 5

Keywords:

Id: 2-CK76cPB9s

Length: 18min 7sec (1087 seconds)

Published: Wed Apr 15 2020