Native LabVIEW MQTT Client & Broker Project

hi my name is francois armande and i'm a labview architect and labview champion today i'm going to present you about the mqtt broker and the associated classes this is a project that is um hosted by the labview open source project and you can find all the information on github okay um i'm gonna jump right into um installing our project so i've chosen to demo this mqtt broker into labview 2020 but it's going to work except for the tls part on all labview versions all the way down to 2013 sp1 so um no worries there you we've got you covered first thing we're going to do is install everything and so i'm going to open a vi package manager and just choose to install the broker at the moment okay so all the packages uh have been installed those are the dependencies for the full project and as you can see the security cp connection was not installed i'm gonna keep that for a little bit later so that we that we can uh cut off the video so for those that are not interested in the tls and and certificates and all that uh and now i i still want to install the broker so i'm going to use this packet this uh vi package and install that luckily it's going to be fast because all the dependencies have been installed already by the way everything is on the labview open source project well as much as possible is being licensed using the xero bsd license which is extremely permissive okay so everything is installed let's go back to labview 2020. i don't need those anymore i'm really gonna start you know super fresh show you off how easy it is to use this framework we're gonna start directly from um from a blank project i'm gonna call that mqtt nemo 2020. uh first thing i want is a server right because you want to test your clients against that and so let's say explore the where where where the uh levy open source project is is set up so it's in the under add-on live view open source project um the mqtt so mqtt contains control packets client and server and all the connections and let's go into the client server so there's a base library here which is as you see there's no constructor this is on purpose you need a client or a server constructor and then your mqtt base is all the generic things that you might want to do like um you know reading the public events uh starting and stopping um subscribing uh which is something that that the client can do um and and destroying and then you've got the client and the the server the broker so the client can create read public events specific to the client you can connect and disconnect see if it's connected publish a message or ping the server and there's a bunch of drop vi's that we're going to use which are as you will see very useful finally on the broker side you have a much easier you know much simpler pallet which is just create server and read the servers public events again there's a drop vi and this is what we're going to use to see how simple it is boom i've got a server already now you'll notice that uh there's a connection here at the beginning and we're we're gonna just just leave it at that and then change it later on this um i don't know why it goes down this much but you're gonna have to find these uh these few uh these few like items indicators it's essentially just the number of connections and what do i do this is my server all right i want to have a client and my client here is going to be a client that i want just to publish data so let's go with the simple publisher of data and again you see i've got the same object for the connection configuration at the beginning i have almost the same thing except that we do have a connection and a disconnection event that we're monitoring through this connect and um and we we can stop the whole thing we are publishing um on the topic my topic name i'm gonna leave that but you can change it to whatever you want obviously and i'm gonna just just change this to a control so that we can set the period uh at different uh different value so it's gonna be you know 100 millisecond publish publish right for now i've put this it's really just a random value and i've put that on the on the on the form so this is the client client one all right and to finish everything i've got this simple client subscriber which yes you've guessed exactly the same pattern except this time not only does it connect but it can also subscribe unsubscribe and i've added some indicators that will change if you get the acknowledgement for from the server that you are indeed correctly subscribed so and this is about it uh we would be ready to to start except uh we need to have a connection now so the palettes there is this uh mqtt connection the connection palette here are generic methods that you can use it's the public api that you can use to either have a client or a server through a listener basically and so what i'm going to do now is choose a configuration uh let's say a tcp configuration for now so all of these vis i think you don't need except if you want to use these connection raw connection palettes so the the configuration node is the same whether you will use a websocket there's a websocket configuration and there is a local queue configuration which you could use if you didn't want to connect to the network so this this arrangement is so that you can you can um extend the framework the connection uh object which uh if you if you override you know the right protected methods you you will end up with a tls config secured uh tcp connection like you'll see when we when we jump into the tls configuration at the end of this demo but uh also if you wanted to do i don't know uh mqtt over can network or you you can you can do it over network streams if you want so so all of that is is possible and you would then then just use the the connection at this point all right this is really all that we need uh we need just uh just uh specify that right now we're still on localhost and you want to have a port so the generic port for the tcp connection is 80 maybe but the generic you know default port for mqtt unsecured is 1883 so i encourage you to use this and you just need to replace like this and this is it so i'm going to use this to create a server that runs on tcp and listens on the 1983 port and i'm gonna just run run the run the server and put it right up there uh client one which is my publisher sorry about this we're going to want to connect exactly the same place so i want to connect to 1883 and let's see if it works if i were to connect yes i'm publishing and i can increase the speed maybe by dropping this so 30 milliseconds there you go and finally my subscribing client and it's as simple as just replacing the connection configuration the rest is completely abstracted away from under under the hood and there you go i'm going to connect and i'm going to subscribe and yes it's there so if i were to to disconnect here and reconnect yes my subscription works because my server didn't stop so there's no problem i can unsubscribe let that run resubscribe and you see that it works disconnect stop and stop everything there you go all right that's it for the uh the demonstration it's really as easy as that and then you can explore other um different clients if you want um examples i've got a simple pub sub so you might have you know your nodes that uh that are listening to commands um and publishing results and so uh definitely you'll need to to merge both publisher and subscriber in the same application there's a template for that there's also a simple reconnecting publisher and you know just just to show you how you can uh can make sure that when the server leaves and and comes back or your own network connection um goes down you can just reconnect uh easily it's as simple as as automating the uh connect disconnect every you know retrying every second or every whatever time period you want and finally there's a more generic client right there code which shows you a bit more how to the different ways that you can decode and so i'd like to spend a little bit of time on that so um essentially um there's a you know the mqtt protocol just specifies binary uh binary transmission of uh you uh basically on in bytes um over the network and so it's useful with uh with labview to obviously just write anything on that payload so what what you're going to find to this built into the client is that it uses a a serializer that serializer um for now is just uh by default using the flattened unflattened labview labview node which means that if you are on the other side and you want to decode and you know the type then you're going to be able to do that very very easily this is something that can can be seen um if we go into the um the subscriber but the the publisher so i'm going to close this i'm going to go back to our client number one which publishes it publishes what it publishes a double resolution value and so as you can see here it's just a variant payload under the hood what will happen is that it's going to serialize i want to show that to you so there's a serializer class that is default to the client and the server and it just uses that to pass the bytes into a jeric mqtt publish publish packet and the rest is ferried to the connection the connection is gonna it's gonna just send those bytes over whichever connection you've decided to to use [Music] so obviously you don't see that here but but because i did decided to use the default serializer but you you can definitely override the serializer and and define it uh as you want so i'm gonna want here to spend just some time to show you that i can i can move to a different serializer and and it's uh and it's gonna work as well so um how to do that um we're gonna need to inject into the client a serializer so how to do that you will go into your mqtt base and you have the get and set serializer so we're gonna just inject here a different type of serializer which as i mentioned you can define your own type as long as as you have the marshalling and unmarshaling of the data which means you know flatten and unflatten if you will or serialize and unserialize or deserialize then then it's going to work so now flattened string is what is the default one which is uh you know by default we need a serializer it might be you know to just take whatever um okay i'm gonna i'm gonna go back here okay um i will keep this um this base open for for your reference but we're going to go back into our add-on labview open source and the open serializer you'll see that here i've got a couple of of different serializer and we used this serializer which is just flat and labview labview code if you if you wire a generic one the default serializer it's going to end up picking this one which is built into the the base uh the basic library and so one of uh of the things that you'll you probably want to do later is to develop let's say i don't know a json serializer or an xml sterilizer or your own encoding for for your for your data if you were to use a very uh lightweight byte you know command where your by your first byte is the is uh is the command the second byte is the argument and the third byte is just a checksum value or crc um and you want to to to communicate with three or four bytes maximum to do super high speed and lightweight and low energy on your network that's entirely up to you so you're gonna need to to go and look into the open serializer here and and essentially override two methods the marshall and non-marshal and by the way i noticed today that i i doubled the l uh in the word i thought it was marshalling and unmarshalling which doubles the l but uh so sorry it looks uh it looks a bit weird for native uh english speakers probably to see a surname there instead of uh of a actual you know english word but uh bear with me it's gonna be fixed for the next uh in the in in the future uh future upgrade all right so you won't mi might want to to to marshall and on marshall directly in binary uh the only difference is that that you know it takes um it's a string that has been been put into bites so so essentially there's no change it's really just just so that you don't have to to insert um a string to bite array or binary to string everywhere it's really just a wrap wrap around but how it works is that you would you would take for example this uh this node here and i want to flatten something and that's something well it has a type and so if you know that type on the other side well you can set the type and when you receive those bytes you're gonna then get variant compared to data you might be wondering why i don't use malleable vi's that's because the base code that it's released in is still 2013 and so malleable vi's are not yet part of the equation but you know there's an exercise if you want you can probably override the these and and and this base and and or just wrap these martial and unmarshall methods with a manual vi which would give you something much much easier and so so if i were to to do this you're going to see that where's my data my data is there so i should be able to to see very easily you know that yes i can flatten and unflatten um easily so this is how it works if you want to write your own serializer make sure that the serializer for the server and your client is the same and and voila it's going to work um i i could go through the process here but i don't have you know anything smart except maybe the serializer which is a plain text sterilizer and the plain text serializer has some some hiccups with if you don't provide a good name or you know things like that but but essentially the most flexible sterilizers you could find out there are the ones that take more time to to to serialize more cpu json for example or xml that might be much more open to you know a wide range of of types now the labview flattened string serializer is going to be very powerful for you provided that you have the exact same type on the other side so it might cause you some pain if you publish data with a single float resolution and not the other side you try to decode it with a double value so so you're going to need to to conform very strictly sorry you're going to need to conform very strictly to the type but uh it's still it's not going to work for for you know very complex types such as clusters and arrays and clusters of arrays or arrays of clusters no worries about that as long as you have the exact same type it's it's gonna work um so i definitely encourage you to develop your your json uh json or xml packets here i think on the vipm um right there there's um if you if you look for the open serializer you're gonna find that there's a base64 so maybe i'm gonna do a demonstration using that right here so let's let's install that okay so my base64 has been installed and compiled let's see what happens unfortunately i don't think that i've put a you know class drop vi like this but still there you go base64 encoder and these are really you know fast and code and code that are wrapped but you don't see it but they inherit from this this open serializer so so if i want to to get in and find my class i'm going to use this you see it's really just a wrapper around it and i'm going to reuse this base base class so if i were to set a base 64 right here i can do that and if i were to do the same thing in the second client now they can talk to each other now you're gonna notice that i don't need to go into the server to do the same because well the server um just deals with uh with with packets with with binary data so so it's really for the clients to to know when they talk to each other what is the protocol that they that they use for decoding the decoding the payload so i'm going to restart my server and uh in c here do i have my client one it's there all right i can connect publish i can connect subscribe and yes indeed i do see that data so now the data has been encoded differently but it's all propagated through the same means through binary strings and non-binary strings by binary streams sorry so that's one of the main the main advantages i think with using this mqtt library our set of libraries is that you can really configure it to whatever you want um change the configuration for the connection boom you're on a different protocol change a configuration for the encoder boom you're on your own encoder you can encode and encrypt you can you know do do whatever you want and and your your your your framework is really as you know flexible as i think it should be for for your your your own application all right now in this next section i'd like to to show you a bit around the connections so we're deep diving a little bit more we've seen the the classes the mqtt client and server classes now let's go down one level and see the connection so if you were to not care too much for my implementation of the the client and server but still want to communicate very lightweight on on over mqtt protocol you can do that with connection and and and even even then if you don't like my connections classes abstraction you can go down and use the packets directly to to implement your own uh your your own all messaging layer so uh i think what i'd like to do is just open an example that is that is straight into the palettes and and these these are um you know probably the best place to to start um whoops sorry the best place to start because you've got the equivalent of a cl of a server at the top where i have a listener in here i'm not i'm not you know doing that into a loop but if you wanted to have n connection uh handling just put put put this listener into into a loop that generates a connection uh asynchronously through maybe vi server or whatever framework you want to use active framework jki state machine objects dqmh or or the messenger library or you know the workers that there are plenty of of actor type process generator out there and so so that that would be i think very very simple for someone used to one of those frameworks to to spawn um you know and and type connection server if you again if you don't want to use the server implementation that we've just discussed and uh and at the bottom it's your client um in infrastructure now here there's only one client uh obviously because there's only one the listener will just shut down as soon as it receives first but you see how simple it is we connect based on the configuration that we've set we check if it's connected uh we can send some bytes and wait uh query for some some bytes and and that's it so here i don't even use the um the open serializer classes i'm just flattening the data unflattening the data because i know what's what's coming up and and the server as you can see doesn't need to run flat and then on and flatten it just echoes back here of course for your own mqtt server you would you would redirect those messages to the subscribers which is again you don't need to do if you use the mqtt server that that was just presented okay so you can just run and spawn a new a new connection every time obviously this is not the way that it should be used you should you should have one server that accepts multiple connections and and just just closes all the connections as soon as a client leaves memory but but you see the point now what i'd like to do is to show you the websocket connection which as also an example and the example i hope you'll appreciate how very very similar it is to the other one except that yes i've just changed the port and change the node which is configure websocket connection and essentially it's the same thing so there you go there's a local cue connection there's also an example are very boring um we just need to define a queue name for for the client to connect to the server and under the hood there's going to be a handshake being done to generate a new randomly randomly created string which will be a unique connection that only the server and the client knows about and so the the local queue connection is going to drop out of the mqtt local and mqtt local is essentially used as a service name if you will for local connection now i definitely encourage you to dive into this particular implementation if you want to create your own um because i've implemented some sort of very very crude um hmac um and shaking where um the uh the server uh re will will connect will allow the connection only if the client has the right credentials and you can basically do that and shaking back and forth which is what the tls you know is is doing in the in the background for you these uh libraries that are that are there or that the websocket connection um api does for you in the background it's gonna end shape and upgrade your your connection to a different uh different protocol so definitely look into this particular implementation if you want to get some inspiration into that but they they run exactly the same way and you can even run them super high speed if you want uh which is probably not useful but uh definitely that works okay so um this is for the clients i mean for the connection connection as you can see you know there's the the basic nodes uh you want to see if there's a connection timeout time get a reason so so get reason text is something that that i've added here so that if you want to give a reason for a disconnection or for protocol violation uh there's a a there's a bit a bit of an override in in the in the in the connection class that that allows you to to specify uh specify an error um message and a reason for for whatever has happened and so um i encourage you to just just look at the code if you're interested in that all right the last thing that we want to explore like to explore the control packets now so control packets again under the mqtt control packets and this is a a very very generic library of of low-level nodes that you might want to use for your own implementation if you want to communicate with the mqtt protocol but you don't recognize yourself in the connection or the client or the server implementations that have been proposed to you this is this is the absolute absolute low level uh mqtt control packets that that you might want to use so there's a polymorphic vi which might be very useful for you you can see you know all the different um different uh payloads if you're a uh you know you want to connect and then when you've received the connect packet well you'll see that there's a byte stream already there so you can just just uh you know set out all the flags payload for connection generate your client id supply your client id do whatever you want keep alive and it will generate you directly the byte stream now you can also go into the different connect packet where if you want to explore what are the connection flags that you've set you'll get that there and so let me let me just complete this if i set this one clean session is true so you know very simple api you can you can see all the control packet more generic you know every everything that the control packet has in common with all the other packets like a publish packet a subscription act knowledge packet a ping packet they all have a few things in in common one is if you want to get the byte stream um know if the packet is invalid or validate you know because you can you can just set that to true and and you'll know if the packet has been is invalid uh you also have um a validate packet node that will give you an error if if it's invalid you can prepare a response in certain cases for example when you receive a publish packet um if you're if you're the client and you send that to to the server well the server will want to prepare a response and so that response based on the qos will will defer so if you are on the quality of service of one you are expected to reply with pub ack packet and so and that pub act packet will be with a packet id that matches the packet id of the publish packet well you might want to go into in here and say read the packet identifier extract that packet identifier and note you know that oh yes i want to get the flags and so these flags will tell me what is the quality of service that is requested and if it's a quality of service requested one then it's a public packet so all of that is included into the actual respond and you'll get a control packet here that if i read its packet type you would see that this is a pub ack but if i were to change qr quality of service to two it would be a you expect to have a pub received as the response and if you're on the other side and you receive a pub receive well then you expect that the control packet type is going to be a pub release let's see if that works there you go so so these things uh these things are all handled if you have a quality of service it's going to be reserved because it's it's well yeah it's going to error out because it's invalid right there's no there is no uh packet id with an invalid um reserved packet here which is don't use that that's the that's the default and the framework not the framework but this this palette here is going to tell you that that there's a problem with that so other thing that you'll notice there's a mqtt 3.1.1 constant and this is because in the future i want to to have a mqtt 5.0 which will allow us to add a another packet type which is the odd authentication packet you'll see that it's already listed there but in mqtt 3.1 it is a invalid packets so so this will arrive basically it's important um to know what is the protocol that is supported by the server and and the client and when they talk to each other they need to be on the on the exact same page so currently the server and the client the only support 3.1.1 so although 5.0 is part of this packet it is not yet available and shown in the palettes because i i prefer that you don't mess uh mess with that now there's another uh cool utility is and you can find it right here it's where you can append incoming bites and so apparent incoming bites is um essentially the way that um that you're going to use to to to take all the bytes that arrive from your tcp connection network stream websocket whatever you want or even a queue you know a local queue and you want to extract very quickly the packets out of there so i created this drop vi that essentially is a demonstration of what you what you can do um let's go through that what you see here is that there's a loop where you would be receiving uh some some packets all right and i i showed that as you know right now it's a cue what you probably want to do first is is to to have your own you know connection that arrives here that will read let's say from tcp and so just just for the sake of it i'd like to to demonstrate um a uh right of a single packet just just so that it goes it's being detected on top so i'm going to use the connect packet and that's it and i'm gonna terminate that right away okay so that there's only a connect packet that is being sent and what i want to show you is that here right there we're gonna handle and receive a packet of type connect and that packet of type connect is going to have you know is going to be exactly the same the same as this as this payload but that's not the point here um what you'll notice is that when when bytes arrive you'll probably have a session and this is this is probably a you know a representation of your private data that you you want to keep all right so if you are developing your own application using directly the control panels instead of you know connection and the client or server architecture that that was shown in the mqtt broker project if you want to use directly uh here you're going to need to have private data so your private data is going to be whatever you know sensor value you want to to keep or commands and and decrypting and whatever session parameters that you have so think of this session you know in qtt session here as if you're using for example a uh jki state machine that would be that would be your your private data bus that that that you keep your data on so in this case i'm going to ignore it completely and the only thing that i want to show you is that whenever you receive data from from somewhere it's going to show up right here through this very simple uh incoming uh decoding coming or append incoming byte it will take whatever incoming bytes and output a packet whenever it's ready and whenever it's full and if it's not full it's going to just send the the bytes on the on the data bus and and append them essentially to what's coming so you don't have to wait for a full frame you could do a very you know bite by bite very simple uh connection communication that is completely materialized you could be losing your connection in between reconnecting and continuing to to send your bytes and that that would still work so this node um will you know decode based on uh a certain protocol uh version and the protocol version that is shown here is 3.1 so so this would be your equivalent of a server essentially receiving data from from a client that is remote and sends you connect all right and yes you

Channel: Francois Normandin

Views: 2,812

Keywords: MQTT, LabVIEW

Id: Y-jrwyfD9DU

Length: 41min 35sec (2495 seconds)

Published: Sat Nov 07 2020