LoRa/LoRaWAN tutorial 53: Payload Formatters (V3), fport and json

in this tutorial i will explain what f port is and a short refresher what a json object is i will also explain what payload formatters are and show you several javascript payload formatters examples this presentation can be found at this location all my lora lorawan tutorials and presentations can be found at this location in this video when v2 is mentioned v2 refers to the things network and when v3 is mentioned v3 refers to the think stack community edition this is to keep this video as short as possible in tutorial 17 i have discussed the lora packet format the lora packet comprises of three elements preamble header which is optional and a payload and this is the payload cyclic redundancy check here is a spectrogram of a lora packet this part is the preamble and starting from here is the header and part of the payload let me see if i can detect lora signals in my area using this rtl sdr dongle i'm not going to use this antenna which came along with this device instead i'm using this sleeve dipole antenna which was discussed in tutorial 33. the rtl sdr dongle attached to my macbook if you want to know more about the rtl sdr dongle i am using go to youtube and search for mobile fish rtl sdr and you will see this video which i have created start the cubic sdr application select this option press start set this frequency range between 867.1 and 868.5 megahertz that is the lower frequency range in europe drag your mouse and this must be 868 867 8.67.1 is here so let's zoom out so 867.1 is over here and 8168.5 is over here let's zoom out further yeah that's correct 867.1 and 868.5 megahertz all right so select fm you can click somewhere here doesn't matter you can hide the top window you can hide this window so let's set it here and this scroll bar set it to 400 lines per second here is a lora signal and here is another lora signal this image is taken from the lorawan specification version 1.0.2 the specification can be found at this location this represents the lora packet format again here's the preamble this is the payload header payload itself and the cyclic redundancy check let's zoom in into 5 payload 5 payload is 5 payload and this is 5 payload all lora uplink and downlink messages carry a 5 payload starting with a mac header followed by a mac payload and ending with a message integrity code let's zoom into mac payload make payload is this mac payload which is the same as this mac payload contains a frame header followed by an optional board field called port and an optional frame payload field fport means frame port if the frame payload field this field is not empty the port field must be present if present an f port value of 0 indicates that the frame payload contains mac commands only our encrypted sensor data is in this field according to the lorawan specification f port values 1 to 223 are application specific f port value 224 is dedicated to lorwan mech layer test protocol and f port values 225 to 255 are reserved for future standardized application extensions but in the technical recommendation fyota process summary technical recommendation dr002 version 1.0.0 regarding firmware update of the air on top of the lorawan protocol f ports 225 and 200 201 202 and 203 are listed for specific usage this table can be found at this pdf file combining the lorawan specification and the fiota process summary technical recommendation users should only use the following ports fport 1-223 except value 200 201 202 and 203 f port is one byte in size allowed values are 1 to 223 except f ports 200 201 202 and 203 this byte is always sent even if you use it or not hackers can read this value but if they change this value the package is dropped by the gateway because of the message integrity code that is this code your encrypted sensor data is stored in this field if a data frame carries a payload frame payload must be encrypted before the message integrity code is calculated the frame payload is encrypted with the network session key if f port is zero otherwise it is encrypted with the application session key but you can ignore the above mentioned information because it is not relevant for this tutorial this is my self-built end device it measures the temperature and humidity my end device uses this arduino sketch this sketch uses the mcci lorawan elmig library in my arduino ide version 1.8.10 the library can be found at this location in the sketch change this function to change the f port value in this example i have changed the f port value to one this is the arduino sketch uploaded to my end device and this is my f port value if you use the mcci lorawan elmig library more sketch examples can be found at this location when i set the f port value to 1 in my arduino sketch f port value 1 is shown here if i set f port value to 2 f port value 2 is shown here if i set my f port value to 0 in my arduino sketch no sensor data is received when a set f port value is 11 in my arduino sketch f port value 11 is shown in my console f port value 223 is the highest number i can set and as you can see over here f port value 223 is shown with my rec 7244c lower one gateway and this gateway uses the raspberry pi 4 i'm able to intercept the f port value more information about this gateway c tutorial 51 to monitor udp messages sent between the rec 7244c and the v3 network install the tcp dump application so first access direct 7244c upgrade the raspberry pi packages by entering this command and install tcpdump by entering this command start tcp dump by entering this command with the tcp dump application you can monitor udp messages sent from gateway to lorawan network server and the other way around the messages which are sent from gateway to lorderwen network server has this received package name if you see this copy the data field more information about udp messages see tutorial 29 i have demonstrated this before there is an online tool available which can decode the lorawan 1.0.x packet that is this online decoder this online tool uses the lora radio packet decoder created by anthony kirby the source code can be found at this github page please note anthony kirby is not the owner of this online tool use the online tool for educational demonstration or test purposes never enter your production secret network session key or app session key from the tcp dump get the base64 encoded data this is the online lorawan packet decoder i've pasted the base64 encoded data in this field and press the decode button as you can see i have not entered the network session key or the app session key and this is the output and here is the f port value that is 10 decimal and this is hexadecimal let's look into my rec 7244c i've already upgraded the raspberry pi packages and installed tcp dump enter this command to start tcp dump copy this information this is the json message i just copied receive package this is the message transmitted by my end device i notice because of this low lssi value my end device is near my gateway copy this data go to this lorawan packet decoder and enter the base64 encoded data and press the decode button and this is the f port value there are lower when end devices where f ports cannot be changed these devices have fixed f port values there are lower one end devices where f ports can be changed programmatically for example in an arduino sketch or with add commands there are lower van and devices where f ports can only be changed by modifying compiling and uploading the new firmware to the end device if you want to know if you can change your lorawan and device f port value check your end device manual it is recommended that you use the f port because this byte is always sent you might as well use it every byte not transmitted improves the end device battery life but be careful of the kind of information the f port value represents hackers can use this information for criminal activities i will give you a silly example but it will get my point across so do not use fport if it contains information which can be useful to criminals this represents a wildlife park park rangers are researching this bear's behavior this bear is unique it is three times its normal size so it's a very large bear and the rangers are monitoring how often this bear visits the lake the feeding area and the waste lump these are motion sensors which are placed near entrances these are entrances to the lake and these are entrances to the feeding area and these are the entrances to the waste dump f port 1 refers to the lake f port 2 refers to the feeding area and f port 3 refers to the waste dump by the way here is the range's gateway let's assume on a mountain nearby this wildlife park the criminal has placed its gateway here so it can receive the sensor data let's assume the criminal knows which area this lake feeding area and waste dump each f port value represents by monitoring the f port value the criminal will know in which area the bear is located criminal does not know exactly where the bear is located but it has a general idea where the bear is located so near the lake or near the feeding area or near the waste dump so the criminal can sell the best location to illegal hunters in this particular example it is not recommended to use the f port to represent a certain area here is another example where using the f port value is very useful so this is a potato form and these are sensors measuring temperature and sole moisture f port 1 refers to field 1 f port 2 refers to field 2 etc the f port value is not interesting to hackers there is no monetary gain if someone knows the f port values json stands for javascript object notation and json is a syntax for storing and exchanging data here is an example of a json object the json object starts and ends with curly brackets here and here data consists of key value pairs separated by commas these are key value pairs and separated by commas the key that's this key must be wrapped in double quotes double quotes key and value are separated by a colon the value these are values can be a number number 2 and number 12.44 a string a boolean an array another json object or the value null this is a javascript object in javascript objects the keys are not required to be wrapped in double quotes there are other differences but i will not discuss these in this tutorial to create this sensor javascript object first you create an empty javascript object and just add the keywords the files the files and assign the value to assign an array first create an empty array and assign separate values to this array and assign the array to the javascript object key in this case temperature temperature to create this javascript object create an empty javascript object assign the value to the javascript object key payload formatters allow you to process data going to and from enterprises when a binary payload is sent from enterprise to lorawan network meaning an uplink the data can be converted using for example a javascript payload formatter the data can be converted to human readable data so in this example binary data is sent from end device to gateway and from gateway to lorawan network server and here is the payload formatter which converts this binary data to a human readable data when a message is sent from the lorawan network to the end device meaning downlink the message can be converted using for example a javascript payload formatter the message can be converted from human readable data to binary data the message can be converted from human readable data to binary data which is then sent to the gateway and the gateway broadcast this data which is received by this enterprise the payload formatters used in the v2 console cannot be copied and pasted directly in the v3 console the v2 and v3 payload formatters are different for example the function names and the function parameters are different in this tutorial i will only focus on the payload formatters used in the v3 console this is the decoder payload formatter used in the v2 console and this is the corresponding decode uplink payload formatter used in the v3 console so the function names are different but also the parameters are different this payload formatter requires a json object as its input in the v3 console several payload formatters can be used javascript payload formatters more information can be found at this location cayenne low power payload formatters which can be found at this location and device specific payload formatters device manufacturers may have payload formatters designed to work with their devices more information can be found at this location the v3 payload formatters can be applied to an entire application or to a specific enterprise here is my application and one device this device is assigned to this application when i enter my payload formatter here the payload formatter will be applied to the entire application and if i enter my payload formatter here this payload formatter will only be applied to this specific device i will call this application level and i will call this device level depending on the selected payload formatter meaning entire application or end the file specific you will see five or six four meter type options if you enter the payload formatter for the entire application you will see these five four meter type options and if you enter a payload formatter for an end device you will see these six formatter type options this is the one extra use application payload formatter on the application level these are the formatter type options as discussed earlier when selecting none any enter payload formatter is removed and replaced with the default payload formatter on the device level these are the formatter type options as discussed earlier when both the files and application payload formatter are specified the device payload formatter is used but when you select the option use application payload formatter the application payload formatter is used it is recommended to use the application payload formatters and only use the device payload formatters when there is a need if your application has two or more lorawan end devices and they all transmit the same kind of sensor data of course you should specify the payload formatter on the application level the payload formatter tester is only available on the device level and not on the application level go to applications select my application payload formatter on the application level select uplink select javascript no payload tester available let's go to the enterprise this is my enterprise assigned to this application select payload formatters and as you can see a payload formatted tester is available when no payload formatter is used you will only see the received binary payload in the event details you will only see the binary payload the payload is not formatted and when a payload formatter is used besides the binary payload you will also see the payload which is formatted in the event details you will see the formatted payload let's first look at the uplink javascript payload formatter a sensor transmits this binary data using this f port the lorawan stack referring to this software creates a json object called input with the following key value pairs f port key with the value 8 this 8 and the bytes key with this array hex value 0 1 is converted to decimal value 1 and hex value 0 a is converted to decimal value 10. when you specify the decode uplink payload formatter this input json is already available with these values and this is how you access the input object values in this example a javascript data object is created instead of data i could give it any name the return javascript object has a key called data and this object with these values is displayed in the console in the decode uplink function the returned object can have the following keys the data key which is a javascript object as value a warnings key and a value is an array of warning strings and an errors key with a value which is an array of error strings to recap the decode uplink function has only one function parameter called input which is a json object the json object is automatically created and always contains two key value pairs the f port contains the f port value when binary data is transmitted for example 09607 this data is automatically converted to an array of separate bytes each in its decimal representation so 0 9 becomes 9 decimal 60 hex value becomes 96 decimal and hex value 0 7 becomes 7 decimal and this array is assigned to the bytes key in the json object the payload formatter outputs a javascript object the javascript object can be empty or contains a data key with a javascript object as its value an errors key with an array of error messages as its value or a warnings key with an array of warning messages as its value this is the payload formatter tester you enter hex values in this field and you enter the f port value the hex values are automatically converted into an array of decimal values as you can see over here and the f port value you enter here is the value of this f port key this becomes the input json object which is used by the decode uplink payload formatter if a warning is present in warnings the warning is displayed the payload is valid and the message will not be dropped if an error is present in errors the error is displayed but the message will be dropped as you can see there's no empty object displayed here in the previous example an empty object is displayed here's another example if you enter this byte payload it creates this input json object and this input json object is used by this decode uplink payload formatter this payload formatter outputs this object and the same applies for these byte payload inputs next i will explain the downlink javascript payload formatter this is the encode downlink function and this is the decode doubling function the encode downlink and the decode downlink functions are inverses of each other which i will explain in a moment the encode downlink function is required but the decode downlink function is optional you start with a json payload and an f port the lorawan stack automatically creates this input object with this data key and f port key this json object is this payload and this f port value is this f port value the encode downlink function has access to this input object and the encode downlink function returns this javascript object with the keys bytes and f port in this example this byte and this f port value are transmitted to the nth device when a decode downlink function is specified it uses this input object and returns this javascript object with a data key in this example the data key contains this payload information in the encode downlink function it uses this payload as input and in the decode downlink function it returns this json as output this is the input and this is the output so they are inverses of each other to recap the user creates this json payload and this f port value with this information an input object is created the encode downlink function has access to this input object the return javascript object can have the following keys bytes f port warnings and errors in this example the returned object is a javascript object using this example it returns this javascript object the javascript object is converted to a json input object as you can see over here the decode downlink function returns a javascript object which can have the following keys data warnings and errors if you specify a downlink formatter the encode downlink function is required as mentioned earlier but not the decode downlink function the decode downlink formats the binary data bytes into human readable data just like the decode uplink function here's another example the encode downlink function has only one function parameter input which is a json object the user creates a json object this object is assigned to the data key of the input json object this is done automatically then code downlink has access to the input object in this example the returned object looks like this this is the payload formatter tester you enter your json payload here this payload is used as the value of the data key and the f port has this value this input json object is automatically generated and encode downlink function can access this input json object in this example an error is generated if an error is present in errors as you can see over here the error is displayed as you can see over here the payload is invalid and the message will be dropped as you can see no message is displayed in this example a warning is displayed if a warning is present in warnings the warning is displayed as you can see over here the payload is valid and the message will not be dropped as you can see over here if you enter an empty json payload as you can see over here this empty payload is assigned to this data key here is another example try to figure out yourself if you can get this result this is the same example but now using the payload formatter tester instead of 10 you will see the hex value 0a thanks for watching i hope you enjoyed the video please hit the like button and don't forget to subscribe if you have questions leave your comments below i'll do my best to answer them

Channel: Mobilefish.com

Views: 1,734

Keywords: LoRa/LoRaWAN tutorial 53 Payload Formatters (V3), fport and json, LoRa tutorial 53, LoRaWAN tutorial 53, Payload Formatters (V3), Payload Formatters, fport, json, LoRaWAN Specification, LMIC_setTxData2, MCCI LoRaWAN LMIC library, Javascript object, downlink, uplink, binary payload, decodeUplink, encodeDownlink, decodeDownlink, binary data, LoRa packet, RTL-SDR dongle, Rafael Micro R820T2, RealTek RTL2832U, RF waterfall

Id: yzd90TkCa54

Length: 31min 54sec (1914 seconds)

Published: Mon Jun 07 2021