IoT Open Source Integration Comparison (Kura, Node-RED, Flogo, Apache Nifi, StreamSets)

hello everybody my name is Karina and I work as technology evangelist for typical software today I want to show you a comparison of different open-source framework for integrating the indite of things a very hot topic these days and there are plenty of different cool frameworks available to everyone explain you when to use which one and how they differ for that let's start with the key takeaways of this session IOT integration frameworks are used to develop lightweight IOT edge applications and micro services they are part of a hybrid integration architecture the reason there is no single framework which solves all the problems you need usually more of them and IOT integration frameworks are complementary to cloud platforms and streaming analytics solutions we will see how these relate to each other at the end of the session with that let's take a look at the agenda for this session first I will talk a little bit about it IOT trends and challenges and why a hybrid integration idtech choice the new default and if you turn most enterprises then I will talk about IOT integration in general and the different kind of frameworks available data pipelines streaming processing process engines and then I will take a deeper look into some of the process engines as open source eclipse cura with RPG camel and node read and flow go and finally I will talk about the relation of these integration frameworks to cloud IOT platform from Google Amazon and so on let's begin with the trends and challenges of IOT first short definition inside of things that's refers to the ever-growing network of physical objects that need connectivity and that need to communicate with each other without that there is no in tired of things they have to communicate with each other and it's not just a few devices I treated the devices talking with each other and sending information to each other if you take a look at many different and I lose the forecast for 2020 you will have it 20 billion devices maybe even more and if you think about 2025 or 2030 the number will be even higher so we have a lot of IOT in the future and we have to work with that somehow already in 2016 there is over 300 IOT platforms on the market with many different frameworks technologies and sensors and all that stuff and all of these have to work with each other which is I think a huge challenge but necessary so that you can leverage the benefits of intent of things if you take a look at Gardner and market guide for IOT integration there's one interesting statement which says through 2018 so only in one and a half years from now half the cost of implementing IOT solutions will be spent on integration and that's simply the reason because as I said in the beginning these devices have to communicate with each other to get added value out of that and therefore without integration there is no intent of things that's really one of the key success factors for IOT projects if you take a look at this from a higher perspective there is the car in decoration and infrastructure and as you see in the middle which runs the business this is your on-premise deployments or in the public cloud now we run all your integration to mission critical systems and legacy systems to cloud services and so on but now we also see the edge which changes the business and in specific in this session now we talk about the inside of things with cars with smartphones with any kind of device which communicates with other devices which might be also in the edge or with devices or with interfaces to systems in the car but we have to interconnect all of that and this IRT stuff creates a lot of new challenges for example these devices are not connected to a cloud they have low bandwidth or latency of connectivity is significantly lower and often the connectivity is not reliable and even if all of that would work it's still very expensive if you have to communicate all of these IOT devices all the time with twenty billion devices we have to do a lot of internet communication so a lot of challenges and from that there are a lot of cool standards available on the market like mqtt coop OPC urs or many many many many more and these are available on different as fraction levels so some are just for messaging some are more on the process layer and for doing standard I processors and so on but really the key is there is not a single one standard many different scenarios and use cases and technologies so we have a lot of different standards on different levels available and that's huge challenges which we have to solve and for that we come to the next point in the agenda the hybrid integration architecture and before we talk about that in more detail the thing again about how we do integration integration in the end has two parts it has connectors to connect to different technologies also IOT technologies or to specific applications like we have seen in the beginning is 300 different IOT applications already available on the market and then we have the process design tools those things like process calls data conversion transitions and decisions and these connectors and processes lines tools together are used to build integration processes these are really the integration flows which build integration between the legacy systems IOT devices and whatever you want to integrate with each other so this is the same for 20 years now but we have many different options for that depending on what scenario we want to solve also in common all of these flows to enterprise integration patterns this is some functionality like filtering or content-based routing or aggregation though these patterns are also used in any kind of integration project no matter if you implement it by yourself of some Java or.net code or whatever you want to use or if you some tooling which has already implemented for you these integration patterns are used always in is integration scenarios but in addition we do not have just one integration solution or framework which we can leverage in the meantime because really need a pervasive integration with many different user roles we have two integration specialists but we also have some kind of citizen integrator which have some technical capabilities but they are not the experts in all the lower-level technologies and we even might have some ad hoc integrators which only have business knowledge and one still to integrate some stuff and then we have all the different technologies we have application to application integration b2b we have cloud services mobile apps and IOT which is the focus of this talk and around all of that we have different governance models operations and the different deployment models therefore we have huge challenges and there is simply no one-size-fits-all IOT integration or integration in general anymore and for that we are talking more and more about hybrid integration platforms and the analysts are sure this is the new default in most enterprises to solve all your integration problems from on-premise as you see here on the left with application integration for example using an enterprise service bus or using modern cloud native integration solutions application integration on a platform as a service like cloud foundry or kubernetes like I pass solutions which are hosted by the integration vendors so you only use it but do not have to think about the infrastructure and operations or like Isis which is used by the business users for doing integration in the public cloud and then we use API management to expose our API to other users and developers to leverage them and combine them I mean in addition to all that in the meantime we now also see H integration this is also really about integrating edge devices and especially important for Internet of Things and so we have a hybrid integration architect with many different components and today I want to focus really on the right side on how to do the edge integration and how that is related also to the other ones therefore after this motivation and short overview let's talk about IOT integration and what that is because there are many different options for different use cases first let's think about this in general and Barry here another new new term which is called for computing or some vendors also call this edge computing and the key difference really is that you do not always have the communication between cloud and the devices but often you have device to device communication so that's really changing these things because now you have much more computing and communication at the edge and the logic cannot be always in a cloud or on-premise data center that's basically the key difference and what many call for computing and the key thing here really is to keep the data closer to the edge for all the reasons we already discussed like latency issues bad connectivity or simply the communication costs between the cloud and the edge all the time with billions of devices and with that let's take a look at a topic like T integration scenario so here on the left side we have these devices and then we have a gateway so that all the devices communicate with the cloud or on-premise business applications and these applications also send all the control events to the devices and they send a feedback back here so there's a lot of communication ongoing all the time and this has some problems like the connectivity is unreliable or when non-trivial latencies in the hops between these communication steps and as I said already natural costs are adding up quickly with all these different IOT devices so a better and more modern IOT integration blueprint would add integration or check also in the gateway here and also ID devices so you do not build all your integration in the business micro services or applications but also add logic in the gateways or in the device this has a lot of benefit so local control is more reliable and you also have less traffic to the cloud applications and that's lower costs and TCO and also you can leverage edge integration and even processing here to do more smart things without even talking to some cloud or remote service so this is the application blueprint which Mitch makes much more sense and IOT scenarios and for that let's take a look at different options how you can integrate with IOT the first one is the data flow pipeline if we talk about data flow that's really more or less something like ETL so you extract from sources you transform it in different types with different operations and then you send the data to the to the final destination and a load part and this is called pipeline or data for pipeline this absolutely makes sense sometimes especially if you want to simply use high throughput s'right through data flows to really do simple or even more complex ETL to get data from A to B you leverage patterns like transformation routing aggregation in that and this is available for both for batch and in the meantime much much more also for stream processing with all these IOT devices so what's important about this data flow pipelines is that this should not be mixed up if stream process stream processing for even correlation which I will talk about in a minute but data flow pipelines already just putting things from A to B with some transformation and so on and with high throughput and for that these are the perfect tools here are a lot of examples which are available on the market there are things like Apache knife I on the left which also includes my knife I a new project which you can deploy closer to the edge in the meantime here's stream sets and a cascade writer that's reopen source projects which also all have a wipe UI for doing the integration and decoding tour developed is ETL and data flows and we have other classic vendors like Helen and Pentaho which is also open source technologies with a business model on top and we have informatica the older and longer available proprietary ETL vendor which also allows building these data flow pipelines so a lot of available on the market and many of them in this case all of these besides informatica use their open source or open core business model for that so great tools for doing data flows but um data flows are not always the right thing so if you want to do more than just data pipelines from A to B then you can also use stream processing or something called as even processing or some called streaming analytics the key really here is you wanna do more correlation while integrating all these devices so here it's really about correlating even streams and you really take a look at continuous queries and sliding windows so that's a key difference to just taking data from A to B you correlated and aggregated while the data is still in motion this is used for scenarios like predictive analytics predictive maintenance fraud detection or cross selling where you Waller want to really act on the data while it is happening and for that in the meantime you can also leverage many more powerful technologies for example you often see now that you apply machine learning and analytic models to stream processing while the data is in motion and here you can leverage many different technologies like R or spark or h2o or commercial vendors like SAS and MATLAB so the key here really is that you can apply all that continuous queries while the data is in motion and that's a very different concept and data flows and ETL which I discussed before but also very learned very relevant for real-time IOT integration Salons there are also very many vendors and frameworks in the market a lot of open source for example on the bottom left chief Ling sparks screaming Apache storm Sam so then you also have of course some commercial vendors like TIBCO IBM software Ichi and also the cloud vendors like Amazon and Microsoft are offering some stream processing frameworks in the cloud already so here is one example where you see this again so I'm streaming analytics for predictive maintenance where you correlate different IOT sensors or voltage temperature in vibration and while the data is in motion you will always apply it to every single event this rule if the vibration spike is followed by a temp spiked then a voltage spike and all of that in a sliding window with in this case four hours then send a flag with high severity alert so that's one example of stream processing for IOT scenarios and this in the end was just an introduction to show you more about a process engine which I want to focus the rest on the talk on because this is really what I mean with IOT integration frameworks because the key difference here is that you really deployed these also at eh it's not just for processing data from A to B or for continuously querying them and analyzing them but also for doing integration at the edge and therefore let's talk about a process engine a process engine is to integrate and wire together different hardware devices sensors api's and online services again you leverage all these integration features like transformation and filtering and Richmond and so on but in addition you do many things which are not done by a typical ETL tool or by a streaming or text tool like retrying re-routing waiting resuming and these are the typical use cases which you can do at the edge for example for device management or device activation where you really want to do logic at the edge so this is much more than just ETL and typically this is done in real-time and it can therefore be synchronously or asynchronously communication this is the basic characteristics of a process engine for the use cases at the edge in the inside of things world you can think about consumer IOT like device registration or management device Diagnostics or usage billing and industrial IOT you can think about device Diagnostics performance optimization or asset management for example there are many more use cases but here you already see how it differs from classical data flows and stream processing use cases and for that I want to no not focus on several different IOT process engines and show them and also there are differences in more detail there are especially three frameworks I want to talk about I think all of these are really great frameworks Apache kora combined with apache camel and then node-red and then flow go these three IOT process engines have a lot in common so all of them are open source frameworks available so that you can do with that whatever you want without any kind of license cost or subscription you can connect to any kind of IOT device with all of them like MQTT co-op or rest or many others they all have a vibe UI for visual coding and testing and debugging and you can redeploy them at the edge and that's also one of the key points why to use them they are all for developers or integration specialists partly also for citizen integrators so if you're not a coder but have some technical knowledge you can at least use node read and flow go very well for Korra I think that's really for developers only and all of them are extendable SDKs api's so all of these frameworks are also to be used by the public for your own projects and you can customize them however you want to do that let's take a look at them in more detail first applied Eclipse Quora and here it's important we use this here in combination of magic camera that's very important and I will talk more about that X Explorer which is a cool IOT gateway but it really focuses on the Gateway features as we will see next it's also open source available as you can see here on github and the key thing here is that it announced a few weeks ago that it's now out-of-the-box integrated with patchy camera an integration framework and this combination of an IOT gateway plus an integration framework makes this a nice combination a little bit more details on Eclipse cure emits an IOT gateway and it's based on Java and OSGi you have a web UI for configuration of the devices networking protocol it's under the eclipse public license or an open source it's a major framework it's around three years old and it uses some kind of Best of Breed idea so usually use it together with other IOT Eclipse projects so use eclipse Quora for the gateway use Eclipse project for smart home and the MQTT server and client and then you can also combine it with patchy camel because kora itself is not really an integration framework it's just a gateway part and so to compare it with the other two with node-red and flow go you have to use it with camel I think that that makes sense also Eclipse Quora and camel that's really more focus for developers and integration specialists so it's not that easy to set up and getting started and simply implement a short integration flow we are a web UI so here really you should use short code and manage the libraries and dependencies and so by yourself so that's basically I'm the characteristics of kora and camel together and here again to differentiate kora from camel so Quora and I have put the screenshot from one from a great introduction to Cora video on YouTube as you can see in this link on the bottom and this really focuses on Gateway cuties for IOT like hard line filled abstraction for sensors and IO access managing Network and connectivity for all the different technologies including Wi-Fi and VPN and you manage things like start and stop remotely install and uninstall application and all of these kind of things therefore the architecture of Cora looks very similar to this IOT gateway features so you really manage connectivity network field protocols and all these things but you want to do in an IOT framework or gateway here's an hello world which you can see I'm from the core website and here you already see and that it is based on Java and OSGi and that you really have to do some kind of coding and configuration for getting started so you need to know about OSGi and bundles and activators and this kind of things to set up an example and then you can use as you see here on the right side a nice ripe UI to manage the devices and all the other connectivity stuff and so on in addition we have to talk about hepatic camel this is an open source integration framework which allows you to integrate all kind of technologies mirror to brain specific language which you see here on the right side and you can connect to many different technologies and endpoints with that you can see here on the right side short code example and camel they call it integration routes and here you always have some kind of import like in this case a file folder and then you leverage some integration patterns like split and content based routing to send this kind of information and data to output endpoints like a file system again or a chain SQ or a mock or whatever kind of technology you want to integrate here that's basically the idea behind apache camel and now a few weeks ago there was the eclipse cure component for apache camel announced so now you can out of the box without any specific coding leverage both together and that's what really makes sense so that now you can leverage the IOT gateway Quora with the integration framework camera to combine this for IOT integration based on the java platform the next one is node-red which i want to show you in more detail node-red is published by IBM but it's also again completely open-source you can use it for free it has no direct I am and it's also a major framework pretty cool to interconnect api's and IOT devices and so on here you see the web UI it's a great visual designer and that's also difference to Eclipse Quora and camel so this one is really focused on developing in the web UI where we'd implement a connectivity and the integration logic for your integration flows characteristics of node-red um its focus is on integration on an IOT gateway it's built on noches on merchants heavily leveraging javascript and it's very very easy to install on your laptop to try it out and play around it's good documentation to create a first integration of law it has good examples it's based on a patchy to Oh license and it's also a major framework around three years old it has very good integration with I don't bluemix cloud platform but again you can also use it anywhere else if any kind of other technology and infrastructure and what's also nice you can deploy it on any kind of platform and you can share as you can see here you flows as very simple JSON strings and that's easy to integrate that into version control or into other automatic tools for example if you want to do some quality assurance on the integration flows you just have to analyze the chase and strings one disadvantage of node-red is that you do not really have binaries in the end to share so you um just develop your integration flows in node-red and then you deploy them there in the engine and run them there that means that you also have to install the runtime on the Gateway device on the device you want to run it on though this will run on an Raspberry Pi for example pretty good but you might have issues on smaller devices where the resource requirements are not as powerful and you simply cannot run this is the JavaScript framework there then this is some disadvantage let's take a short look at node read because as I said it's very easy to use and though I want to show it to you so here we see no tread running on my local laptop I can access it via web UI and installation is very easy we can use rpm here on a Mac or you can use a Windows install and so on and it's pretty easy so let's go to the web UI and here no I want to first start with a very simple hello world example I want to use the inject to obtain some dummy data and I want to use a debugger to print it out and that's basically it for my first hello world flow and then I can deploy this one very easily and then we go to debug and I can send messages here and this simply sends a timestamp and the timestamp here is on all the time I click it sends a new event and the debug component prints it I want to do it a little bit more powerful and by the way I also it just it was the hello world example from the could not read documentation for that and therefore I also use a function component and in this function component that's added here and now I can double click and write some code and as you see here so you need some kind of technical knowledge to do some of the more powerful things in these kind of tools so it's not really for the business user and okay we have note function that's also give it a name transform and what we are doing here we are transforming the date so that we have a nice output so we need to deploy this again and now it should show the date in the right way so this looks much better this is a first integration flow and build with node right it's pretty easy to learn and understand and getting started as you can see here but what you also can see already here is that it's not really the tool for the very powerful and complex integrations so if you have seen some tools which allow yes be like integrations with complex mappings and powerful routines and so on then here you are a little bit more limited and the same is true because it's a web IDE you cannot do as powerful things as an eclipse or any other visual studio or like IDE but for this kind of IOT scenarios it's pretty cool and usually always sufficient remember this is H use cases and not running the core infrastructure systems and so what's also cool is we have the node-red library with some different examples which you can take a look at and use usually you cannot really use them in practice them but you can take a look and play with them and and learn how to build other components and as I said with a node read and also later the flow go this kind of implemented flows are just JSON files which you can then export and import and also integrate into any kind of tooling so here now I import a new flow in this case is a real world on a real world but I'm at least an unreal IOT example with MQTT and so on and I just imported here and as you can see here now we have a more complex flow here so here you'll see that we integrate some sensors in this case it's your an MQTT broker and we see the MQTT topic and which quality of service we use and then we can do some other things like and for example here we have fun delay node where we can say sent just one message per second and this way you use all the different activities and then configure them and also here you see we have again some JavaScript code for doing some more powerful coding also and this is how a typical more complex the flow runs and then you always have simply to deploy it and then you can test it and analyze the log and find errors and redevelop it and finally then you can run it in production again to do that you have to run a node read instance on the device so you have to deploy it here and then import the flows to random there that's fine if you can live with deploying node read em so travel script on that device like an Raspberry Pi if you were one or more memory less memory consuming stuff then note read might be a wrong tool for that but here it's pretty cool and it's also very maître so you can use all of that pretty good it works well but also and what I found out because of the library if you import some things with custom stuff um that might always create problems so for example um after I imported this one which has some new activities which I do not have here and it might be true that this flow does not work anymore so you here you see and contain some unknown notes and I still can deploy it um but now I see an error in node not deployed and so on and be careful I'm importing something from a library because afterwards what I did and I have to remove all the stored data here and then when I run it again it's all empty and then I can start from the beginning again so be careful with what you and download from the Public Library a little bit but it's still cool to learn about new activities and see some other examples this was the example with node red pretty cool for IOT integration I'm going to push on coding editor and let's now also take a look at logo logo was open sourced in October 2016 so it's a very very young integration framework for IOT um it's right now available in the developer elizy so it's really a preview in the end and will get much more mature in the next month of course and you also see here it's also available on github you can go to the code or the documentation and examples community and so on like with every one source project here you see one example of the web UI it's very similar to node-red from the coding perspective you can use the web UI to configure all the stuff and run it here so it's also pretty nice and intuitive and as easy to install and getting started as no Brad a little bit about the facts here the focus of logo is again also on integration or an IOT gateway like node-red but it's also intended to be used for very light white edge application this can either be on an a IOT gateway also but even on much more limited resources like sensors or less powerful computers so not always an Raspberry Pi can be used but we need less powerful things and then you might struggle with JavaScript or Java applications and might be happy if you can use things like flow go which is powered by go-go programming languages is very light white and therefore this is probably also the key differentiator to things like node-red and Cora if you need light white integration and micro-services it's also open source and had a very permissive BSD license so you can use it and do what that whatever you want and as I said before it's right now in Developer Preview released in October 2016 and this one can also be run on a variety of platforms you can also share um the different flows and chasten strings i'm the same story like with no tread but what's pretty cool here is that you can also export it as a your application or micro service as a lightweight binary with zero dependencies and therefore you can install that binary one single file on device and this can be a hdy as a sensor or whatever then works with that and that's a key difference to node red which you have to run always have to run the complete node red javascript unknown ojs engine so one cool feature I want to mention here is the web native step back debugger that's pretty cool because you can really indirect IDE and to debugging without restarting the complete application again and again you can interactively design at the park and stimulate sensory events even change data and configuration and you can even leverage that for remote ops debugging so let's think about you have deployed an integration flow on a sensor or small device and you want to analyze it there and because IOT debugging often is very complex and this helps you there are a lot if you want to connect to that let's also take a short look at Global connection so here we are now and development environment I just want to show you here on two different services one is the weather service let's first run it before I talk more about the details you can see here now I'm that when I click run that the end compiles the code runs example here and so you'll see here in blue that this one run successfully otherwise it would go to the error handler you can also implement logic but what you basically do is you have triggers it's very similar to node-red so i'm here you receive an event HTTP or MQTT or whatever and then you use the different activities for locking or filtering for invoking other rest services like me to here with the weather service you also see the response here in this case I requested a video from my hometown in Germany for an onion and it's right now six degrees Celsius and then we can also reply this to the trigger so this is a very example simple example for integration flow but it shows you the basic concepts and here now you can't could add on different actions and activities and I also have exported this one and that's one of these key issues or differentiators also you can export this as binary for different devices and then you can also run this on the on the local laptop or on any edge device and that's one of the key things which I want to show you shortly here so let's go to my folder head with them so here I'm logo builds or if exported a few different um logo applications and as you see here that's really this is very light white binaries which you can execute so here for example I can execute a better service this is exactly the flow which I build in the web IDE you see it stars in a few milliseconds in this case and I can access it from the web page for example as it as a rest service in this case it just returns the flow ID but I can also double check here in the turn that this works as expected and here it also returns my weather for my hometown Erlangen in this example I'll just showed you how to export and start a flow go example and what you also can do and let's take a look at a little bit more complex example in this example I leveraged an arrest swagger interface so mega which is that in fact a standard for rest interfaces for interface building documentation and testing it and here we use the pet store from swagger which everybody can use and we try to use a cat service to get the pet with the ID - - - in this case now the pet is existing and what our integration flow here does it does query if the pad is existing and based on the condition if it is existing or not it does some different integration logic or application logic in this case either you use the post rest service here of the pet store to create the pet if it is not existing or if it exists in you use the delete method with this rest service to delete the pet so if we run the service once we will now see the flow that as the pet is already existing it close here and deletes the pet we can double check that and swagger here also and now we see the pet is not existing and here now to highlight this one this step-back debugger feature and if I want to run it again I can also start here from the middle in an activity let's run it from the career pet here I say run from tile and now it starts just from here and executes the flow and in this case the condition is that the pet is not existing so it creates the pet again let's double check check this tool you see now the pet was created again and this is really one of the key benefits on how you can debug and test and also change configuration here in this step back debugger and only test parts of the flows and if you think about more complex IOT scenarios or even as I said remote debugging on some edge devices then this is a huge benefit instead of deploying the complete app all the time and doing some debugging and testing with that though that's pretty cool I think that was a short overview about flow go let's now go on back to the presentation and let's highlight a little bit more about why logo and has some golem as language and why this light white micro-services so even if you want to not use a framework but coded by yourself why it's important for edge apps and that it's not that heavy light and therefore Java and this was never an action for logo and probably the same for node-red so it's too heavy weight and also with Oracle with the licenses it's always a little bit of licensing risk for new products and therefore we thought about using node.js for logo this is more lightweight and this is also used by node-red but as we wanted to really create a differentiating project with logo we thought it's still not lightweight unless we want to deploy it really at the edge and not just at a raspberry pi or so and therefore probably the lightest one is C C++ that's pretty cool for that and often used for devices and so on but it's not really a child and not really a modern language and therefore what we chose was the go language and that one is for more modern programming language and used in things like doctor and kubernetes it has a pretty cool concurrency model and built into the language with go routines channels no explicit strike programming like in Java I mean as modern memory management and what I also like it's statically typed that's pretty cool and it's partly object-oriented so it's not all the power and complexity of Java like inheritance but still it has a flexible type system and leverages things like composition and pretty important especially for IOT use cases is this zero dependency programming model that you do not have to think about all the external libraries like you have to do with Java and OS tree I mean all all know about these libraries issues we have all the time and finally that key differentiator for using golang was the speed the ultra-fast compilation and startup time and a very lightweight footprint which we wanted to use because this now you see also and the infrastructure layers here we see the different frameworks we discussed in the past minutes I clip squirrel no tread and flow go and here you see that flow go and contrary to the other tool does not have this um heavyweight framework in the middle with either Java and OSGi or JavaScript and nodejs so that's simply one of the reasons why it's much more lightweight and also all is included in the zero dependency model so as I showed you you have just one binary which you can deploy at the edge you do not have to have the complete runtime like I've not read deployed at the edge now you want to execute it and therefore if you compare the resource requirements you simply see that I'm both on disk and at runtime but also for startup time and memory consumption it's simply I'm completely a different story and that's really um dependent what we want to do and what's a use case often this is fine to have a little bit of memory requirements and usage then for example node-red might be a pretty cool integration framework because one of the advantages is that it's very mature and will go in countries right now as we have seen developer preview so very new so you simply have to think about what makes sense for you if you still want to go with Java and it's fine for you with the IOT gateway and apache camel in addition with Korra it's still pretty cool for some kind of use cases so there is no this is the best or so it depends on the use cases one more outlook about flow go I'm logo we also want to really deploy flow gonna know services on microcontrollers so this is the next step that we deploy part of the flow go runtime even on a nano controller as you see here this is really very very small devices and even these will get some kind of integration logic as for some simple things like filtering or aggregation or that you can do some things they're at the end of the edge I cannot just indicate way so um this was an overview about a lot of different integration options including all these process engines let's know in the end also talk about how this is related to cloud IOT platforms therefore I simply want to show you a few of the different reference I had take just here you see there in reference architecture but you have things the devices on the left then network with the IOT gateway in the middle and the cloud on the right side and in the end these integration frameworks can be used especially in the network gateway a location and sometimes if it's lightweight enough you might even use it directly in the devices so this is for example them may be flow go might be easier to deploy then I know tread or eclipse for Amber Camelot so the same is true for the Amazon IOT reference architecture Amazon is a lot of cool services but of course all of them are running in their public cloud including authentication and authorization at the Wyss gateway rules engine and so on but you also have to deploy integration logic at the edge as we discussed in the last hour and therefore it makes also sense to use these kind of things like horror note read or logo at the edge google io detectors i'm very similar a lot of Google services and here again you have two gateway at the edge and here again you can use no dread Cora or logo there are two one final slide about ibm's open whisk this is an open-source project so you can use it with IBM cloud but you can also use it on your own infrastructure or another cloud and it's a surrealist computing framework it's even based and here the benefit is for this kind of server less architectures that you really just pay only for what you use you do not care about Linux or Windows instances you have to manage or the docker containers you have to manage on instances you really just call think about the request which you make and this is even based and you pay per request for the memory and CPU you use and for that you can think about where does it make sense to use even these IOT integration frameworks and you can deploy them here directly in the server as computing infrastructure here again that's perfect of logo of course because it's so lightweight the hello world uses remake of memory and therefore it's very lightweight and therefore very cost effective for the severest architectures you can also deploy it in the edge here and combine the service computing that's then true of course for all of them logo not red and Cora and for the edge app on the device again you could deploy things like logo for example so to summarize this up a hybrid integration is the new default and there is no one-size-fits-all we've seen a lot of technologies today and open source frameworks for dataflow pipelines and for stream processing and for IOT process engines all of them make sense and think about a scenario and which one is best for you so let's finish with the key takeaways IOT integration frameworks are used to develop lightweight IOT edge applications and micro services they are part of a hybrid integration architecture so they are just one part of the complete solution at most enterprises and they are very complementary to IOT cloud platforms and streaming analytics and often it makes sense to combine different IOT integration frameworks as we have seen for data flows it's sometime absolutely valid to do ETL and use one of these frameworks and on the other side if you need a process engine then something like kora with camel or node-red of logo might be the best choice and with that let's finish this up I hope I'd go if you know good overview and if you have any questions just come back to me or post under this video and thanks for watching

Channel: Kai Wähner

Views: 26,557

Keywords: Internet of Things, IoT, Microservices, Edge Computing, Fog Computing, Open Source, Go, Golang, Flogo, Project Flogo, Node-RED, node red, IBM, Bluemix, TIBCO, Eclipse, Kura, Apache Camel, Serverless, Cloud, AWS, Azure, Google, GCP, Core, Edge, API Management, Integration, Cloud native, Java, JavaScript, OSGi, node js, IoT Gateway, Gateway, Github, Apache, BSD, Nifi, Minifi, StreamSets, Cask, Hydrator, Dataflow, Data Pipeline, Process Engine, Storm, Flink, Samza, Spark Streaming, Streaming Analytics

Id: uOrLOgsCpNE

Length: 48min 17sec (2897 seconds)

Published: Thu Nov 03 2016