Best Practices for Middleware and Integration Architecture Modernization with Apache Camel

hello and welcome to this presentation about this practice for integration architect or modernization wait apache camel my name is Klaus Ibsen and I'm a software engineer from Red Hat I've been working on camel for a long time as detect lead wrote a couple of books on camel I'm from Denmark currently live in Denmark but I have lived for 10 years in Sweden in Malmo so yeah it was near Denmark anyway so in this presentation we are mainly talking about camel thought you know how camel has progressed over the time from you know traditional enterprise ESP systems to modern cloud native so before we start what is camel well camel is the swiss army of integration it's one of the most active products at apache camel has been active for over 12 years it's actually a giant Swiss Army knife with a lot of functionality so what do you use camel for you use it for integration so to integrate different systems you know and then you can use camel to plug in between and glue the systems together so camel is a java-based integration framework it has great runtime support for many of the great runtimes you know today with spring dueled and we're gonna see about quark O's Java EE Michael profile and so on it's based on the ideas from a book called enterprise integration patterns and comes with a lot of components which you can think of like kaffir connectors and you've mainly use DSL mister you can write in Java XML to define and describe how systems are integrated in camera routes and it can integrate with almost everything and speaking of routes from your two examples very simple one is just straight one-to-one integration between a file and a DMS cue so in Java code you just say from file to JMS and XML you can do the same at a runtime it doesn't matter from camera it becomes the same I wanna do it so just to give you a quick overview of apache camel tree and the products we have here which will be relevant for this presentation so we have camel which is the swiss knife of integration then you can run camel own to bring boot and then we have camel on OSGi and then the falling tree we're gonna focus on in this presentation is chemic a for service camel on kubernetes then the camel caucus which is a fast camel also native compiled with growl and then at the end we talked a bit about the latest product camera capture connectors so let me just spend a little time talking about the evolution of integration from the lens of camel so oh 10 years ago this was the camel website this is still oldest i could find is from january 2009 and this was our website for a long time but today we have an modern website so back then many years ago they were based on the principles of Sora it was good principle the contract first do service coupling reuse abilities and composability and then so on but it tends to be this centralized software and teams were things they become a bottleneck it's not designed for chains and it's not cloud finned is not the immutable containers and all that kind of things so if you look at an architecture from back then this is a very very old one you may not know it but is based on a specification called jbi so there was a product call it has a service mix that was crazed created around this specification and service mix and was one of the safe aris to camo so inside service mix there was a routing engine that became camel today's ESB architecture is based on primarily based on runtime with host di called basic graph and then you can have many camera application running in 1 John JVM you know kind of like a monolith but in recent time micro service architecture has evolved and you know where it's more like me so design for chains you know individual teams bounded context and all that yada yada you know so a 1.0 based on 12 factor and Netflix stack and so on but and here in in camel we saw people picking up that and you know primary using spring dude to build their different micro service with camel so we say this is micro service 1.0 you can do that today it's been good and camel and but lately and what we're gonna focus on in this presentation is focused on cloud native architectures well you know we have a infrastructure that can aid your application so you have application concerns separate from the infrastructure concerns so the infrastructure take care of networking and security and discoverability and many other things so it's truly cloud native and how can you design or camel applications or integration application in cloud native you can still use spring boot but spring though is becoming a bit of a heavy bottleneck there and you run those in and when I say cloud I mean kubernetes and the concept in companies is to Peggy's your application into a part as the deployment unit and in depart you can maybe in recent time start to use service measures I'm not gonna talk too much Service mixes but it's it is about and you can also not use service mix and about that but this was probably in the beginning BC service mix become more and more key component in the infrastructure and it's also a part that you can get in in the latest versions of urban shift if you're also building integration application you don't need to bring boot you can just use camel we have something called camel main for that and also there is a great product caucus and this is really really an awesome product I encourage people to take a look at and camel works great with Corvis as well and then service architectures we have very exciting product called cam ok I'm happy to present to you today you can run on in the company raised with and without a Canadian and speaking of service what are some of the requirements for running service so in a service platform we want to be able to scale quickly up to and down the pace based on the map because we only want to service requests and and when there's a demand for it so we should have rapid scaling but also very important we should be able to scale to zero so there's no cost of having the service on the platform when there's no demand for and the platform should have a vending and routing so the service can you know pass data using events and be able to have a lie reliable routing and network on the application side of things to put some requirements to it that the implication should be able to stop movie fast but also be able to give a fast first hunter response which gonna see later and also your applications should not be loaded in the in the containers in the cloud so they should have no restrictions on memory and CPU and this size as well and here are some of the products that I build that give us all that of course on the platform we're talking about kubernetes openshift can date it but on the application side you know I'm really proud to say that we have camaleón chorus that can really deliver those on for example with Java technologies and speaking of CAM okay so what is that so I really like this phrase that is a lightweight integration based on Apache camel born on kubernetes with serverless superpowers so it runs on vanilla kubernetes but you can of course also run it on OpenShift and it gives you the best if you have jainism so what is well how do you do that this is what cam Kay is all about you know there is a trend where develops just want to focus on dealing with business logic and not deal with runtimes and all that kind of things they just want to integrate systems and go soulless so what they can do become okay is to write camera routes in a single file so what we have here is a camera written in infox am Java GUI so this is a telegram so every time there is a chat from a telegram we do some custom transformations in the message so Kafka and take the missile from the calf gets the coal a used to be serviced then we could say a camel run a camp Kate comes with a command-line tool called camel with a K so obvious a camel run and then the name of the file and then viola it runs in the in the cluster so how does that work so this is the high-level architect of CAM okay so on one side we have the development varmint and the other side we have T cloud environment and when I say cloud it can be local you can run the cloud bar on you look laptop like Mini Cooper mean shift something like that or you can run on your Red Cloud so what happens is that as a developer you using your environment and you write your code and then you use the tool to update your code into the cluster and this is done kubernetes allow us to extend itself using something called a custom resource definition CRD so what we're done in in camel is to create an integration custom resource and then there is a operator kamek a operator that reacts on these changes to the CR DS so it listens to all the custom resource definitions that are related to camera and integration and then the operator has all the idiotic this is where all this Mars is so the operator figures out what to do is they take your code look at it figure okay what set of dependencies and runtimes and whatnot do I need to may make this run any figures I'm about to do and you do that really fast and we're gonna see a slide here and and this was one of the key goals of cam Kay was to also to be able to run it really fast so one of the issues with running applications on containers is that you have to every time you do a change you have to rebuild the container image because and that takes time and those are the red yellow and green bars here those are different depending on let's say listen if you're building the you image locally or remotely saltiness and all kind of things but they always take time but camel cake andrew has really fast now this is the blue one maybe the first time it has to build a new Amos or prepare for it but every time you do code changes on monarch you can reuse the existing image and do it really fast and you're gonna see that in a moment in a demo you can install concave from operator hub operate hope is a website where you can find operators for many things to run in a container or kubernetes so go look there you can find how to run databases Kafka and many other things so it's a one-click install so this is Dee has a quick game of camelcase on the top we have it text editor it's just a Java code it's a simple timer to lock and on the bottom we have command lines I used the command line and then on the other side of the we have where I locked the activity in the thoughts so let's see the demo it's gonna pre-recorded it who I'm playing the video so pay attention so here in the command line in the bottom I looking at the camel key examples they could see this number files they're out on the other side I've start to get a watch teapots it keeps it till get past watch and you can see there's only the operator running I go back to the examples and now I'm want to run one of the examples of first I say camel version so you can see which version I'm using this is RC 1 we have RC 2 today so as a camel run and then the name of the file in its last last dev mode which means it will run the development mode so it will actually tail the Lord and wreck when I do source code changes now you can see it says hello cam ok it's always running now I say change the file to say bye cam okay I press save and I can see it's instantly updating the kubernetes was quickly to spin over new power and term that you won and I say PI again cam care you can see it's updates immediately and this works in any company's cluster whether it's local or remote I can also do a typo for example forget the semicolon so I'll get a compiler as you can see I get a compiler wrong so this is also one of the magic of cam ok so I can also do make it run faster saying the time us to not trigger ever 12 milliseconds instead of every second and it goes fast as you can see so that's videos let's move on so another product I want to focus on his camel caucus so camel or caucus is trying to solve one of this big problem that Java is slow especially Java has specially a problem with density on containers so you can pack together a lot more node applications or go applications in the same space than a traditional cloud native Java stack that takes up you know potentially hundreds of megabytes worth Giada's can run intense and that's a big problem on on the containers also the AMA has a scaling problem it tends to it was built for another time where you know you boot up a single JVM a big process and keep it running for a long time and then over time it warms up and gets faster and better and what ha whereas in modern cloud world you know you only you have most shortly pros tossed at one for only when it needs and then you should be able to scale up and scale down quickly and certainly also scale to zero when there's no demand fit so these two things is what caucus is trying to help it from the Java point of view on the website they sake caucus is a so supersonic subatomic java which is yeah yeah nonsense so what is it so it's a kubernetes Navy Java stack Taylor for crawl and open JDK hotspot crafted from the best of breeds Java libraries and Stannis okay some like these stack lines I don't but what it is is just fast Java fast Java in two modes native mode with chromium compile which is the fastest but also the hardest one to get to and then you have in the middle which is just gave a mouse it's an optimized T they note this is what we're really also and many people are using so Quackers has also a very minimal footprint this is a slide from the website so you can see here the gray ones it's a traditional cloud native stacks 140 megabytes but even just by using Quercus in JVM oh so you just act workers to your java product it can reduce the memory to half and if you go full cloud it is only tens of megabytes but also pay attention that is really fast on the first response time so a traditional one might take four or five seconds to boot up warm up and give a response even in JD mode the blue one it can get around a second or so and if you go cloud native is you know milliseconds and here's an example from camel just a screenshot saying that camel starts off in 13 milliseconds it takes up 16 megabytes of total memory RSS resident set and the image size is 68 megabyte that's the big topic we working on reducing that I think we are down to 20 or so in the latest country to development with currently working on so stay tuned for most faster and better girl-- and let's give us a couple of demos so this one is it's the timer to lock demo so is from the Camel Corps website exams timer to lock so what I'm going to do I have checked out that source code and I'm going to the editor this is visual code and hoping the example and again it's just the timer to log it's from timer hello world and down here in the terminal I'm typing maven compile forecast if so I've run in a caucus in development mode so the idea is that you boot up you have the casein and then quark is keep on monitoring and does hot deployment when you do code changes just as you saw an excellent so I say bye world and as you can see it's merely updating down in the terminal and this is not running any cloud this is running locally in this is just standard Java but you know it of course also works in the cloud as well now I can of course change the to go a bit quicker and this tool in missile code also have plugins and one of the plugins is for camel so what I'm going to see here is that you can actually for the camel timer component you can get a list of all the options you can configure on that under the documentation as well so this is really handy so we don't have to switch to the website and we you know on the website what the options are and what they can do that's really lovely and also we can see that the tool understands that all the existing cam components you can use so here's a list of them the Amazon ABS the piece young SQL FTP google hazel cast and so on kubernetes even MongoDB and so on sale force so does a lot ok let's move on so this is the second demo which is actually very interesting odd thing so to get start with focus is that you can go through the website course IO and then click start coding and you can create a new product where you can type in you know goo body artefact ID for maven or choose Gradle to build you can look for extensions so extension 2 is actually you know like custom components for corpus and we there a lot of different for hibernate and many other things but you know I like camel so what I'm gonna do here is I'm picking some extension I want to have health check so I choose small right health checks and I'm looking for the metrics apparently I can't find it so I can type it in in the top box metrics yes and then I also need D comma bits scrolling down hunting camels in the bottom integration Oh as you see there's so the Adina matrix and health checks so basically I just wanna build a basic breast application or his Pig application we build without of the box health checks and and matrix using these extensions and then you can click the generate button and it will download that I already did that and I create a little X application here there's a rest service here stairs hello / hello you can see you can get that and then this is my camel route so I have my route extends repeal repeal is from a passive camel and I at the application scoped notation so the the Robbie's discover then I said from platform is the P to integrate with is P platform from Quercus I say hello camo and then I say camel runs on a hostname so I can get the name of the host word runs and now I am running this one I already did pre-compiled this one I already also so we can see it run this is a fat jar at one point three seconds to run put up with all these features I can go on the local host 8080 and this is the welcome page for Quercus I can slash slash hello and I get the hello response and I can take hello camel of course and I can see it runs on my laptop and I should also be able to get the hell check it's up and you get camel status as well so it's a combined pill check and I can get the metrics so you can get these metrics and gather them into a centralized dashboard you know from each is curve on and all that one versus process he says exchange total want of 0 now we should say - if i refresh the screen you can see now it's 2.0 ok so what we would like to do now is to get out the memory so I did a little script here to try to gather the game total memory uses of that one in one hundred ninety eight megabytes that's the resident set so it's the total heap whatever you know in Java and that matters because that is the memory that is will be reserved and used on on a company's cluster for example that's a bit high so that was the JVM mode so now I'm running the native compiled and it starts in twenty milliseconds and camel is two milliseconds so that's fantastic fast so that takes about 20 mega so little little less because the the the script is a bit of a little bit and I can of course do the same thing hello camel okay so what I'm gonna do now is to have a script that say run many so I can run 100 instance of that application from the range of 8,000 to 8,000 one port numbers so now I start one hundred instance I started one her instant and I get the total memory now it's one point eight gigabytes so that's eighteen megabytes pea a piece and I can of course still do some port 44 hello and hello camel so that works I guess choose another port number 96 also runs of course now I can actually kill them all now they all did I click missed it in the browser and I will start them again and I preface you know so that is really really fast to start up and service the first time to response okay let's move on to the last item the camel kafka connector I'm not gonna say I'm not an expert in Kafka but of course it's very popular streaming platform but one of the features that Dee Kafka has is that it has Catholic connectors and what is a coffee connector here well again it's a way of being able to feed data into the kafka and take data out of caffeine into order external systems so it's kind of like a camel Lobby and they have something they call it sauce connectors and sync connectors so what we've done is camel is to create a product called hammock after connectors it was started as a prototype by Red Hat and then donated to a passes of a foundation in the last year's so in the same way we want to be able to use as many or if not all the common components as sync and sources to Kafka and these are the first prototypes we done these are the one we had tested the most and whatnot and just as a part of a prototype and there is a little demo of that and we're costantino a colleague of mine has created you can find it on that link on the github so it's taken from history into Kafka and into a JMS missus program now we have to say that all these connectors are 100% open-source free to use what not they are not part of a commercial platform anybody can use them so you're not logged into okay I go to Kafka connector or marketplace and I can see some of them's a community and some are only from a specific vendor I cannot use them and the source code is no way about this is 100 and open-source and then you can run Kafka on kubernetes using this rimsey project with is also an operator hub so the attention with the connectors chemical negative is to have as little coding as possible may smoke and primary only configuration so these are the meat from the what do you need to write or what Android it is to set up the camel source and then the camel sink so the sauce is the one that texts from history to Kafka and it uses the camel and a AWS s3 connectors and then on the other side you use the camel DMS components there's one called symbol GM s as s DM s + 2 is 14 is to API and then just a little bit of configuration and off you are and there's also a dim of that there's a three minute video here I just put it in the slide so you can find it here but we are running out of time so I'm going to continue just to give you some links for more material these are camel of course the comic a website walkers catholic connectors there's also youtube on that little coming demo showed if you're interested in hearing about when comic a was created and by whom you know Nikola and Luka has some blocks there's also a great webinar but these two guys and then there's also a video on camel K&K need this is a bit harder to run because you need a big cluster still to run k native but this one he didn't bail on dinner and there's also a couple of quick webinars or talks me myself and Andrea did a more in-depth overview of camel 3 you can find it on on the first link and then I will recommend bilkins Q can talk in 2020 about the evolution of distribution kubernetes where he really is talking about design patterns and whatnot this is really interesting he's really a great source of this kind of on a higher abstraction level and his book is very polar the company's patterns you can download it from that link on my book camel in action is all about camel and for Tony it's not free you have to pay a little bit some people have to live live in these times but you can find it on Amazon and all the bookstores Red Hat Red Hat the commercial wise we have to get an integration product which comes with camel and all these kind of things and you can find it all we have the middleware products and the Edie Edie a Architect us well thank you very much and now we'll have open for questions and sorry for running over time a bit

Channel: Claus Ibsen

Views: 8,098

Rating: 4.9142857 out of 5

Keywords: apache camel, kubernetes, knative, quarkus, software, open source, integration

Id: d1Hr78a7Lww

Length: 31min 25sec (1885 seconds)

Published: Tue Mar 24 2020