OpenSearch: The Open Source Successor of Elasticsearch?

[Music] hello everyone and welcome to open observability talks and today joining forces with open search community i'm dan horvitz i'm a developer advocate at logs dot io and open observability talks if you're new here is a podcast and a live stream where we talk about anything devops observability and open source each episode we talk with experts from the community and at the end we'll have some news and industry updates and for those joining on the live stream on youtube or twitch feel free to share questions or comments uh on the on the chat uh whether in twitch or youtube and we'll try to get to those uh after after our chat uh i'd like to uh thank our sponsors logs.io for backing the podcast uh if you have something interesting uh uh to share reach out and suggest your topic you can do that on the on the webpage webpage open observability dot io or on twitch at openobserve or directly to me at horowitz whichever way is comfortable for you we're more than happy to have some more people from the community participating uh on the on the podcast and with that let's move on to today's episode and today uh i have a topic that's uh really i'm really excited about near and dear to my heart the new open search open source project um several of my my person my colleagues are already contributing there and playing with it so really exciting and there's no better person to talk to about that than my guest today kyle davis uh kyle is a senior developer advocate for open search at aws and uh let's add kyle to the stream [Music] hey kyle hey everybody good morning good afternoon okay great great to have you here and um kyle maybe uh let's start with uh a few words from you introducing you and maybe also your capacity uh these days yeah sure uh look like the fan height said my name's kyle davis uh i'm a senior developer advocate uh for open search um at aws uh you know i've been doing the developer advocacy for a while now i was head developer advocacy at redis labs previous to this and you know i've been involved with data and databases and software development for most of my life and i i have to say if i wasn't paid to do this stuff i would probably be doing it anyway um which is uh stupid for me to say because i'm sure people who might potentially uh pay my paycheck or or hire me later in my career will be listening to this and say oh we don't pay this guy um but i live in edmonton alberta canada and um i i will apologize there's probably some outside noise it's the first day that's actually warm here it's 11 degrees celsius which is warm for us so um yeah but in my spare time uh when i'm not tinkering with software or databases or data stores i'm um you know gardening or uh you know um tinkering with electronics uh i in 3d printing and stuff like that so i'm just a all-round geek great great and and actually uh uh related to uh today being uh intimately involved with with open source that's actually very exciting uh we at loxio follow that from the very beginning that the big bang when when back when elastic uh decided to uh re-license uh elasticsearch and cabana and essentially closed-source them actually uh for for those following uh following the the podcast uh you probably remember the episode we had back in january exactly talking about that like i called it put the open in uh in open source uh in observability in uh because that's essentially like was a crisis uh for the community and uh even back then and not knowing that will ultimately uh shape up as open search we already started uh discussing with uh many in the community around the uh um forking elasticsearch and cabana to let them carry on as community initiatives and apache to open open activities so maybe interesting uh to hear about uh um you know the the current open source how we are we uh how it kicked off what how the king the team uh shaped up to be uh if you can share that with us yeah so um it's actually kind of a funny story uh so previous to being the developer advocate for open search i was the developer advocate for um open distro for elasticsearch which we'll probably talk about later um but you know that was something amazon had worked on which is a collection of plugins and tools and and um that used open source elasticsearch and kibana so um it wasn't a fork or anything like that but it was just like something that you could add on to it to round out your feature set and give you a deployable um that was that was straightforward and uh so i i i work a few hours ahead of um most of my colleagues um because where my time zone is and just my working hours a lot of them are specific time zone on a different time zone and um so i had a couple hours one morning uh in january and then we had my first meeting was was kind of the middle of day and uh one of my colleagues was basically like we were talking about something kind of mundane for open distro and we're just saying like oh you know our next release will be 1.12 or whatever it was and one of my colleagues was like i don't want to talk about this i want to talk about this and pasted a link to the blog post where elastic said that they were um and i i wouldn't be fair i mean it's not really closed source it's it's just proprietary right it's not open source um where they were taking it to the sspl and elastic license and all that stuff and i was like well it's been a good run uh you know uh it looks like i'll be reassigned to some other project um and uh after that meeting i got called into another meeting and and i was kind of you know i was like well when we went folks i mean you know am i going to work for like sorting packages for amazon prime or something i didn't know what was going to be my next role um and i said no no this is this is not this is not something where um it's necessarily you know a thing that's that's uh that's going to end right and um you know there had never been a situation where we were like seriously considering uh you know forking elasticsearch it was certainly hard to do something like open distro when um another organization would just kind of drop a new version and we'd have to scramble to to make our plugins work and all sorts of things like that um so um you know there's a lot of talks for that kind of first week and there was a pretty good delay right like i think between when we announced that we were doing this then and when um the announcement of uh elasticsearch no longer being apache 2 was i think at least two weeks maybe three um but in that time it became clear that um you know it made the most sense to carry forward with apache two um and take seven point ten point two of elasticsearch in kibana and um you know we can try to make sure that there is an option uh just for our audience that's the latest version that was under apache to uh before they uh switched to the uh sspl and the elastic license right um and so um at that point we we started thinking you know what is this gonna take um and we started you know talking to people in who might have similar points of view you know people that have worked with open distro or people like logs who who are also providing uh elasticsearch as part of their services um and and just trying to figure out what it was going to look like um and that was that was hard i mean you know we had some idea based on running our service uh you know what the code base looked like right and and and what would be required here but it's a big code base and it was basically you know we quickly got support inside amazon to do this the thing that we were kind of realizing in the days after was this is kind of like you know building something from scratch really as far as what would be needed to support it you know the number of developers the number of product managers and and all this stuff um even if we were to hire as fast as possible would would just still be something that we were going to have to make do with what we had while we built more and more um people and more and more structures around it so um that's that was pretty stressful um and then just kind of realizing you know um there was also a lot of debate on how it would look you know would it be something where um you know aws would just kind of create something or would it be a true uh community driven and you know a project that would be destined for kind of a shared governance and ultimately that's that's kind of um you know what we decided that was that was what was right for customers of aws and for users of what would become open search yeah and definitely we're going to talk about that also today so this journey led us to uh today and uh really exciting to uh see just you know a couple of weeks ago that uh we released uh the beta 1.0 uh it was forked off as you said the elasticsearch and cabana 7.10.2 which is again the the latest version uh that that was still under the apache 2 license of elasticsearch and cabana and we have downloadable artifacts so people can actually take you know linux stars or docker images and have a an easier way uh compared to the alpha let's say that uh to start off with that um so maybe if you can just uh summarize what's been achieved so far now to this point of the uh the beta yeah that's a good question so as you mentioned the alpha was released a few months i guess more than a month ago now uh and the beta's been a couple weeks out and you know when we started looking at kind of how we would think about this but uh we were the alpha was basically just let's do a couple things one introduce the name um and have some something that someone could feasibly uh you know compile um and then the beta was where we were kind of basically inviting folks in to say uh you know try it out and tell us what's going on with it um you know does it work does it not work and it's been really illustrative of you know what what the differences are now um you know the process getting there maybe we'll talk about a little bit later but there's there was a long process getting there because it wasn't just a matter of you know clicking the fork button on github and i think that's what some people are like well it's taking so long just click the fork button um and and there was particular complications in this project um having used licensing and things like that but the alpha basically provides you that kind of basic level of you know it could you could run gradle assemble and you could uh you know uh run the build scripts for um you know open source open search dashboards um and by the way though i probably should just preface this for those who aren't following along open search is the replacement for elasticsearch but also the name of the project in itself and then open search dashboards is the replacement or the the new name for the kibana fork um and um so so having those available in the alpha were you know things that would take a while and frankly i did a presentation at a conference and i had to use the alpha and i spent all weekend uh getting it where it would would run you know um it was it was um it was definitely one of those things where you you felt like you're walking into someone's mechanics shop and trying to assemble a car right where the beta was was basically here's a car you can go drive around and it may rattle and it may uh you know the knob for the radio may fall off but it's still a car right something that you can identify as software versus source code so and you know some of the biggest challenges we've had has been figuring out the right structures to put together the software because there is a lot of moving parts right one of the things i do want to focus on a little bit is is kind of construction of this because open search project encompasses like i said open search and open source dashboard but as well as all those kind of additional um plugins and tools that were with open distro um so those would be things sorry to interrupt just uh maybe not all our audience knows and you did mention that before but just to set the stage of what open destroy is and maybe also uh now that you're talking about open search how this this fits in yeah sure so open distro like i said was uh was a collection of plugins and tools um and an installer for open source elasticsearch so uh when you would download uh and start running open distro for elasticsearch it would basically take those artifacts uh and then start providing configuration on top of it and then a series of plugins that would do things like security anomaly detection so those are two kind of like very different things so like the security provides you like ssl and rtsl and um plus yeah yeah um and uh things like uh you know really login and user management and then find green access control um but also things like index management and um a lot of these things that are kind of like common alerting things that you'd really want to use with elasticsearch but then anomaly detection is this kind of um you know ai based tool to detect anomalies based on random cut forest so it had a whole range of features um many of those are things most people probably want to use like you can use certainly you could have used elasticsearch by itself and many people do but oftentimes people found that it was there's a lot of things that just weren't there and and maybe that was the motion from um you know elastic to to provide those other plugins or whatever but um open distro provided a set of free apache 2.0 uh plugins that would fill a lot of those gaps in and you know from aws perspective that what was um the basis for uh amazon elasticsearch service or aes um so that was also there and we had a lot of people in the community that were running at larger organizations things like that so um open distro plus the two forked pieces of code is the open search project um and um the beta included the alpha was just open search and open search dashboards and uh the beta included open search open source dashboards and all the plugins um as well so those things kind of was the first time you could actually do something that was similar to what most would run in production right um so uh it was very different and getting those pieces to work together is actually interesting um uh process like i never until this project found um you know build uh sequences or uh you know build scripts very interesting but i uh understand the complexities of such a large project um and it with all these moving parts working together and you know if you make one change here how do you test everything else uh so i have a lot of respect for those folks um on our and we call them infra team that build um you know all the automation for us because um that's the real lifeblood that'll make this project move forward for the long term this is all that automation that's for sure and uh you know i can also uh testify from from seeing you know the amount of effort also on the peripherals that people don't tend to pay attention to like the technical documentation and all sorts of build processes as you mentioned and others that uh definitely looking from the side at least it didn't look like a trivial task at all and and uh gladly now again it's it's a beta now and uh another big news uh it was just published is that finally we have a formal uh a formal world map that we can uh we can show the community maybe i can also uh share it on the screen to those that are viewing us uh essentially it's it's really exciting and uh you're now able to uh to see uh essentially everything that is up and coming of course it's proposed and obviously it's a community discussion but uh seeing i guess the most int the most uh exciting part for me was to see a rc one in in june 2nd so uh like in a couple of uh days we're going to have uh the rc at least according to this uh schedule and then seeing the the ga plan the 1.0 then 1.1 and we with the relevant items and as as i said it's having that as a community activity that people can actually take part in this is really exciting for me in making things really concrete and also really visible and transparent uh i'd be glad to hear from you what's uh you know the way that we build you know what's you mentioned what's alpha and what's better maybe if you can tell us what's the meaning and the uh uh of rc in this context and what should we expect of it and the later stages and also maybe the process around the contributing and taking part of the roadmap yeah so the next step would be in rc and i think june 2nd would be great um are we going to make june 2nd it's possible um i think there is some optimism uh in that date uh from from the you know things we want to accomplish and i know some of the statuses of those and you know the important thing to know about this uh roadmap is this is the roadmap this is not like the outward facing road map there is no inward facing and outward facing it is a single road map so um you know developers uh inside aws that are working on this project are working from the same thing um there's of course discussions that are going on um you know internally but that's mostly like you know management discussions um things like you know how are we allocating folks to different things um and you know how we would you know navigate getting you know our open search into our service and stuff like that that's normal that wouldn't be things we would expose um to the community but when it comes to actually developing on it we're working off of this this um this road map as well um so uh rc is our release candidate stage um right now the plan is to do one release candidate um and release candidates have different definitions at different places the way we've kind of framed alpha beta and release candidate and finally ga general availability is alpha is untested and feature incomplete beta is untested and feature complete rc is basically tested but not validated and feature complete and then ga is tested validated and feature complete so the goal of open search 1.0 is to basically have minimal number of features different from the open source so elasticsearch 7102 and kibano 710 2. um so those are the goals for for that so of course there's not a whole lot of features there but but as we have seen there's a few things that were just like kind of either low hanging fruit that we could accomplish really easily without affecting anything else or things that were um needed to replace some missing functionality that was there um so that's where we're going with rc1 um so rc1 should be pretty darn close to what ga will be and uh hopefully we do find additional bugs and i i'm you know the funny thing is um internally um they're in testing phase for rc one so in my mind rc1 has already happened um so internally um you know all the plug-in owners that were from open distro things and security are now testing on open search uh rc1 and we'll find things and we'll make small changes here and there um between now and then but you know if we do find things that may push that date out you know by days um not by weeks or anything um and then once we get validation from the community from people who like logs like um you know we hope that logs will take rc one and then start trying to figure out how it's going to fit into the business people will start trying to deploy it on in their own environments um and seriously seeing where you know a more mature version of beta looks and all the findings we have in beta have been fixed uh with exception of a few things that are still kind of pending um and then from there we'll fix anything that we find in rc uh one and um it'll be ga 4 1.0 one thing to look at on the roadmap though is what happens beyond 1.0 that's where things get really interesting i think from a project perspective that's when we'll start putting things in that are additions um 2.0 uh we're using the semantic versioning so 2.0 will be where we'll be uh introducing any breaking changes so you should see no breaking changes uh between open search and elasticsearch 710 to you and kibana 7102 unless there was something that was you know completely unavoidable but i i can't think of anything off top of my head that i'm aware of that would would be a breaking change so you should be able to run it the we're working on a few things like versioning and how that works for some external clients um so yeah um the other thing i want to mention about the roadmap is on the roadmap um there's a way for you to get things onto it right so um that was something that was really important to me working with the community and working with uh you know people outside of aws i said we can't listen in fact i was the delay getting the road map out for at least a few days um because there was i said we can't get this out here without having a place for people in a for a process for people to get something onto this because this is a community driven project um so there is a fact item that gives you kind of an outline of the best way to contribute um you know and try to get something onto the roadmap um the best way to do that of course is say hey i i'm willing myself to build all this code um and and we'll work with you and find out where it fits and try to get it in a logical place on the roadmap if you want to say you know the funny thing is we've had this question a couple times in previous community meeting we had um we had somebody say well look you know is there any plans to rewrite open search and go or rust and i said well that's an idea um it's it's a challenging thing because we have to write rewrite lucine which is the luciferian a library that's in the middle of this and you know if you were to say this and say i want to get this on my roadmap i think that would be a very high bar to get on the roadmap because like you know if you if you said i have 100 developers to to put towards this well that's that's a talk but if you just say i want somebody else to do it for me um that's a much bigger ask so we really hope that that people do um start thinking about the project and how things get onto the roadmap um you know we're looking kind of i think when you go from the 1.0 to 2.0 like that range i think you know we're probably looking at things that will make wise people's lives better um not huge broad scoping features i think that's where 2.0 comes in and we have a whole group of people that will help the community kind of reason through where it comes and and we want to be realistic with people right like you know the priorities here to get this out there running and start this kind of ability for us to deliver i want to see us the entire community deliver features at a regular pace right um that being said you know gigantic features are certain something we're not um we're not opposed to those but they'll probably be uh you know not not something we would put in one point one we're not going to switch open search to go lang in 1.1 um for sure yeah that's one thing we can say for sure yeah i think first of all it's uh it's it's really exciting look but at logs we already installed the alpha and and start playing with it so definitely we're going to uh play with the with the beta and the rc um one thing is interesting you mentioned a few uh a few times uh is i by the way if you're on the live stream i i did post also the uh both the link to uh github with the uh so you can see both the open search project in general and also the specific the the link to the roadmap and also the blog post that details everything that kyle said about how you can get involved how you can post an faq and all the details about that so do look it up and if you're on the listening the podcast then i'll also post it as part of the podcast as well and we have an an interesting question from the uh from the audience uh how many people are actively maintaining open search and i would actually want to expand on this question because again we at logs are proud to uh to take part of this project but uh interesting to learn who else both in in maintainers in contributors not just maintainers but the broader community involved there and of course partner companies all the whoever is involved i'd be glad to hear about that yeah uh it's a great question so i think the easiest way to answer that question is if you actually look on our github you can actually see the people who are working on it i think the number is 100 and some odd 123 or something like that that that's a pretty accurate number um and that would be inclusive of people inside and outside so those are contributors to the project um and you know so there's gonna be some fluidity to that so that might include like we've had you know people that have come and gone from the project so they'll be uh you know just as normal stop fired like we have uh two developers have taken jobs at different places inside aws right like so so they'll be counted are they actively maintaining so you know that's probably not uh but you can see that and we're working in the open here so you can see all of our contributions to it there is a um you know i wouldn't count these people as actively main active maintainers of open search uh there are people who are working on our um on our control plane for our service but they are frankly torqued by um you know everybody else who's working on it um you know external uh contributors i think that if you were to look at it right now it's a lot of aws books and and um just to give people a context before alpha which was just you know you know a month or so ago um this was this project was something we talked about but we didn't have the code out there so there wasn't even an opportunity for folks to really contribute to it um so we right now i think for beta one we had um a couple dozen people that we would say contributed to it um so that's a decent ratio um you know so there's that and and i actually see the chat coming through 123 developers wow um you know that does include folks that are doing some very minor work um like i'm counted in that and i'm you know the website i'm maintaining the website and and i have contributed to open search personally uh but that would be things like i wrote contributing file or updated it so uh it's not like 123 people committing a thousand lines of code a day um so but but all those contributions count um and that number doesn't include those who are providing contributions that are outside of github so we we have a very broad definition of contribution um so those people who are actively participating in our forums or uh you know that that's counts towards contribution in our mind um and that adds probably of the you know few dozen people who are um external that work for beta one that is um you know probably adding you know half of those folks are people who have just commented and given us direction on things so um but 123 people are people who've committed code right um so it's in some way or another that could be marked down but it probably is more like java or whatever um so yeah obviously that as a company decided to uh to back this not just individuals that's i think worth mentioning as well and not only logs i o we're not the only ones don't worry yeah uh that's a good point and i i i'm sorry i missed that when i was answering your question here tom um we do have a number of partner organizations and that's a project that's building up so uh when we talk about partners those are people who have um indicated support um and you know when they indicate support that means basically um you know they're publishing about it or um have told us um in some way or a firm commitment to do so so like we have ivan which is a finnish company that um does uh great work in the data space and they have a lot of different services they've been working with us um you know we have bonsai search uh which if you have been around elasticsearch for a long time you will be familiar with bonsai search they um are just a phenomenal company that have um you know an amazing uh kind of history with the projects um and they were the they were the first people that to to get the alpha on a service um it was shocking it was it was fairly shocking they were like hey we got it running on our service and and you know internally at us we looked around like we don't have that going you know and we've had this for a long time so they're real smart folks there of course logs and and you know um i'm not saying this because i'm on a podcast sponsored by logs but um you know i personally worked with folks at logs um and i've been really impressed by the contribution and commitment to the project um i know there's a blog post that's kind of um we're waiting on the finishing touches on it uh but amateur stern has just been working really hard on this and has written a very comprehensive guide to writing uh an open search plug-in um and that will be that's that's pending right now on the opensearch.org website uh so you can you can read it now if you if you really want to know about it it's it's uh it's all in the open to our website it's open source as well it's bsd3 licensed um it's the only um it's the only part of the project that's not apache 2 licensed i think apache 2 and bsd3 that's a good pairing like like those are ones most people can agree with um there's not anything else in it so yeah sounds good sounds good and um do you have any uh no interesting anecdotes from this forking process so far you you've done amazing work there digging into as you said massive amounts of code both in elasticsearch and in cabana and trying in a very short period of time you know converting it into open search which is again cleaning and getting it in a good shape uh purely apache too and and everything any any interesting anecdotes from that experience that was a really interesting experience and it was like um turning over rocks and finding new things every day um for those who don't know elasticsearch and kibana um were kind of split up in the one repo they contained mixed licenses which made it extremely difficult to just like what i talked about earlier click that fork button if you did that you potentially had some you know risk of having things that you weren't allowed to do if you were to do that so um that was tricky um so it had the a lot elastic license built into it so there's these things called x-pac and x-pac plug-ins are not something that's um you know freely available to the community as far as a license is concerned they might be something you could download and you know use felt cost but you did not have the license to make modifications and have your own versions of those um i'm not an expert or a lawyer so if you are interested in those things please read those licenses and have your counsel read them because um that's always the tricky thing right like these things aren't made for humans to understand they're made for courts to understand uh open source license that's for sure oss license not osi approved in any way shape or form so uh yeah that's about that that's the bottom line that's the bottom line and for those who don't know osi is the uh you know basically says what is open source and what's not so um that was a lot of like basically kind of pulling things apart we had to just pull constantly uh those things out and um you know a lot of it was going line by line through code which is just an extremely tedious process and we kind of isolated those folks off um and they were working solely on that um and we had other people working on open distro we had a couple releases of open distro while we were doing this which those two people like they didn't meet basically they were two divided groups um so we found a lot of things that were interesting there there were quite a bit of things in kind of the marketing stuff in it um that we had to separate out um there were messages that would pop up if you were doing certain things that would say you know try something here or whatever that just weren't appropriate anymore um you know and then you get down into the branding um where um that's actually really challenging especially when you get to open search dashboards um there's a there's a long-standing desire for people to say look i'm delivering this to my customer they don't need to know that it's you know um kibana if you were doing it before open search or open search dashboards so um trying to to split all that out uh in fact i'll give you this story right before we released beta we were doing a bug bash and we're basically sticking everybody on it and saying you know what do we find here and i was sitting there using source graph which is like a source code search engine it's actually pretty neat and finding instances of all these words um you know that that would not be appropriate any longer and removing those um down to if you've used elasticsearch for a long time and there's like a tag line it's like you know for search um in it and that's actually a line from or a derivative of a line from um i think the big lebowski um and we're like that's the sort of thing we have to remove we were removing things and they used um you know names of people that work for elastic inside of it as like kind of like test words in the tests we had to remove those but the most insidious thing that we found i mean this is not bad but um right before beta 1 uh we realized that there was uh an elastic logo that popped up for under 500 milliseconds at certain parts we have one person was like i think i saw it and um we weren't delivering that logo in any way but it was baked into eui which is um which is a library that's called elastic uh user interface library uh it's baked into that so it was the loading icon would come up in certain circumstances so that was one of those things where we had to basically make that change at the last minute restart the build processes and do everything because we had that one which functionally was it was in a template it was a function like an html tag um that we had to do so there's this constant process of every time we turn around you know turning over something and discovering another instance or another time that was there there was um telemetry that was collected the build processes had to be thrown out entirely um so it was uh you know a very uh methodical task um and you know i i we're in a much better place now um i think you if you wanted to we don't think we want to work with everybody we hope that people don't just fork uh uh you know indiscriminately if you wanted to you could click that forking button and probably not be in peril like that's that's the thing that was so was so tedious in this last uh round was just making sure that we were allowing that i think that's a really important open source um freedom is the ability to fork right like if you don't agree with the direction of the project um a good project is one that you're not there's no gotchas in that project um in that the process of forking um so i i i found it amazing i have to admit and you know we have a team that is uh well familiar with both cabana and elasticsearch and still for me it was dazzling to understand how how intermingled the the open source and the proprietary code is and how how not open it is in the sense of as you said being able to okay if you want to use the open source without you know marrying the the commercial part you can just use the open source or fork it or do whatever and that i i found it striking so that was amazing to see and in a way as you said i think one of the achievements you don't see like a rich feature set on the version one of open search but one thing that for me is main and i i i hope that people don't miss out on this significant value is that you actually have a version that is pure open source the pure open source equivalent of what we loved in elasticsearch and cabana but cleaned out of all this dial home baked in and the logos baked in and all sorts of we had something about it using a back-end service that suddenly was blocked although you know it's part of the open source if you open source it then at least and either disconnect it from a service or allow the service access but so all sorts of things that are really uh i i think we're in a better shape uh significantly around that um and another thing i saw a post earlier this month with uh from wu shang from apache skywalking he wrote an excellent blog post i really liked it showing their move from elasticsearch to to open search and as their storage and how they they made a very comprehensive evaluation of open search and and why this is the right path forward for them specifically uh which for me was a good testament of someone else external evaluating open search but i'm curious do you have other uh feedback that you get out of the audience about usage about uh maybe also influencing the roadmap steering the community what kind of signals do you get from the community yeah you know skywalking was a surprise to us it's not like they they um you know came to us and said hey let's let's do this um but i think this is something we're going to see more often you know as projects look at the kind of dependencies or um things that they are integrating into to their own open source project it becomes really a strange decision point to say hey uh let's use this thing that that isn't open source uh in the open source project so um i think we'll see that happening more um as people start integrating those things in and and i wanna indicate as well uh we're working on kind of um you know uh a a version a set of artifacts in fact that would allow folks who just want just like open search nothing else no plug-ins or anything else for these kind of like embedding tasks we're working on an artifact for that because uh you know frankly uh most of our people that we had been in contact with were using it as a product you know kind of a here's the box i use the box and there it is it goes forward but the surprise to us was the number of folks that were really taking this and making it part of their own project um so i think we'll see that happen more often um as far as other open source projects i you know i know that there are folks out there that probably will surprise us but i don't have like a list of like 10 open source projects that have adopted it um but i i'm pretty sure that we'll see more people especially as we go into rce and that sort of thing like if i was running an open source project i don't think i would um you know evaluate it on the alpha right like probably once we get ga and have artifacts that's when people will start increasing their frequency of making that decision to um to go from elasticsearch um sounds good so i think you mentioned before that uh one of the earliest uh decisions that uh aws at least in the internal discussions made was that it's not aws doesn't want to own this open source but rather open it as a community and i think this is a very uh important statement i can testify as logs uh io a member of vlogs io that has we've been collaborating our people and our um our uh contributors uh uh have been collaborating to to you know the testing that you mentioned and on issues and so on uh but looking broadly and there's a very uh active discussion there on the community about what's the path forward and about governance and maybe later on also having that as part of one of the foundations you know apache foundation or cnca for others i'm curious can you share with us what's aws's take on on these moves yeah you know aws from the beginning like the first blog post we wrote about this um we talked about like shared governance and we wanted to be something that uh is not just governed by the whims of one company um and and i think that lots will have been burned by that uh in the past right uh you go to a project that's a benevolent dictatorship and they start going a different direction i mean you know you've seen things i give you countless examples i don't want to call people out but you know like i was just reading about one yesterday where it's basically said like yeah if you don't align with with uh the organization then we're just you know tough right it's not gonna happen um that was behind it so we didn't think that was right um and um so we're still aligning with that we haven't changed our commitment to that so as far as pathway is concerned it's difficult not without lots of challenges and there's this kind of principle at amazon internally that uh you know it's okay to be misunderstood for a while right um and it's one of the principles it's like feel free to be uh misunderstood for long periods of time and i think we're in that phase right now um and our priority right now is getting 1.0 out um and the next kind of step we will look at is bringing people on as not only contributors but people who can commit code right that's the next step you can't you could it would be weird but you couldn't say like we're going to have governance without external contributors like that seems very weird to do so if i was think about this in phases right we're in phase 0 right now where amazon is working with contributors uh in the community and we're the only people that approve the code that's where we're at right now just to be 100 honest with you the next phase will bring on the contributors so people who are not amazonians uh don't work for aws can say yes this is code that can be committed directly into it or they can commit directly into it so they can prove prs and that sort of thing so at that point then we have people who are not only amazonians that are going towards this um then after that's accomplished we start thinking about how more firm definitions of how decisions are made and how code is committed and you know how the roadmap is put together and that's where we start getting into you looking at some sort of governance model right um and then after that you start looking at okay you've got a governance model you have an active community then we start thinking about okay where does this live long term right it's possible that people will go in and say you know what um that's close fine um i i don't know it's also possible that say people will say this is something where it goes to a foundation or a foundation is made to us that has to we have to look at the preceding steps before we can make that decision right um you know a lot of foundations will look at it and say do you have a community of uh contributors from across many organizations um so it's not like we could jump the line and just say okay you know um let's just make it insert foundation here right um that would be putting the cart before the horse so um that's what we're looking at now as far as timeline on this i think it's gonna be a while right um i think the first thing you'll see and this is something you'll start seeing some information about pretty soon is how the path towards contributorship uh there are things that we have to accomplish inside amazon to make this happen and that's been a big surprise too um you know uh amazon is like a fine swiss watch right it's got like little gears that move back and forth and all these mechanisms um and um effectively we're taking a fine swiss watch and saying let's uh take it apart uh add in something else to it like maybe a calendar and put it back together you can do it but it's gonna be difficult and it's gonna take some time um and so that's what we're working on right now um so um that's i wouldn't say like next week you're gonna see anything about it but in the coming weeks we're gonna start talking about the very methodical process to get contributors um and they're not contributing excuse me committers um so um that will set us on the the pathway towards um you know what the long-term governance for this this project is but uh we haven't i mean we wrote it down in the first blog post that shared governance is you know where we want to go with it um so we stand behind that we want to make sure that this is not just amazon calling the shots and if you don't agree with that that that's not something that you know we're interested in doing um and it was never the aim well i think it's it's important news to uh for the community to know that there is going to be a clear uh um progression path and then a way for for uh people to go up the ladder of course the people that are actively involved and contribute to have a clear path on how they can gain influence how they become committers become maintainers and so on and the reviewers and then you know many many standard projects uh as many standard projects go so uh this these are good news and uh i know that um uh also the uh the meetups are going to go down a more open path and i don't know you want to say a few words about the format is going to be allow more collaboration about that sure yeah i mean we're piloting a project this upcoming uh community meeting which will be held on june 2nd i believe um next tuesday um you can find on opensource.org we have a collaborative agenda uh where you can go in and take notes on it during the meeting if you want to add something to it we've enabled commenting on it um so you can comment on it and we'll try to get the pathway in for that uh we're using hack md for that so um it's anybody can go in and contribute to that um and i think it'll be interesting to see how that plays out in our context that's cool i see also we have uh a question from the community i thought we talked about open distro but it's it's i think it's a good opportunity to clarify uh someone asked how is open search setup working together with open distro is there a path forward that so i think yeah we touched about it before but maybe a good point to uh to clarify yeah so one of the things we're working on right now is a fact about um backwards compatibility um and in that it'll outline specifically what you can do but there should be there's a path forward um and in fact uh open search um can have um you'll be able to basically have a mixed cluster with open distro um and then you'll be able to just migrate uh pretty directly directly with that or or if you want to do a blue green type of deployment so we'll have more information on that coming soon but it should be pretty straightforward it should be equivalent to upgrading versions of elasticsearch if you've done that before so it you know let me straightforward and in general in terms of the the vision for the plugins for open search how how is it how is it going to be is that going to be part you mentioned this is going it's packaged on the beta but going forward what's the is there going to be 100 compatibility uh what's the uh uh vision on the plug inside yeah uh so the plugins as far as like compatibility with the open distro plugins they should be the same um the the thing that's being deprecated right now uh is all the plugins were namespaced underscore open distro uh that will probably change to underscore open search or uh underscore plugins we haven't quite landed on what the new thing will be but that won't happen until two so it should be a straightforward um upgrade path uh sounds good so we're about to uh run out of time unfortunately because i have so many other questions that i wanted to ask you but i i think the most important uh question for for our listeners is uh maybe if you can tell us how can people uh get involved in the project yeah absolutely um so first thing i'd say you could do is go to our website opensearch.org on that website you can get to the um github uh project which you can see all of our projects two pinned repositories there are open search and open search dashboards and then you can see all the other plugins are beneath that um in every uh repo we have a contributing document that outlines what you need to know to get contributing um so um the other thing i would say is look at our forums our forums are linked off of opensearch.org as well um our forums are kind of the the communication platform of choice at the moment um and uh that's the best place to talk to folks and see uh you know figure out where you might fit in sounds good sounds good i can only summarize with a comment that i see here from one of our participants eric wrote the royalty predicted this years ago and he thought it would be called the fantastic search so although he decided open search i think it's uh still fantastic uh we're out of time i do want to say that that was our internal uh code name for it before we had a name we called it fantastic search internally um so eric you are really accurate uh we couldn't call it that yeah sounds good and uh and really i think it's a it's a fantastic uh move as a community move and the community play and really excited in seeing the uh drc hopefully even next week and being able to start playing with rc1 and more importantly having the people being able to now uh contribute to the to the roadmap raising points and getting involved because the more and this is where i i approach everyone here on the uh listening to this podcast if you are interested especially if you have a prior experience with elasticsearch and cabana and this ecosystem and then you want to uh to make it better it's up to us so uh let's say reach out and get involved and this is the way to build a strong community and a strong open source that will benefit all of us together uh kyle thank you very very much for uh for joining me today that was a fascinating talk and uh and look forward to some more good news yeah sure thing my pleasure thank you very much and uh with that i would like to go over uh uh to some news uh that i prepared that i collected uh from the uh uh from the interesting updates today um the first one is about uh a new relic new relic acquired if you remember they acquired pixie pixy labs just a few months ago and uh following this acquisition now and eureka decided to open source pixie and pixy is a cloud native uh in cluster observability platform so uh it's really um exciting to have that as part of the cncf they're going to contribute to to the cncf under apache2 license so uh this is going to be a really uh exciting uh new open source available for for observability so uh do follow that by the way i propose that aws the pixel is also going to uh offer a open source to open source to run on aws so just a propose if your aws ecosystem here so that's one piece of interesting news item that i wanted to share with you another one is is uh about uh kubecon uh on the just at the cubecon eu europe uh going and there was a couple of uh lots of interesting talks i think two uh interesting talks that i think uh relatively uh relevant to the to the open observability side of things are uh one by uh we weave works with a very interesting project that demoed um let me just share that for the ones who are available and it can be difficult so anyway uh a project that it's called case span and essentially it can take events from kubernetes and kubernetes ecosystem and turn them into traces that you can actually visualize your kubernetes activity whether the kubernetes native processes like you know k proxy and and and so on or your own pod event so a very interesting open source project still in the works that essentially translate these listens on on these events you know the same event that you can get from cube cattle uh uh events uh but translates them into uh spans that can then be outlined in in a chronological order or causality order in in a in jager or other projects so uh very interesting uh another interesting talk on cubecon was by albert theo that discussed uh using uh trace data for monitoring and alerting uh of applications not just for uh debugging uh very interesting again because as distributed tracing is gaining uh momentum uh people want to take it to the next level and beyond the individual traces and being able to trace them people want to zoom out and be able to see the um sort of the aggregated view and the metrics the performance on the service or maybe in the service operation the individual operation and then following some anomalous behavior then drill down to the specific uh traces so a step back if you like from from the individual trace and this is something that is uh has been contributed recently both on the open telemetry side of things in the collector to be able to calculate these metrics as part of the collection process uh and then visualize that uh query and visualize these metrics on over uh over jaeger tracing so a combination actually of contributions both on open telemetry and jager uh really worthwhile look for uh using trace data for monitoring and alerting on the on cubecon eu um i think uh as we are about to run out of time i'll wrap up here and uh we can follow up with more interesting news on the on the next one uh so just to summarize i would like to thank everyone i'd like to of course thank again uh kyle uh for joining me for this uh fascinating talk and uh thanks everyone for tuning in to this episode of open observability talks all the episodes uh are available uh on your favorite podcast app uh so you're more than welcome to listen that or uh on you on our youtube channel if you prefer the the video format of the of the stream um we typically stream the live streams uh the last uh thursday of every month uh streamed on twitch and youtube so if you listen to the podcast uh uh but you want to join the next one live do look us up on twitch or youtube you can find all the details at openobservability.io on the website or just follow us on twitter at openobserve for updates and of course to share your comments suggestions news bits and again we're more than happy to get more people from the from the community if you have good topics that you'd like to discuss reach out via these channels or directly to me at horowitz eduardo vits and i'd be more than happy to get more ideas for chats for episodes and with that uh thank you for listening i'm dotan horvitz and see you next month on the episode open observable detox thank you [Music] you

Channel: OpenObservability

Views: 970

Rating: 4.5 out of 5

Keywords:

Id: UDvWdTeH5V4

Length: 61min 41sec (3701 seconds)

Published: Thu May 27 2021