AWS On Air ft. Amazon OpenSearch Service | AWS Events

[Music] another interpreter asl interpreter with us genesis thank you for joining us as well uh mahol tell us a little bit about yourself what is your role at aws sure uh so i'm mehul shah i'm the general manager of search services at aws this includes the amazon open search service open search the open source software and one of our legacy services cloud search okay all right so what did you launch so we did two things actually recently uh the first thing is that um we renamed our managed service used to be called the amazon elasticsearch service to the amazon open search service and we launched a brand new engine that you can use called open search 1.0 okay so i i think make sure let's make sure all the the names and brands and terminology is really clear to our audience so perhaps explain elastic search and open search and put those into perspective and relationship if you could fair enough um so let me tell you what open search is um open search is a distributed um data system uh it's an open source search and analytics suite okay so you can just go download it and start using it it includes um open search the engine itself as well as a visualization layer called open search dashboards and you can use it for a wide variety of use cases it's really a broad set of use cases that it supports a lot of our customers use it for log analytics if you're trying to go and figure out what's going on with your system to do root cause analysis for example they use it for real-time application monitoring and we also see it um used for website search uh pretty frequently um one of the cool things about open search is that it actually indexes all of your data and you can get answers really within seconds and under a second you can run queries over that data over petabytes of data and get answers really fast and then if you're not if you really prefer visual tools dashboards allows you to kind of discover and browse the data sets that you have and so that's that's what open search is and it's really an open source continuation of elasticsearch which was the original um engine that we're based off of and um and that's the open source engine and the managed service uh is is just a it lets you uh run it basically manages um running a cluster um of open search um uh clusters or domains um in your account okay now mahu you've said all your data um what what how how do you draw the lines around that what what kind of aws sources does that actually entail uh what a great question um so our customers actually you put a variety of data sets in there you can you can put structured data sets in there you can also put semi-structured data sets in there and these things include things like your machine logs click streams um you know json data the kinds of things that you use that's really coming out of various applications and machines so that you can go and understand what's going on in your system so uh we we see customers using it for um also documents as well so if you're trying to do search across you know your your your your website um that's another example of a kind of data set that you can put in there so you've you've talked about open search a little bit what's what is special about open search 1.0 what do we need to know about that sure so open search 1.0 is um is really a a fork of um elasticsearch 7.10 and open search dashboards is a a fork of kibana 7.10. you see customers our customers came to us and they sort of said listen you know we we're we're really used to elasticsearch and you know we're actually quite customer obsessed um but what what's happened is elasticsearch and it's its license has changed to a proprietary license and they really really enjoyed the open source apache uh licensed version two um they want to be able to you know take advantage of the open source community the open source tools and the ecosystem and you know the variety of isvs and vendors that they can go and rely on and since they didn't have an open source option we decided to fork and create create a new distribution called open search of the engine and all of the the components that were within it got it so uh can you talk a little bit about on the existing open distro for elasticsearch project and uh open search itself sure uh so uh open distro for elasticsearch was really a collection of extensions or plugins uh that we were adding to big search uh elasticsearch needed a uh last six years was adding a lot of proprietary plugins um into elasticsearch and so what we did is we came up with a different suite that you could use for doing search for doing security for uh doing enterprise-grade anomaly detection and sort of a variety of advanced features and functionality odfe is what we call it for short open district for elasticsearch and all of those components now are actually built in into open search so it as of version 1.13 all future versions of all the plugins in odf are now available in open search so what was the thinking behind um our actually doing the rename on this ah fair uh that's a great question so why do we fork um why did we do the rename well um the the biggest reason is because we really wanted to have an identity independent of uh of elastic and the elastic search we wanted to take this in a direction that was built and and maintained by the community we are just so we believe building uh software in the open is really the right way to be building a software it gives you an opportunity to get closer to your customers they have an opportunity to go and actually give you feedback directly very quickly that gets into the code base and so you can iterate much much faster um and so one of the reasons why we changed the name uh created the new uh the new distribution is so that we could keep this sort of thriving ecosystem going okay okay and you can have a lot of one of our customers is already using elasticsearch uh what is their migration path or process do they need to rewrite rebuild migrate or is it transparent how does that all work oh i think that's a great question so if you're using an open source version of elasticsearch you don't have to worry about anything if you're on the managed service don't worry about a thing it continues to run exactly the way that you hoped that it would run now if you're thinking about migrating and going to open search 1.0 it's just like migrating into any new version of a newer version of elasticsearch you go to the web you know you go to your console you click on the upgrade button you wait for a little bit and we will automatically migrate the data to the new uh to the new engine and open search 1.0 is 100 the apis are 100 compatible with the elasticsearch 7.10 apis it's wire compatible so you don't have to worry about changing your applications all your clients we've actually even forked all the open source clients um that are out there for uh for elasticsearch so that they will continue to work with uh with open search so if you have your existing application don't have to worry about it and if you want to move to open search and use your functionality that are in these clients you can use one of these four clients as well so don't have to worry about it and the ecosystem is going to come along with us yeah that's great what uh so we're developing this in public uh is there a public road map and what does that what does that mean for our customers oh well what a great question um so this is new for us in aws yeah um and and you know we we struggled with this but i think at the end of the day you know we want to we want to we we want to be true uh to the to the open source ecosystem so not just you know putting this the source code out there that's not the only thing that we're doing right we're actually also putting our entire road map out there in the public as we you know go along and decide what we're going to build and we're actually doing the building in the public you'll see commits that are sort of intermediate commits um in in github and all of this stuff is in github repos that you can watch yourself and if you want to give us feedback um on our public roadmap or some of the things that that are that are coming through feel free to just go directly to github and you know open it open an issue or send us a comment and we'll respond you know as quickly as we can there are other ways that you can give us feedback as well you can go to the forums and create a new topic or a discussion you can just email us directly and you know other ways that you can actually uh influence us is by you know standing up and contributing collaborating with us on documentation on blogs or just even actually just writing code and and doing a pull request we have a number of people that have already uh submitted a bunch of things from the community you know that's great to hear that you are you're taking so much community input and watching the chat and all the great questions and feedback in the chat i see that we've got some of our viewers are asking about job opportunities at aws and at amazon it seems to me that one interesting avenue to do that might be to get involved as a as a open source community contributor to to the the product uh start sending sending feedback sending pull requests kind of show that you're interested so that you've got the passion for it and then at a certain point kind of raise your hand and say hey you you may already know me would you like to hire me is that is that legit uh absolutely if you're active um you're sending us stuff we're taking you know we're taking your commits and and and and um and your so your pull requests and bring them into the code base and you're interested in working on open source please give us a call we actually have lots of luck or actually just go to the amazon.com jobs website and we have a number of jobs already posted there if you're interested in working on open source come come to open search you can apply directly over there you can send us a a note over email if you go to opensearch.org soon enough there are going to be job postings there as well and you can go and look over there as well so there's a bunch of different ways but absolutely if you're interested and you don't have to be just a developer by the way we're hiring product managers um go to market specialists um as well as uh uh open source uh engineers people that really just you know care a lot about open source but want to do development um in a in you know in a fast-moving uh growing environment like ours sure yeah yeah there's a number of links that have been dropped in the chat uh related to uh both getting involved as well as finding jobs in this space as well uh so one one concern that i think um people might have is that elasticsearch might fall behind open search can you um can you talk a little bit about that and uh you've mentioned multiple times now that there's 100 backwards compatibility uh and the tools are going to continue to work but what what are we doing to make sure that that that is the case i think that's a that's a brilliant question a lot of our customers um i really are worried about it and that and you know i know where they're coming from right there's a there you know there's a name change we're telling you everything is exactly the same as it used to be but is it really um is the community going to continue to come with us and i don't want i want to make sure that there's no doubt about this when amazon and aws decide that they're going to go invest in something and give our customers an offering we plan on doing it for a very very very long time pretty much forever and so we will continue to invest in open search independently uh so that open search and its ecosystem continues to thrive we're already doing it we've done before if you look at open search 1.0 there are a number of new features in there that we're going to be showing you shortly um i'll talk about those in a second um there are also a number of things that are in the managed service uh today um advanced features that we plan on uh moving into the open source uh so you're gonna see a lot more of the stuff coming um in the future things that i think are well you know that's really far out not so much for example on the roadmap we have cross-cluster replication it gives you much higher availability than you you're able to get today improvements and machine learning and anomaly detection um stuff that's really that felt like science fiction a few years ago is going to be in the open source turned off so so we're pushing on that but also all of the tools that work um with open source we're committed to those as well um the clients as we said we you know we're forking um and you'll be able to continue to evolve your applications with you know your client of your choice open search dashboards but all the programming language compliance and then there are ingestion tools as well and we're investing in data prepper um and that's going to be sort of the the tool for ingestion as we go forward wow it sounds really cool can we actually see this in action oh absolutely um i don't know if uh carl is available here but we should get it started and it can show you how you can actually just migrate it's pretty simple and then he'll show you two or three um i think two of the three main features that we have in open search data streams notebooks and transforms he's going to focus on the last two i'll hand it over thank you hi carl hi thanks for thanks for having me yeah thanks for joining us uh tell us what you do at aws so i run uh product management for mobile uh for across our search services line of business um so yeah i uh help us figure out the stuff we want to build talk to customers do all the fun stuff excellent and what are you going to show us i'm going to give us a give you a walk through the the newly renamed amazon open search service and then we're going to pull up open search 1.0 and see what it looks like okay i'm going to share your screen okay great so for those of you familiar this is the amazon management uh console the um you know first off this amazon open search service is a rename so it's still the same service you know and love you just now you'll see that if you go to your services and you go to analytics there's no elastic search it says amazon open search service but to help you got this little tag here that says successor to elasticsearch so there's no surprises and if this menu is daunting you can always just type in open search and you'll find it it's right here and once you get there if you're an amazon elastic search service user this will look very familiar it is the same service it's just that now we have this new column for engine and you can see all of my elastic search environments are still running nothing broke they're all very happy but i have this new one that also shows open search 1.0 and if i go to create a new domain i can choose so like by default it's gonna the latest version is the 1.0 but all of my apps and thank you docker if all of my apps are um still coded to say elasticsearch 6.8 no worries i can still deploy elasticsearch 6.8 uh we'll still support it we still take care of it but uh for the latest and greatest i'll go to 1.0 and if i uh if it's if i'm now ready to take the leap and let's see i've got an elasticsearch 7.8 domain like this one here and i want to upgrade it to open search 1.0 it's just a simple click i can upgrade it to 1.0 and one click it's going to go and do an in-place upgrade and bring that domain up to the latest version so with that let's actually jump over and look at open search 1.0 and to do that we're going to look at the open search dashboards so open search dashboards is a is the visualization interface that uh pairs with uh open search it's the fork of cabana which fun fact uh cabana uh grafana which we talked about quite a bit earlier is also a fork of cabana so the open dashboards does actually probably share some code deep in the bowel somewhere with what um you know helen was talking about earlier i'm logging in and now once i log into open search dashboards as i mentioned it's a visualization tool from exploring and analyzing the data that i have in open search so the standard features that i had in there for kibana are all here in open search dashboards if i want to go just look at some dashboards i've already created i've got here some sample data for global flights this is the type of thing i can do with the open search dashboards i can build all sorts of rich virtual visualizations and uh um graphics here in open search dashboards there you go so i can see all sorts of visualizations now as mobile mentioned the um open search 1.0 isn't just a port of elasticsearch 710 we've actually also introduced some new features and with every version we're releasing new features so with open search 1.0 you can see we've got these panels here's the standard panels that existed with uh cabana and dashboards these plug-ins are the extensions that were developed as part of odf or open open distro for elasticsearch that we've also ported over to in search so much of the this functionality came from open distro now uh with open search 1.0 we've added a few new features so i was going to show you two of the three we added data streams which is a new type of indexing and time series index patterns and we've got uh transforms is the first one i want to show you transforms is a function that's included in our index management section you use index management to manage the life cycle of indexes over time you can move them across different storage tiers or roll them off the cluster but you can also do things to change how the data is structured in an index so an existing feature we had was called a roll-up where you'd roll up data the new feature that's introduced in 1.0 is called a transform so with the transform we'll go ahead and make a job it's pretty straightforward it's a lot like creating a materialized view in a database where what i'll do is i'm just going to call this transform and first thing i do is i select the source index so i'm going to select that sample flight data data source i was using for that dashboard a minute ago and then i specify a target so i'm going to send it to a new index called transform and then i specify the which fields i want to put in that trend in that index so this is pretty straightforward so from the from the source index i want to grab my destination cities i'm going to grab the average ticket price i'm going to grab my flake distance and miles and i'm going to grab flights let's do delays in minutes and i'm going to pull this destination up to the top okay and then we can see uh what i want to do with this data and then target index is i'm going to sorry there it is group by terms and so i can see down below what the target index is going to look like i want to do an aggregate by count i'm going to do an average aggregate by average for the ticket price i'm going to do an aggregate by average for the distance and i'm going to do let's do an aggregate by max for the flight delays and if i go ahead and click next it's going to create a job and i'll say create and then i'll see i have my transform job here created now if we and it's as simple as that now i can actually go to one of our other plugins this query workbench and the query workbench is where i can explore my data using either sql or ppl processing language is the language that we developed which uses standard you know unix style pipes to explore data uh and i'm just gonna do just super simple source transform index and head 20. i run that oh and there's my index there's my transformed index the new index that we just created um so great so that's how i do a transform job the next feature i want to show you and i'll build on this one is a new one quick question for you yeah good part i i was about to ask if if this was basically a one-time thing or if it was live and dynamic but i think i briefly saw a screen or two ago that there was some kind of a schedule available to run the jobs is that right it is it's a it's a it's a one time you can do it by api and you'd schedule it but it's essential right now it's not a fully dynamic operation it's essentially kind of doing a fancy re-index so on a very large set of indexes it could be a kind of an expensive operation so it might be something you'd say schedule for once an hour or once a night but uh it's certainly feedback that i think would be interesting to see if we could have this sort of as a more of a dynamic feature to look at down the road okay yeah um so from there i was going let's let's jump over to another new feature that came out in open search 1.0 and this is called a notebooks so we saw a dashboard earlier and you know dashboards are essential to like rafana and kibana and dashboards and what they are is a set of visualizations that are usually linked by a single time window so you'll have a bunch of visualizations and you can change the time window and all the visualizations refresh a notebook is similar in concept of a different with a notebook what you're doing is you're using you know text and visualizations to tell more of a story so the data the data in the notebook isn't linked by time it's linked by you know story so you can organize your notebook like in paragraphs so this is a sample one but i'll just show you how easy it is let's go ahead and add a paragraph at the top and you can do code blocks or visualizations i'm gonna go ahead and do a quick code block that's size there we go and you see here the code block can be markdown sql or ppl i'm just going to do that same ppl statement i did a second ago and i'll say run and now i've got that table i just showed you and i can add another paragraph at the top let's do a code block again and this time i can say um that data i told you about um and then if i wanted to after that i could say just throw a visualization to go along with it and the thing about these visualizations so if you were doing some a a root cause analysis on an event you could describe what set up the event do a graph before the event happened and then talk about the steps do another graph that showed the intermediates and then talk about it and then do a graph after the event and you could store it all in this document to share um but adding a visualization is as simple as uh i can go to my um let's go to the some of these visualizations i have like say flight delays and i specify my time and then i can just embed this in this in this notebook and when i'm done i can see like i've got this output only view where i can actually see all of this data and if i wanted to i could go ahead and save it out to a pdf and share it with somebody or any of my other users and my dashboards that have access to this notebook can come in and see the data in real time and see it constantly refreshed so we have a question from uh jar know about indexed views um you just mentioned in real time can this monitor the data actively is that what you're talking about i'm trying to think if i uh make sure i understand the question can index okay oh so i can transform monitor the data actively it is a these are scheduled events like i think that was similar to what jeff had asked as well and it is a scheduled event so when you run a transform job you can trigger it like say to run on the hour you could trigger it to run faster if it was a very lightweight job um but it isn't uh it doesn't like dynamically then monitor the index and keep it up to date refreshed that that is a cool feature and um if you've heard about a glue elastic used that's one of the things that it does so well it's in preview now but it allows you to do materialized views across data sources and open search will be one of those targets got it i love this idea that that our customers can use us to actually tell stories around their data and kind of interesting that we we've reinvented effectively paper and notebooks and a linear flow in order to to to make that natural and human yeah absolutely uh is there anything else that you'd like to show us today nope i'll keep it you know i just want to give you guys a highlight some of the highlights and make sure folks knew that uh everything was fine it has a new name all your elastic searches still love [Music] you

Channel: AWS Events

Views: 260

Rating: 5 out of 5

Keywords: AWS, Events, Webinars, Amazon Web Services, AWS Cloud, Amazon Cloud, AWS re:Invent, AWS Summit, AWS re:Inforce, AWSome Day Online, aws tutorial, aws demo, aws webinar, #aws, #demo, #grafana

Id: ff5Rnoh8Ty4

Length: 29min 58sec (1798 seconds)

Published: Fri Sep 24 2021