Elasticsearch from the bottom up

Oh thank you so who here is using elasticsearch already awesome so elasticsearch is becoming quite popular these days whether it's for backing your apps search or your web search or having your applications and servers logs all in one central place elastic search is gaining lots of mind fair however it's as a search engine it's quite different from more traditional data stores so this talk is about some how a search engine works and how a distributed one like elastic search in particular my name is Alex I work for found we do hosted elastic search as a service my background from the university is wended within search that's mostly what I've been doing ever since and through found I've been in contact with hundreds of developers and have an impression of what kind of challenges they face when they go from the basic usage of elastic search so this is about the sort of background theory I have great experience from sharing with other developers so the kinds of questions you'll hopefully be better able to to deduce the answer to are things like why isn't my search returning what I expect even if I search for exactly the same text as in my document or how can it make sense that deleting documents doesn't immediately shrink the index but adding documents can cause it to be smaller and why does elastic search use so much memory so before I get into the good stuff I just want to set some context around what we're going to talk about this is sort of like an edge in Reverse I'm going to first go in and then back out later on so when you work with the last exert you have a cluster of nodes and within the cluster you have lots of elastic search indexes that can span multiple nodes through shards and a shard is essentially a Lucene index leucine is the full-text search library elastic search is built on elastic search makes Lu seems awesomeness up available in a distributed setting so this talk is also a lot about how we've seen works and lots of elastic search documentation sort of assumes some familiarity with leucine as well so within a leucine index you have segments which is sort of like mini indexes and within the segment's we have certain data structures like an inverted index stored fields document values and so on and this is where we'll start so the inverted index is the key data structure to understand when you work with search it consists of two parts the sorted dictionary which contains the index terms and for every term you have a posting list which is the documents containing the term so when you do a search you first operate on the sorted dictionary and then process the postings so if you have this quite simple document you can turn you can index it by first lower casing the text removing some raishin and splitting or tokenizing on whitespace so when you want to search for the theory for example we first find the terms in the dictionary and then intersect or Union the postings depending on what kind of search you want so this is quite a basic example but the principle is the same for all kinds of circles first you operate on the dictionary to find candidate terms and then operate on the postings so that terms you generate that end up in your index structure decide how you can search there for how you analyze and process the text is key when you work with the search you really need to understand the text processing that's happening so for example if you want to do a prefix search like in this case find everything with C starting with AC in more realistic case things like Auto completion you can easily do so by doing a binary search in the dictionary but if you want to for example find every term containing the substring our you have to essentially go through every term in the index and this is quite expensive and doesn't scale but it's what happens if you for example wrap white cards around your search so the right approach in this case would be to generate the proper terms there is lots of different things you can you can do when what you have is the inverted index you want to transform the search problem until it looks like a problem where you have to find some prefix so if you want to search for suffixes you can index the reverse text and search for the reverse when there's things like geo locations leucine will convert the data into a do hash which as your prefix is longer means more precision and something similar is done for numerical data because just indexing the string 1 2 3 doesn't really allow for good numerical range searches so even things that doesn't appear to be about string prefix lookups get converted to it so this ranges from the rather simple to the mind-bogglingly complex which we won't really go to get into but it's an interesting story about how some really bright people came up with we can use what's called Levenstein automatons to sort of go through and find misspellings in a really efficient way and they found a Python library that they use to generate some Java code and they didn't know exactly what was going on but the tests proved it worked and the benchmark said it was like a hundred times faster by now it's cleaned up but it's just an example of the really hardcore things Lusine will do to make things insanely fast so when you work with search text processing is really important the inverted index is not very useful however when you want to look up the value given a document like what's the title for document number two so to do that there's other data structures like stored fields which is essentially a simple key value store where we have some a data blob that you want to retrieve when you want to render the search results by default elastic search will store the entire jason source using this but even this kind of structure isn't very helpful when you need to read millions of values for a field such as when you sort or facet or aggregate because you would be reading lots of data that you don't really need so there's another structure called document values which is sort of like a columnar store it's highly optimized for storing values of the same type so this is quite useful when you want to aggregate or sort on millions of valves if you don't specify that you want these document values elasticsearch will use what's called the field cache which means that it'll load all the values for the field in the entire index into memory it'll be quite fast to use but it'll use tons of memory so these data structures the inverted index stored fields document values and certain caches are chunked up into what's called segments so when leucine searches across an index it searches all the segments and merge the results there is a few properties with segments that's quite important first they are immutable so they never change so this means for example when you delete a document uh there's a bitmap that marks the document as deleted and leucine will filter it out for every subsequent search but the segment itself doesn't change so when update for example is essentially a delete followed by a reindex so keep that in mind for example if you store things like rapidly updated counters your index on the upside however Lucene can use all the tricks in the book to compress things leucine is really great at compressing data and as it turns out segments are a great scope for caches and we'll get back to why so these segments get created in one of two ways first as you index new documents elasticsearch will buffer these documents and then every refresh interval which defaults to every second it will write a new segment and the documents will become available for search this of course means that over time you'll get lots of segments so every now and then elasticsearch will merge them together and during this process deleted documents are finally completely removed so that's why adding documents can cause the index to this molar it can trigger a merge which causes more compassion so say you have these two segments that get merged they'll then be completely replaced by the new segment and we'll get back to it a bit later but this new segment will of course have cold caches but the majority of the data is in the older untouched segments at this point which has warmed caches and this is key for elasticsearch real time capabilities as new data comes in the amount of cache invalidation it has to do is quite limited so always happens within a single Lucene index which is a shard in the elastic search index which is allocated across nodes in your cluster so when you search these shards it's pretty much the same as searching segments you search them all and then merge things together but at this point the searching can happen across different nodes and as you merge data here you need to transfer things across the network one key thing to notice is that an elastic search index with two shards searching one elastic search index with two shards is essentially the same asserting two elastic search indexes with one shard each in both cases you are searching across two shards that is two leucine indexes so charting and partitioning into different indexes are two different yet similar approaches to slicing up your data to prepare for handling massive amounts of data you can easily feel I talked about different approaches to this but one approach is so common it's worth mentioning when you have a log like data with the time stamp it's often a good idea to partition it into one index per day for example this will massively reduce the search space when you only need to search today's data for example or last weeks and when you need to delete older data you can simply delete the entire index you don't have to delete have documents marked as deleted and then eventually removed later up and also the indexing performance on today's data isn't affected by the fact that you have all data in other indexes so we have multiple elastic search indexes with two shards each in this case so shards are used to evenly distribute data across one index in this case because you don't you have too much data for one single node to cope so when you plan how you're going to scale it's important to remember that you cannot split a shard you can easily add more nodes and move data move shards around but you cannot add turn one chart into two while this might be possible in the future that the reason is that if by the time you realize you need more shards you probably have a high enough load that adding the extra load of redistributing everything would be problematic so it's important to plan ahead so lots of people try to avoid the problem by okay I just going to make a thousand shards and forget about a problem but then you have lots of duplicated internal data structures like the dictionary and there is also overhead to searching multiple shards so you want to have a balance between having enough and having too few so these shards get allocated to nodes in your cluster you can associate any attribute with the nodes like this node is running in data center a in a certain black or is quite powerful machine so you can do things like make sure there's a replica in every zone or make sure this popular index is hosted under more powerful machines the cluster also has what's called a cluster state which is replicated to all the nodes it has things like mappings which is sort of like the schema that tells how a certain field has its text processed for example it has the entire shard routing table so any node in the cluster knows how to route any search requests so at this point we're essentially back on top abstraction wise so we'll all try to piece things together by looking at how a real search request is processed so say you have this search with a query the query is of type filtered it has a simple term filter and a match vary across multiple fields we also have an aggregation on authors we want to top ten authors as well as the top ten hits and I also specify shard size which is something I'll get back to so this search request can be sent to any cost any node in your cluster that node becomes the coordinator for that search request it'll decide which charts to route the request you based on what indexes you have specified to search across and which replicas are available and so on so it sends the request to the relevant charts but before the search can actually be be executed on the shard there's a certain amount of rewriting that needs to happen elastic search query DSL is sometimes criticized for being quite verbose and deeply nested I actually think it's quite awesome for precisely the same reasons when it's deeply or it's nested structure makes it a lot easier to work with in code you don't have to compile this huge search string and there's also quite a close match between how elasticsearch defines it filters and queries and how the Lucene operators it ends up being converted to works so your knowledge of elasticsearch or leucine will sort of go both ways one exception to this rule however is the match family of queries and the match query is something you're going to become quite familiar with because it's it's the kind of query that will look up in the mapping and see how the text is processed and as we remember how text gets process is really important when you deal with the search and quite a common source for pulling out hairs when you work with with elasticsearch is having incompatible text processing when your index and when you search so when you do not get the results you expect the text processing should be your first suspect but the match match query does not exist in Lucene so it's a elasticsearch abstraction to make different things up quite a lot nicer than having to do it yourself what it would actually look like when converted to the scene is something like this the match is actually converted to a full query that puts together the different fields and the text Holy Grail in this case has been processed it has been lower cased and so on if you were to configure your match very differently say by specifying fuzziness this would be rewritten to something with fussy query in the in the bottom so at this point you have a lysene query that can be run it'll be run on all the segments and at this point it matters what has happened before often you need to use the same filter or the same fields you aggregate or circle across across multiple requests and elastic search will cache this as we remember per segment so assuming these two red segments here are newly created because of new documents or a merge it'll have Kol caches and that filters and fields will need be reprocessed but the majority of the data will is in the segments with warm caches this is sort of the source for elastic search mind-boggling performance when the filter and the fields are already in the cache using them is really fast so our filters are pretty much the same per search they can be cached as a really compact bitmap whereas queries are scored it's not just whether the document matches the document matches to a certain degree so queries are not cached if you need to do the same query over and over again you should probably cache it in your application later so knowing this you should prefer to use filters when you can and use queries only when you need score so this is run on all the segments within the Lucene index which is a char in the elastic search index and the results get sent back to interesting to the search coordinator and the amount of data transferred here can matter a lot by default elastic search will just ask for the IDS of the documents for the top hit because it doesn't really need all the documents sources it just needs it for the top 10 results but this is quite different when you do aggregations it's quite possible that an author that should be in the top 10 the global top 10 is in the 11th position of one of the Sharps that's why we specified a short size of a hundred to make it less likely that that happens of course it's still possible so we always need to weigh and balance the amount of data you transfer to that precision you need and this is inherent in any distributed aggregation so the coordinator has all the data it merges it together asks the charts again for hey can you please give me the source for these documents and send it back to you as the user so at this point we have been through we've looked at the inverted index and see how the index terms you generate largely dictate how you can search and that the text processing that generates these terms are quite important we have looked at how a search happens by segment and how a segment has several data structures some used when you search some use when you aggregate and so on we discuss the consequences of these segments being immutable and that this can affect indexing performance for example when you need real-time or when you need great indexing through but you may want to for example add just the Refresh interval so we don't constantly merge new segments we've seen how a shard is essentially the same as a separate Lucene index and that the elastic search index is come is generally just an abstraction on top of a leucine indexes and you can combine them either in as shards in one elastic search index or across multiple indexes and at this point of course across nodes in your cluster it's a distributed search engine you can easily add nodes but you need to also be aware of the kind of data being transferred between the nodes as you search so this was intended to be an introduction to different things I hope you want to learn more about the talk is based on an article of the same name and you can find it in foundation which is our article collection about elastic search we try to keep them as helpful as possible for anyone using elastic search is just it's not just for found customers there's also a elastic search shmita later today it's here around 6:00 so if you want to learn more about the last six urge I hope to see you there and if you have some questions now now's your time thank you no questions well I have a question about replication so for example if I have some important documents that I would like to search even if one of the notes also will not go down what's the recommended way to do it in elastic search you want to you have documents index in replicas and I know it goes down well so I'm adding a new document to the index what's the recommended way to add it in such a way that one node failure doesn't take downs doesn't yeah okay so this talk wasn't that much about elastic search in production I used to do another talk about it there's lots of different things to keep in mind when you run a cluster of a distributed system you want to have a majority of nodes available for example to avoid things like split brains you want to have a replicas available in different for example if you're in Amazon in different availability zones to make sure that you always have a replica available when errors happen and in a distributed system failure is guaranteed to happen so in any production configuration you should have multiple nodes in in running on infrastructure that's not have any common failure points you should make sure you have in bigger production clusters you should have dedicated master nodes for example and you need to have at least three to have a majority in the event of failure a quite common setup is to have two nodes in your cluster one with one replicas each but when there is a network partition between these you cannot have a majority when you have just a single node and your cluster is composed of two so there's lots of different aspects and I'm happy to talk more about the last exertion production after so just come find me alright thanks thanks a lot for the fascinating talk would you say what would you say about the code base of elasticsearch is it worth reading through is it how's the quality would one actually learn something by looking through it eh yeah it's it's quite a complicated system it's I think the code quality is generally quite high compared to to other search systems I've read we've seen has really really high quality code ah it's it's a bit higher than the lasting search I'd say but elasticsearch is still quite quite good it's you can see the fact that you now have tons of new developers which is good but it's a code base either I'd recommend looking at its future alright yeah ok thanks a lot one over there Thanks so if I recall it right Lusine has this precedes formula to rent documents and how is this working between charts our deck how is the ranking working between charts if you like having documents and turn frequencies and inverted document frequencies just like you um yeah I'll try to find the relevant slider still have a few minutes so when leucine is scouring documents it takes into account things like the frequency of the term for example the words like if the and in are don't add much value but more rare words are considered to be more relevant so it uses sort of like it tries to find real words in your query and prioritize them while sort of not caring too much about the really common words but of course these frequencies can be different across the different charts so that's it's possible to tell elasticsearch to as an before the search itself happens have all the charts report the true frequencies so you can get more accurate scoring but when it comes to to actually ranking and scoring I would look pay just as much attention to things like function score where you can boost based on for example filters you can say prefer new documents or preferred documents within a certain section of your content and so on so do not just judge relevancy out of the default relevancy that Lusine gives you but also look at all the tools elasticsearch has to tweak your scoring do we have any more questions it is it measurable - to compare it if you like just have a single Lucene index and you put all the documents and them single design index and then you have the same index across charge do you get the same results or is it different because you have statistics between it so when if your data can fit in a single shard and you don't need to scale it for example you should prefer to have a single chart storing it in two charts will be more than twice as expensive so usually you want to prefer having fewer charts when you search multiple charts these frequencies can differ between so you can get different results so mixing everything into just a single shard can yield different results from having it into usually it shouldn't be huge differences and again you probably want to also look a lot into a function scoring for example thank you any more questions people are hungry yeah thank you very much Alex please give around you

Channel: EuroPython 2014

Views: 211,068

Keywords: python, euro

Id: PpX7J-G2PEo

Length: 36min 54sec (2214 seconds)

Published: Thu Jul 24 2014