Learn By Building: Language Server Protocol

I think the best way to demystify a topic in programming is to just take the topic and build a toy project in it it doesn't have to be perfect it doesn't have to explore every single aspect or every single edge case but building something can give you an Insight that you really were missing beforehand and so today we're going to build a language server from scratch in goang we'll create a brand new go project and we will not add a single external dependency to the project we'll write every important aspect of the protocol encoding decoding the types how to communicate and even how to handle these different messages we'll hook it up to neovim and we'll even be able to see the completions and the Diagnostics and the suggestions that we're going to build inside of an editor live let's just get right to building people often ask me how did I learn about the language server protocol and I know this is kind of sort of like anti-marketing for my own video but the way that I learned about it was just reading the specification I didn't find any articles or videos or anything like that about it I just sat down looked at the spec and started focusing today that's how we're going to implement our support in goang it's true that I do have some experience with the protocol so it's a bit easier for me now than it was the first time around but I just want to show you that it's really possible to take a specification like this build it block by block and end up with with a working product I know I'm basically saying read the friendly manual instead of watch my video but that's my best advice so that's what we're going to do today and that's where we're going to start the language server protocol communicates with a protocol that is built of two main parts a header and some content the header is effectively built by containing these different header Fields the primary one of importance that we're going to deal with in this video is content length because content type is optional and content length says hey here's how many bytes are going to be in the content section of our message now generally speaking a language server communicates over standard in and standard out but that's the transport mechanism so don't get it confused the protocol is messages that look like this some content length with the amount of bytes a carriage return in a new line a carriage return in a new line and then the Json it is not about how that is transported you could have LSP on built on carrier pigeon or smoke signals your response time your latency and your throughput would probably be terrible but it's possible and it's legal the main thing is does your client support this transport many editors support standard in standard out and TCP as the main ways of communicating with an LSP and the by far the most common of those to is standard in standard out so the first thing that we're going to do is we're going to build a way to encode messages now this is the simpler part decoding is a bit more complicated that's why we're going to start there and the very first thing that we're going to do then is we're going to go ahead and go to our project and in go we just make something like go mod anit educational LP and you can see that we have this file and it tells us the module and which Go version we're using what's cool is like I said we're not going to add any dependencies to this so that's what that's going to look like when the project is all done to start creating our RPC we're going to make a new package called RPC and inside RPC we're going to open up a new file called RPC Dogo inside of here we're going to write our first function which is going to be called Funk encode message and inside of here we're going to say okay what do I want to encode I want to be able tocode some message type and I want to return a string this is just like the idea here where we're going to have a message come in we're going to serialize that to this Json then we need to know how many bytes that is and we're going to create this entire message so to encode this message we need to first turn whatever our messag is into our Json blob right and so we're going to do that by doing something like this content error and we're going to set that equal to the Marshall value of message this is effectively like json. stringify if that's more your speed and if we have an error here normally we would return it but in this case it's kind of like if we can't encode our own messages we are in huge trouble we should just stop if you were writing maybe a more serious non-educational LSP you might want to handle this more gracefully but instead for us we'll just Panic with the error and so now what we need to do if we think back to this spec is we need to say okay this is the content section so we need to figure out how many bytes this is and then put that right here and otherwise just return the string so we can do that pretty easily in go with s printf which just like string print formatted string so it's like kind of a long way to say that but that's okay and so we need to say content length we're going to say percent D that's the way that we say a number we're going to do registered nurse registered nurse and then we're going to send these bytes here and so we can do that by doing length of content and then the content itself using this we're going to create this exact same message that we have here but just to verify this we'll write a simple test case to ensure that we're doing everything nice and happy here so to write our tests we'll just use Go's buil-in test Library if you're not familiar that's okay we're just going to use some very basic features from it today the simplest way to do this is we can create a new file called rpct test. go and we can make it of package RPC test we're going to write some function that tests encoding and we're going to make sure that we can actually encode a simple message now we can actually just write out the expected value by thinking through the specification this isn't covering every case but it at least gets us to have some confidence that we're on the right track so we can make some expected function here and when we do this we can actually say okay this should look like something like this we have content length some number our registered nurse and registered nurse and then now we need to put some message after this and in this case we can just make up our own it doesn't have to actually be something from the language server protocol we're just implementing the RPC right so we can do something like encoding example struct here and we could say testing buol all right great so what that's going to look like when we encoded is going to be something like testing true and here and now to find out what that number is we can just count the bytes so that should be 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 so we go back to here we can change this to 16 so that's what we expect the message to look like what does it actually look like well we can do this and we can just send an encoding example with testing is true now all we have to do after that is just check does expected equal actual if not then we're going to fail this and say expected and actual and we're going to pass our expected and actual values once we have that all we need to do is run go test on this project and you'll see oops we messed something up that's because we said testing here should be a capital T so we'll just make sure that our test string actually has the right values when we run the test again we'll see that it passes so that that's the very first part we now have the ability to take some go object and encode it into the Json RPC format now we have to do the slightly trickier version of that which is take something in the format and decode it to an actual value so we're not exactly sure what the signature of decode message should look like yet but we're going to iterate towards a really nice solution so we're going to start with funk decode message and we know that we should take in some content some message maybe and this time it's actually going to be an array of btes that's normally what we use in sort of this Json serialization world right and we're not sure what we're going to return yet so let's not do anything that's where we're going to start what we need to do next is we need to say okay I need to find out which part of this message is the header so I need to find this section and go actually has a really nice function inside of its standard Library called bites. cut so what we're going to do is we're going to get bytes. cut we're going to use that and what does bytes. cut do it takes some slice and it takes some separator and it gives us all the bytes before that all the bytes after and whether or not we found it so we can just take our message and we can do something like bite and we want to do our separator here now we actually want to look for two of these because we're looking for when this header ends so we're sort looking for that chunk of text right there when we do that we get our header and we get our content and we get whether that was found that's how we know which of those we've already done now if we didn't find this we definitely need to return an error so we can go if not found we're going to return an error return error uh. new and let's just say did not find separator right so that means okay we actually need to return an error from this function okay and then our success case is that we have no error that's great that's part one that gives us we now found this line so now we know somewhere in this header here we have this content length so what we can do now is you see okay I have this slice of bytes here that look something like content length something number here and then maybe other text we we're not really sure but in this case we're just going to sort of handle the normative case which is that where we just have this as our header what we need to do next is we need to say okay how can I turn this into a number so I can know how many more bytes to read well the first thing that we can do is we can say okay the the content length bytes is going to look something like our header and we want to take everything after the length of content length colon and the space right so we want to make sure we're getting everything after this section so this is kind of like saying okay move us past this section here and give me to the end of that slice all right now we have the bytes what we can try and do with those bytes is we can say can we turn those into a number so we can do that by saying something like this content length and that's going to equal stiron now if you're not super familiar with this this is okay but this is basically saying asky to integer and what we're going to do is we're going to take those bytes the content length bytes and we're going to say can please try and turn those into an integer but go Lan saying hey actually can't do that this also could possibly return an error so we say okay that's fine if we can then do if our error is not equal to nil then we'll just return that new error but otherwise now we have the actual length we're going to need to do more in this function but first we can just test that this is working by going up to the beginning of this function and instead of just returning an error let's return an INT and the error as is saying this so in our error cases we'll return nothing here and then we need to return our content length like this we have one little error here because goang tells us hey you haven't used content that's okay too we'll get to this momentarily all right so now we can do is we can write a few tests or just one test really in this case to make sure that we're getting that actual value out of the message that we're trying to decode to do that test we'll just go back to our same test file that we had before and we'll make a new function test decode and we'll use the same testing strategy that we were doing before and with this we can just actually take the same sort of message that we had here and this is sort of like incoming message and we're going to try and decode this and hope that we get out 16 so we can say content length air RPC decode message incoming message and to deal with with this we need to first check okay if error is not equal to nil then what are we going to do we're going to t. fatal with the error right so if we had some problem decoding this then we're going to air right away then we also need to check did we actually get the 16 value out that we wanted so we can check if content length not equal 16 then we're going to do t. fatal F and we're going to say expected 16 got content length and then otherwise this passes and ideally if we wrote this correctly we're going to see perfect we passed this test as well so now we're able to get out the content length but probably should do something with the content itself but the question we need to be asking ourselves now is what are we going to do with the content now we're going to do quite a bit later but the main thing we need to be focusing on right now is this method key inside of the Json struct every single one of the messages that we send or receive will always have this method and we're sort of going to index off of that to decide what else to do if we get a message that says hey a text document updated then we need to update our internal State representing that text document and we'll need to encode and decode differently so the primary next goal that we need to do is we need to say okay how do we pull out that method from the content great question what we're going to do here is we need to create some type of Base message and normally this has Json rpc2 which we don't care about and sometimes it has ID and sometimes it perim the only one we care about right now is Method and so what we can do is we can create a new type and we'll just call it base message and the only thing we care about is this meth method string and we can tell go to look for that inside of a lowercase m method field we're going to try an unmarshall that right that's sort of the opposite of Marshall which you can kind of think of like of string or whatever other sort of of common pattern you're used to we're going to do here is we're going to say base message base message we're going to try and encode this or decode this sorry and we're going to do something like this Json onm Marshall and we're going to say base message and then if uh if the error is not equal to nil we're going to return this error otherwise now we have oh and of course we need to actually pass the content so this would be content up until the content length right so this is sort of the first content length bytes we can get rid of this now and now if we do this this means we know for sure that we've actually unpacked this message but we're not doing anything with it we should inform the caller of what this method is and so we can go here and we can say all right let's do Bas message. method and we're going to return that we say okay well now we need to actually update the return type so all the other places handle this true we can go up to the top of this function here and we can go here and then we're going to say we're going to return a string as well and now it says we don't have enough arguments so we're going to go handle all of those and make sure that everything's happy when we go back to our test we're going to need to update our test because now this is going to be saying method unfortunately the example one that we did before with testing doesn't have a method field so let's just go ahead and change what we have here to say method and that's I think one character short of testing te e s i n g yep and Method yes so we need to decrement this by one and go to our 16 here and decrement this and now we need to say if method not equal to we should probably shouldn't say true cuz that doesn't really make any sense we'll say hi because that's the same amount of characters method not equal to high then we're going to t. fetal F expected high got here and we can check and make sure that that works just fine we run back to our go test we run the test for this and everything passes so that's wonderful we're able to now unmarshal or decode if you want right that message and the correct length and find out what the method is later like I said we're going to be sort of indexing off of that method to decide what to do next but this function still has one kind of goofy little Quirk which is that we're returning the length of the content but we actually don't know what the content is because we're going to need to decode that later so instead of returning the content length we can go back and we say let's actually just return the bytes that are in that content we can go ahead and change these spots all to say nil and then instead of returning the content length here we can do that same thing that we just did a moment ago to say here's all of the bytes that we want when we go back to here we can no longer say okay the content length this type is actually a by slice we need to say that this is actually the content and then content length is just the length of content with these two changes we should be able to run the tests and they'll pass once again now the next thing that we need to do is to start hooking this up so we can actually run something and if you think about how a language server Works effectively it's something that can be started whether that's a binary itself or some script and that's run by some interpreter on the system and it's going to start up and what it's doing it's going to say hey client send me something I I'm waiting I'm waiting it's going to wait and it's going to get a message back and once it gets that message back it's going to reply or send notifications or do whatever it is so the next thing we need to do is we need to create an executable and go and then we need to be able to start listening for messages we're not going to do anything interesting with them yet but we're just going to be sort of like waiting on the line and listening and listening and listening and we'll just maybe add some logging so we can actually see that those messages are coming in so so the way that we're going to do that is we're going to make a new executable and go by opening main.go we'll create a new package Main and a new function main that we're going to use inside of here we can just double check that everything's working right by printing out high and we can run this with gun main.go you see that we got our high great so we need to now build that sort of main I'll say event Loop even though that's not really what we're doing right we're effectively just saying Hey I want to keep on reading from standard in until I get a new message and then I want to handle that message okay and that would look something like this we could have some function here handle handle message and we're not sure what we're going to pass here right now so we'll just say any and we don't know what it wants to return okay and we basically need some way to say okay for next message we want to handle message right and we don't know what that actually is going to look like what we can do though is use a really really convenient package go standard lib inside of buff so we're going to make a new buff scanner and scanners are cool because they do exactly this they say hey give me something to read from and then I'll read until you get a new message in this case we're going to read from standard in so we have our scanner like this and we're going to say basically while the scanner right we're going to look for scan this is going to say hey keep on running this until we're ready now the problem is right now we don't know how to get message well we can actually pull that out of the scanner and get the text we may use bytes later but that's fine so that's sort of our Loop right problem is the scanner doesn't know how to read LSP messages and so we need to basically tell it hey scanner here's your new split function we need to implement that to tell it when to split inside of its reading so we're going to just Implement something that matches this interface inside of RPC so we can take advantage of this nice Library inside of ghost standard lib so when we look at this split function we see ah split takes a split Funk which looks like this a split function takes two arguments a slice of data and whether we're at the end of the slice and returns how many bytes we need to advance the token that is sort of part of these bytes and whether we've received an err now in this case we're not really going to care if we're at the end of the file or not because this is a long running process we're only concerned about sort of active communication between the LSP and so we just need to check hey you sent me a Content length and you told me how many bytes that was going to be have I gotten that many things yet and if so then we want to advance by that amount and return the bytes that we just should have read let's go ahead and do that so I'll put the type signature here so we can sort of keep this in mind of what we're trying to implement and we want to do something like this funk split and this is going to take some data just like we've got here whether we're at the eof which we're not actually going to use and we're going to say how far we're going to advance we're going to say the token itself and we're going to say whether we've returned any errors okay those are all the things that we're going to do for right now our simple case would be we say I well I'm not going to advance any I'm not going to send you a slice and there's no error this would basically just never do anything right that's kind of like our base Case Model we should be thinking about out here but we know basically what we need to do we need to say okay let's see if we have a Content length here with some amount of bytes if we don't have this then we definitely can't do anything we need to wait until later so we can basically take the same idea that we have here and we'll just copy this because it's pretty simple later you could split this into a separate function if you wanted but this all sort of the same idea here right and then instead of having message we're going to call this data so we're going to try and split this and if we don't have this then we're just going to return nothing and we can just say zero nil and no error this isn't a problem this just means we're not ready yet so we're waiting for more information right so we can just say no this is going to say just just wait till later and and we'll get there now we're going to say okay we were able to split the incoming data by our two sort of carriage return new lines now what can we do well we need to say all right can we actually get a Content length out and if we can then we're going to say okay if we can't do that we're going to return this we are going to return the error in this case because this says I actually don't know what to do with this message you told me you were going to send this content length and a number and you didn't send a number here if you send ABCD in place of a number I I'm just done forever we're going to send an error and we're going to quit out so that's why we pass the error here now what we need to do is we can say okay if the length of content is less than the content length now we're also not ready this means we haven't read enough bytes from our incoming stream yet so we need to wait until later so that's okay this isn't an error either this just says we're not ready give me another chance later right this we're just going to wait but if we have all of these items now we can actually say hey scanner in main go we're going to say hey you can advance you can go a little further and give me this message so what's that going to look like we're going to say okay we have to do the total length in bytes that we've done thus far and so that's going to be the length of header we're going to plus four four why that's because of 1 2 3 4 okay that's sort of our separate Ator right plus the content length that's our total length that we need to move forward and we can put that here as total length now we can say okay we only give them up until total length here and then there's no error once we have this split function we can go back to main and we can say hey this is actually RPC dosit and this satisfies the interface so this says okay I need a function that looks like this you have a function that looks like this and now we're going to be waiting on standard out for a new message and once we get that message the scanner will give us the text and then we can handle it the first thing we're going to do is we're just going to add a little logger to be able to see that the messages are coming in and confirm that we're doing that while we connected to neovim so the first thing that we need to do is let's actually add just a touch of logging here which will allow us to make sure that we're writing to a file when we want to log something because we can't print to standard out that's the channel that we're using to communicate with our client we could print the Standard air but it's a lot easier for us to sort of check and make sure things are working if we're able to just write it to standard out so a really simple way we're just going to write sort of a really hacky small logger using Go's regular logging sort of functionality and just do something like this get logger file name with a string and that's going to return some log log logger right like this right and what do we need to do here is we need to open up some file so I'll just say log file air and we're just going to we're just going to open the file it's going to exist for the whole time we're running we don't have to worry about closing it it'll get closed when we're all done and if you're not super familiar with everything we've got going on here which I always sometimes forget too and I'm going to copy from my notes is we're going to open from the file name then we're going to say all right I want to create a file I want to truncate this file so that we can sort of get a new one every time we run this and I'm going to just make sure that it's read right and we're going to open this so that everyone can read this and write to it it's fine it's just on my computer and if we don't have an error here we're just going to say Panic hey you didn't give me a good file okay it's fine this is just for educational purposes right all right so now what we need to do is we can just do return and we're going to return some new fi new logger here with this log file and then we're going to say all right this is like education NSP classic really easy to say and we're going to log some stuff like the date and the time just so we know that we're sort of like getting updated stuff here and we're going to shorten the file okay so this is sort of just like a really quick and dirty logger that we can do so that we can instead go back to main here and we can go something like hey I want to get a logger and I'll just do get logger here and for me I'll just put it at home TJ De we'll just put it in inside of this project educational SP and we'll just put it as log.txt okay so that's sort of like the logger that we're going to use and now we can do something like logger dotinfo or log or print I guess sorry print line and we'll say hey I started okay so now if we run this with like gun main.go we should be able to see in our log. txd it should say hey I started that's great okay so this is where we're going to print a bunch of our information from the LSP to confirm that things are working before we actually know what to do with handling messages so like I said we don't actually know what to do to handle the message so for now we'll just say that this can be any message and we'll just make sure we also pass in the logger so that we can actually you know use this logger inside of here we'll pass our logger like this and we'll just say that we got a new message so we can say logger do printline and we can just put the message here this will make sure that every time we get a message we just print that we got something later we'll sort of trim this up but this at least lets us know hey are we decoding messages and then passing them forward correctly through our scanner so the next step is let's connect this to neovim and see what happens so connecting this to neovim is actually super easy we just need to create a new file that's going to get loaded when we start neovim and we want to tell neovim hey can you just like start something this will make sure that we actually able to start the LSP that we want because remember a language server is controlled by the actual editor okay it it doesn't just start up on its own generally speaking so we need to tell NE of them hey can you start this client and we need to do something so first we need to tell the of them hey here's where you can find this thing and for me that's just going to be home TJ re just like we had before education LSP and it's just main okay we're going to build it as Main and that's fine that's the command that you need to run to start this if this was written in JavaScript or something then you would have node and the path to the bundle Javascript file or something like that if it was python same idea right so it's all sort of the same idea and go we get a nice binary so that's all we need to do we can also tell it a name which is pretty nice and we'll just call it educational SP just so it's easier to sort of track and then for neovim we also want to do something like require using an onattach and for me I've got an onattach function already and there's nothing too crazy about that part but this just lets me use my key bindings okay and we can ignore the error uh that we've got here so what this does is says okay I've got some client here um if not client then I'm just going to return here I'm not going to try and do anything else and we'll just let me know hey you didn't do the client thing good now right now that's what's going to happen because we haven't run go build now we should have a main file right here right this is an executable that we can run when we run like this main it's going to try and do something okay now that's not super exciting yet but that's okay we're going to actually do something exciting with it soon the only other thing we need to do for neovim here is we need to tell it hey I want to connect for something so in this case we can just do something simple and we'll say we want you to connect when you're doing markdown files and there's nothing too crazy here if you're not a neovim fan you can just close your eyes for a second and pretend this is Json config or something like that pattern is markdown so this is going to say hey when you have a markdown file what do I want you to do well I want you to go like this I want you to then be able to say let's do Vim LSP buff uh attach client I believe and we're going to actually pass the current buffer and the client here okay so this basically says all right I want to whenever I open a markdown file I want to smush this LSP and that file to together later that's going to make a lot more sense and we can test this right now by opening up we can just go ahead and go to our same spot and we'll open up the readme file and what we should see here is that now we have our very first message right it says hey I started and now here's the message that we received from neovim you're thinking how do we know what this message means we're going to go back to the spec so now that we are receiving messages we can actually try and do something interesting with them which is really exciting so what we're going to do is we're going to go to the spec we're going to check out this idea of what is the server life cycle what is going to happen and what we see here is the very first message we're going to get is an initialize request okay well one I don't know what a request means so we're going to need to handle that but we can see in our log right here sure enough the very first thing we get is this method initialize so we're already see seeing right neovim is sending the messages as described by the life cycle right here so now we need to figure out okay what do we do with requests and this is basically sort of the main idea here is that we're going to have request types and we'll write this type shortly and we have response types and we always send a response for every request even if it's an error or something like that will let them know it's under sort of the same idea of request response we also have later notifications and those are sort of the last type of message that we're trying to handle so let's write out a few of these structs in go so that we can start trying to unpack right to decode these messages into proper go types that we can work with we're going to make a new package now because this is not specific to Json RPC we're actually now going to do things specific to LSP right so we're sort of splitting out this idea that we have this way of sending length encoded Json messages that's sort of just like the protocol sort of base layer but now we're going to start building things on top of that that's the LSP the language server protocol itself so let's make a new package called LSP and inside of here we'll just call something like message go this is all going to be part of a package LSP and we're going to start defining some structs that map to these ideas in the specification the first one that we have here is this idea of a message and it includes this Json RPC string that always says 2.0 great I don't care we're not going to even try and encode this or decode this in any way sort of this is sort of just pointless stuff that we really don't need to be managing but that's okay we'll still show that in each of the other messages so we have this type and let's say we have a request and it's a struct it's is going to have an RPC which will be a string and we'll say that that's encoded by doing Json RPC so always be 2.0 okay so it's not exciting we're not really going to do anything with it like I said but but it's there okay what is specific to the request message well we have some ID that's an integer or a string so we're going to have ID and I don't remember I think neovim always is going to send ins if you have something that sends ins or strings it's a little bit more complicated to decode in go but it's the same idea here so we can just say ID and then we also need to have a method it's always going to have a method which is going to look like this method we want that to be public so that it's accessible other places and then there's going to be some pams now in this case we could put pams here and put something here but this actually isn't useful we will just specify the type of the pams in all the request types later okay so we don't need to worry about that part sort of right now we can just move on past that and we'll we'll do that when we actually have a request and now we have our response message a response can have some type response struct okay and it's also going to have an RPC because we have to send Jason RPC 2.0 every time Json RPC same idea we're going to still have an ID but it could be null so we can say ID and we can actually encode that by doing this ID omit empty like this and if we have that that's going to be just fine so it's going to sometimes send the ID but if we don't have anything it's going to skip it and then we sometimes have a result or an error in this case we don't need to handle each of each of these because the same idea here is that we're going to instead of specifying what the prams or the requests are up here we're going to say okay this always has some result or some error and each of the types can sort of say what those are separately this is because go doesn't really have inheritance it just has struck sort of like coupling and we can put them inside and they're sort of hidden inside of there but they're not actually inheriting them so that's okay we could turn all of these into interfaces and we could make a bunch of different stuff but this is go leg let's just like write the code and move on right the last thing that we have here are notifications we can see what notifications are and you'll not they don't have an ID right so a notification just looks like this right and it just has the RPC ID string Json Json RPC and it has some method here so that's the only thing that it has method string and this is going to be some method like this so that's sort of our base structs and what we see when we're looking at life cycle messages here is the very first thing that we should be getting is this initialize request and the specification tells us what that request should look like it says hey you're going to get something that says method is initialized and if we look back here we see sure enough we got method is initialized it has some ID because it's a request and then it has some params and there is a ton of stuff in here that we actually just don't really care about handling in our educational SSP here right if we were trying to make sure that we could handle every combination of the ways that you could snc documents and types of requests that could be made okay we would do a lot of stuff with this but in our case we just really don't care right that's sort of way too in the weeds for what we're shooting for here but what you're going to see is we're going to see this pattern over and over there's going to be a request it's going to have some method and then it's going to have some prams and these prams are going to just tell us a bunch of things about the type of request that it is you can see right this says oh it's going to send some client info so if we look for client info we'll say oh hey here's the neovim version and the name is neovim okay right so you can see how these sort of map exactly back and forth between the spec and the messages that we're getting so what does the spec tell us to do it says the initialized request is sent as the first request from the client to the server and so then what we should do is we should respond with an initialized result so we need to send this back to the client but first we need to figure out how to properly decode this initialized message so let's go back to our scanner here and we want to say actually instead of getting the string here let's get the bytes so we have this message and it's the btes right what we want to do now is we say hey and we just write something a little bit ago that let us I don't know decode a bunch of bytes into some useful information why yes TJ we did so we can do our decode message and this takes some message here what do we do with this is going to return a method contents and the error and if we have some error we can maybe just continue until the next message right let's not let's not sort of panic here and be all done we'll just sort of we can log uh and we'll say hey we got got an error and we'll just do something like this right so we can say this that's okay not amazing and we'll continue all right now though instead of just passing this message directly let's instead pass the method and contents and so we can say method is going to be a string and contents is some slice of btes and we can say we received this right so let's say something like this we received message with method here and let's do this method okay now we're sort of handling the message as we go and if we build this again and we open up a new neovim here we close this we open this up again we'll see now we received message with method initialize that's great that's exactly what we're hoping to see so now we need to say okay we have this initialized method we need to be able to read this out we need to be able to turn this into a g struct that we can do something with now in this case like I mentioned we really don't care about a lot of this information right it's just a lot of extra stuff we don't care about but just for the sake of being able to pull these out let's go ahead and pull out from here the client info right so let's just pull this out so that we can get that name inv version to confirm that we are getting what we expect so to do that let's open up our LSP folder again and let's make a new thing just called initialize Dogo and what we're going to do here is this part of package LSP we want to say all right we have this new struct right we have this new initialize thing and so we can say type initialize request it's going to be a struct and it's going to have a request inside of it if you're not super familiar you're going to see sort of this pattern emerging here inside of go that we can say this struct also has these fields okay and so now we need to say though hey it's got some prams and that's going to be an initialized request pams which we haven't written yet and when we encode this we're going to encode it as prams like this and says hey you haven't done this yet Dre I haven't so now we can say what do we want to put inside of these pams and that's where we have in the spec it just tells us right here's the pams and here's our initial pams okay an initialize params has a client info so we could say client info client info and that can contain something like a name in a version so let's just add a new thing called client info and this is going to be done like client info like this and this is optional so we should probably do a pointer just in case but uh we'll we'll do it like this is okay for now type client info and that's going to be a struck that has name String Json name and we can also do version string and we're going to encode that like version okay so now what we have here is we said all right we're going to get an initialized request it has some prams these prams look like initialize request prams and obviously there's tons more that goes here if you were making a proper LSP for a real language you would want to handle all these different combinations we really don't need need to do that so what are we going to do with this now we have some way to say what we've got but we don't know what to do with it let's go ahead let's go back to our main what we see here is we have a method and what the specification promis us it says Hey whenever you get something that has method equals initialize I promise I promise that the pams are going to be initialized pram so we say okay well can we just decode these right can we just do something to decode this into the right type so the very first thing that we can try and do here is we can write something like switch method we can say case initialize and what we want to do with this is we say okay we have some request and this is going to look like lp. initialize request right this is that type that we just made and now we're going to do that same idea that we've done before where we says something like this okay we've got an error and it's going to equal on un Marshall and we're going to say what do we want to do we want to pass in the contents right this is just the Jon that we're trying to un Marshal we already handled that earlier in our decode message and I want to put this in request and if the error is not equal nil then we're going to do something like let's just logger and we'll print hey we couldn't parse this okay and we'll send the error with it all right but now we know we have a request so let's go something like logger print F and we can say connected to and let's do percent s percent s and we'll do request pam. client info name request prams client info version okay and we'll sort of put both of these here right this is sort of what we're going to say so ideally now what's going to happen is after we build this and we connect neovim again we should hopefully see those things pop up in our log and when we go back to here we'll say sure enough we've got our neovim and the version that we're connecting to that's really exciting right hopefully you can start seeing okay we we waited for a message we got our first message ah initialize okay what are we going to do with that we need to parse it like it's initialized for Rams okay we did ah look we have a client info and there's more stuff here right we have a client info what can we do with that oh we've got a name and a version let's print those and see what we've got but the life cycle says we need to do something and what do we need to do we need to reply with an initialize response let's do that next and here's what that result looks like and so recognize here though if we go back to our initialize we have our initialize response we can say here right that's sort of this response to this message what we have is that we have a response message here and it's result key so that's what this result colon means here is we're going to have result and that's going to be an initialize result and we're going to encode that as result like this and it says we haven't figured out how we're going to do that sure we you are correct so we need to make a type initialized result struct and now we need to add these new fields that we've just found out about so there's a capabilities and a server capabilities we're going to have to put something here so we'll say capabilities and server capabilities right with Json capabilities which hopefully I can spell correctly we'll find out later I guess and then some server info here which is actually basically the same exact thing as the client info so we can say server info here and we can do the same idea here for server info and we're going to once again encode this as server info like this and so we can build now this server capabilities let's just put nothing here for right now cuz we'll sort of come back to this in a sec and that's a struct with nothing inside and now we can do type server info and that's a struct and we can say same things we've been saying which is name String name and version string version okay and so now we're able to encode these two things here and this is what we need to reply with Okay we need to reply with this idea um but before we do that we probably should just at least look at what these server capabilities are if you're not familiar with this concept of capabilities this is basically the way that a client and a server can negotiate the features that they're allowed to use and we're basically going to give the most minimal set of features because we're not interested in handling like incremental SE sync and all the different kinds of things that could be given to an a language server instead we're going to handle just the main ones to once again give you a picture of what is happening so the first thing that we're going to be able to do is we're going to handle text documents syn but we're not quite there yet first we should at least try and send this message back and make sure it's getting sent and then we're going to be able to start providing some more capabilities and work our way towards our server doing something at least so that very first something is going to be replying with an initialized response so let's make something for new initialized response and we're going to need an ID and we're going to return an initialize response so that's going to look like this and we can make this initialized response like this and it's going to say let's fill the initialized restruct fields and we'll fill this and we'll fill this as well and here we're actually just going to pass pass the int and we can say Json RPC uh we can say 2.0 for this and we can say ID here so this is kind of like our response okay these are the things that we're going to keep that are the same between all responses and then what we have here is we need to say hey what are the actual things that this particular server is so we can say educational SP and we can say 0. 0.0.0.0 do- beta1 final right kind of like turning in your homework assignments at school right and so this is sort of what we're going to be able to send back to the LSP and we don't know yet what to put inside of This Server capabilities right we're going to go over that momentarily but first let's just try and reply with this initialized response so remember we're back in main go and we have this message and we know we have an initialized message so what we need to do here is we need to say hey let's reply right that's sort of the main goal of what we need to do now but we don't have any way yet to do a reply so we can just say for right now and we'll refactor this momentarily right we want to do something like this we have our standard out and we want to write to it so let's say our our like writer is something like os. standard out standard out okay and we want to write to this so we should be able will say writer. write and we want to write a a bunch of btes that's all we want to do okay well we kind of know how to write things we know how to encode our message right so we can say something like this Let's do let's get our message and that's going to be LSP new initialize response and we're going to get that from the request. ID okay now we have this message which is an initialized response and we can say hey here's the things that we actually want to write so this is like our reply and this is LSP do encode message or sorry RPC do encod message and we're going to encode the message so now we have a string and we can just say right here we're going to do this with reply and just convert it back to bytes once we do this this is going to take the standard out of the current process which is connected neovim and reply back with this sequence of bytes let's see if this actually works right and we can write something here like logger print sent the reply and we'll see if neov is actually getting this reply and if it knows that we've responded to check that we're going to go back we'll build ourselves up a new executable we'll restart neovim and then we'll open things up again what we should hopefully see here sent the reply then what do we see we received a new message with initialized notice this additional D not super obvious at the beginning and if we go back to here and we look at this overview what we see is we're going to do this initialize and then respond with initialize once that's done we then later receive an initialized notification this means that the client is all set it's figured out how to talk to us and we're all good ideally then we should see is that now we have our educational SP neovim knows about this it knows that it's connected to this LSP and that it was able to talk to each other this is really exciting this is the very first step this sort of the first confirmation on the client side that yeah we' we've basically had our handshake and we both agree yes you're a language server I'm a language server client and we know how to talk to each other so now let's start doing language server things and that's what we're going to do and so like I said we're going to need to go back to these server capabilities so that in that handshake we can tell neovim hey I know how to handle changes in a file currently if we look at our log and we start doing something like editing this we're not really going to see any additional messages that's bad the language server needs to know that you've changed things in the editor I mean think about it if you if the language server only has access to saved files it's always going to be behind the editor State the language server sort of primitive building block is this idea of text Doc document synchronization which we need so that whenever you type a key immediately that key and the updated buffer contents the updated file content is going to be sent to the language server so it can begin analysis on the most upto-date version of the file you don't want the saved version from five or six seconds ago how are you going to offer good completions or know that there's an error right you don't want to wait until save so the very first thing that we need to do is we need to figure out how do we do text document synchronization and that first part is built off of right here what we need to do is in our server capabilities we need to say hey there's a text document sync kind okay and what we want to do here is we're going to use the simple one which is just a number here we're going to send a number okay and this is we can just say that it's going to be an INT and we can serialize this with text document sync like this now if we were doing this in a more serious one we wouldn't just send sort of full here we would handle incremental updates and do all of the fancy diffs and we don't need that okay we're just going to say hey send me the entire contents every time you update the file and every time you do that I'll make sure that my internal state is fixed okay uh if that doesn't make sense yet that's okay that's okay we're going to get there we're going to get there because what we're going to do here is we're going to say okay what's my text document sync I'm going to say one right because when we look at this we say full documents are synced by always sending the full content of the document this is where things are about to get exciting when we rebuild right here we are going to start getting a lot of new messages when we connect with neovim again so notice here we now have our very first text document synchronization message we got our initialized message which is what we had last time but this time we've got a did open and here's what's did open hey funny you should ask we can just go ahead and check in the spec this says the document open notification is sent from the client to the server to Signal newly open text documents right so this is saying hey hey language server I'm neovim I opened this file you probably want to know you you told me that you want to know about these kind of files and now we can start doing something with those files let's try and unpack this from the notification that we're getting and so here's where we see that same pattern emerging again this time though instead of being a request and response we're just going to see a notification this is kind of like the client is just shooting this off into the void and expecting the language server to handle it and we're going to do that so we see our method is text document did open and we have this did open text document params which includes a text document item so we have a few new structs that we kind of need to handle so let's go back to our LSP package here and let's go ahead and make a text document file here to keep a few of these items around we need to create a new text docu doent item struct so we have type text document item that's a struct and looks like this right and we have these few things here and we'll just we'll format all these in a moment right and so this should be a document URI we can click hey what's a document you're right oh that's just a string okay well I'm just going to call that a string for now when we're here we're going to get a language ID and then we're going to get a version which is going to be an INT and then we're going to get some text which is the content of the file these are now the different items that we have here for a text document item now we have a little bit of a problem right because we actually need to change this to be something like URI and then we need to tell Json that hey uh you actually need to put this in as URI okay that's fine we need to say language ID because go needs these to be um visible like this and to be honest I kind of like doing language ID with both caps sorry if you're not a fan of that but too bad then we need to do version and I spelled I noticed that you thought I was going to miss it but I didn't miss it and then we need our last one here for text text Okay so we've got all of our text document items here this is great we go back inside here we're able to do this we go back to where we were before and we see okay we have our method and our prams let's go ahead and go to text document and let's do did open we're just going to going to put these in different files not because they need to be but just so that it's really clear what I'm doing for each sort of request separately you could easily put these all in one big file and go and that would be totally fine but I'm separating them out just so that it's really clear what's going on for the video so what we have here is we say okay I should be getting a did open text document notification so we do we're going to make a new did open text document notification and that's a struct and it should have a notification and then we have the pams here and that's going to look like this did open text document pams so we have pams did open text document pams thinking wow these are kind of getting long yes it's okay if you were doing this by yourself you might have a few different helpers for a few of these or something like this or maybe you're going to use some AI to autocomplete those that's really good and it makes it pretty easy but for right now this is great so we have a text document here and this is going to be a text document item and we're going to encode this like this so now we have our very first text document style message what we want to do with this is we go back to main and we said hey I see a case statement here I see a case statement let's handle this case so we have a new case and it's going to be text document slid open open which is exactly what we see here for this method okay we're going to use this and we're going to say huh it's almost like this exact same pattern where we're going to go here and we're going to do the same idea but instead of doing LSP initialize request we're going to do text document uh did open text document notification and now beautiful go tells us those aren't valid right and so we can instead say something like this we can say opened and we'll just delete one of these here and we'll say instead of client info it's going to be text document and URI and that's good enough for now shortly we'll be able to show you that we also know the content so to do print contents that's fine well why not we'll just print it right now we'll just print it right now request forams teent text great okay so now what we're hoping to see is we're hoping that each time we open a file we're going to get this message that tells us all of the text that's in the file we build this we go over to here we open it again oh it's just so beautiful isn't it it's just so fun what we see here is we have opened we have our file and then we have the contents of the file right so this is exactly what I'm seeing when I open my other neovim we sent this entire file over to the language server and now it's able to know things about this file now we're not really doing anything with this information yet but we probably should be so here's where we're going to do a little bit of magic hand waving I'm still going to show you all the code writing but we're just going to pretend that we have something like a compiler that knows things about the file right in this case we don't we're just going to make up sort of fun examples along the way of what we're doing but if if you were writing this for a real language you would use some compiler tools or some complicated parsers and static analysis and a whole bunch of other stuff to be able to say ah we've got a new file let's kick off a parse job or okay we're not going to do that because that gets away from the point of this video which is understanding how language servers work so let's start building out our compiler and we'll then be able to use that to provide intelligence or analysis throughout the rest of the project we're going to once again create a new folder and we'll just call it compiler we can just call it analysis maybe because that's a little bit more correct you know if your language doesn't have a compiler it's sort of all the same idea here and here we can just make a new one and we'll call it like state or something right so we have package analysis and we're going to use this right to then be able to sort of say what what's our current state of what's going on so we'll make a new thing here and we'll call it the state and we need to keep track of like our documents and let's make that a map of strings to Strings right and so this is sort of like whatever the current state of all of the open documents are we're going to save all of those here and so what we can do now is we can say all right let's make a new state and that's going to return a state here we're going to return state with uh with a documents set to be some new map right like this so we've initialized the state what we can do now is we can say all right let's take some State and let's open document okay and we're going to pass in a document and some text which are both strings and we're just going to mutate what we've got here right so we've got s documents uh and we're going to create document here and we're going to set this equal to the text right so this is just literally a map map of file names to contents okay so this is like really not complicated in this case you would obviously do way more complicated stuff I'll just say it one more time right you do way more complicated stuff if you were actually doing this for a real language but we're going to do it just so that we're able to manage these different ideas so if we go back to main here and we say hey when we start this when we start our whole Loop let's get a new state right and so that's going to be from analysis. New State here okay this is our new state and what we want to do with that is we can pass this to our handle message right so let's pass the state right here and then this needs to say all right you've got some State and that's going to be an analysis. state right so this is sort of each of the things we're doing so we're sort of threading this state through all the different places that we're going to handle messages and when we get a new document what we're going to do is we're just going to say all right well State open document and we're going to pass request pams uh URI right and request pams text document text so now we're basically saying all right I've got a document and we've opened it that's all we're doing and we're ENT essentially syncing the state of our analysis Engine with the state that's in the editor our problem is that right now all we do is sync this at the very beginning when we first open a file if only there was some way to find out what happened when a document changed if we go ahead and go back over here and we notice when we change something in here something we notice now aha we actually do have a way to track that that's text document did change so that's the next message type that we need to handle and it's actually not too complicated to start doing this we can just look for the did change notification here and we see we effectively get the same idea again right we notice that there's some notes about the different types of capabilities that server and a client might have and we can register to handle these we're not going to worry about any of the complicated versions of this for us right now we already told the server hey or we already told the client hey I'm a server I only know how to handle when you send me the whole document each time and so we get these did change did change text document prams we're going to do the same strategy again exactly what we did last time so we go back to here we go back to text document and now instead of opening a new open we're going to create a new file text document did change and we're going to go along the same ideas that we did last time we're going to create a new notification which is the did change notification so type did change notification we could probably say text document in front as well and then this is going to be a struct and what's it going to have it's going to be a notification and then its pams are going to be a did change text document pams so we've got pams again did change right this is we'll just copy exactly the same name that we've got here did change text documents RAM and we'll encode this the exact same way that we did last time with params like this we don't have this type yet so we can look at what it is it has a versioned text document identifier and we're going to handle this as well in a moment right so we have we should put this in our text document spot here so we've got a type versioned text document identifier struct we can go ahead and pop into this guy and see ah it's a text document identifier with a version right so notice that it's extending this so we go here and say ah we already have a text document identifi or we can we can make a text document identifier right so type text document identifier and we can make this we'll pop this up here just so it's a little easier to read struct and we say ah this just has a URI URI which is going to be a string and we're going to encode that is just URI like this now we're going to do the same thing here we're going to say ah this is a text document identifier like this with right with a version so now we have some version that's an integer and we're going to encode that as version so same sort of idea going here right as we're working through the spec we just keep saying okay sure you introduce us to a new type we just write that type out so we can handle it and then we can add that to the stru that we have right so when we go pop back one more time again to our did change okay we've got text did change text document forams the first one right we see is that we have a text document which is a version text document identifier and we need to make that public so that it can be Json encoded text document and then we also need to include these content changes so this is what's actually changing when we get an update right cuz you got to keep in mind as we're working through this the main thing that we're trying to do is synchronize the state of the editor with the server right and so we we need to send messages to make sure they're in the same place and that's what these content changes actually are so we need to make these text document change events so we can make a new type and we'll just copy the same thing that's going on here here and in this case ours is actually a bit simpler because we don't have to handle this sort of incremental style update we're not doing that we said all we know how to handle is getting new text and so this is actually quite a simple struct for us we can just say this is a struct like this and we have text and it's a string and it needs to be here and once again we're going to encode it like that as you're probably getting used to so that's that same sort of idea here right of just a new struct this event so we say we go back to this content changes content changes and that's going to be a text document change events and that's just encoded as content changes all right so those are the two new things that we have now as part of this text document did change notification since it's a notification we don't have a corresponding response we just have the notification to handle this notification we have to go back to our main file and we have our did open handling here and this did open we actually can pretty much just copy paste it because to me very similar this becomes did change and we can change this to did change uh text document or sorry text document did change notification right here now you see aha now that we've decoded this we have some errors here we no longer have text anymore right we have to handle the changes and so we can change this to be something like changed right and let's actually just update our error messages here as well just so that we can get a little bit more sort of uh information in case we we have an error trying to decode these we can get a little bit more info of what is going on inside of the our logs but with that pass we can say okay we have this changed thing and instead of saying the text we're going to put content changes and now we need to handle each of these and so a very easy way for us to do this would we can just do is we can just do something like this we're going to do four change in the range of request params content changes and now instead of Open document right we need to do something like update document so open is going to say uh update document right and these should probably both say URI cuz that's really what it is right and in our case we are just getting text so there's nothing special to do here we can just change this to be change. text all right so instead of update we're going to do or instead of open we're going to say update here we're going to update this document right so we're going to take all the changes there should basically always only be one in our case we're going to update the state of our analysis and so now what's going to happen is let's let's not actually print all of this every time CU it's really going to muddy up our um logs and we'll delete this too because now you've seen that the text is printed there when we build this again and we reconnect with neovim to our LSP we're going to be able to see here that we've got our opened right and now when I'm changing this we see that we get our open and received and changed and received and changed right so we're getting our changes as we're editing in the file right and if I say here we're going to have a bunch more ch changed file notifications okay so now is where if we had a real sort of analysis engine going on it would be updating typechecking linting all of that stuff we're going to add a few examples of what that would look like without actually doing any language analysis and the first way that we're going to do that is we're going to start responding to hover requests the reason that I want to tackle the hover request first is because I think it's kind of the most obvious one that can show that we're both updating the internal State as well as providing a really obvious action in the editor and so the hover request is sort of like the documentation thing sometimes it happens if you leave your cursor for a while on an object or Neo can happen generally when you press something like shift K and so what we want to do with this is we first need to tell neovim hey or whatever our client is hey uh I can do hover I'm a hover provider and to do that we can look for where do we see something like hover provider it says Hey the server capability right here is property name and we can just say true as in yeah we do sort of like the basics right so we go back to our initialize and where we have this text document sync we also need to say hey I'm a hover provider and we can just make that a Bo because like I said we're going to sort of provide the basic options here we don't need to go into all the other possible hover options that exist we're going to do the simple ones so we say I'm a hover provider and then now we need to say we actually need to store the value as true so we put hover proprieter and we put true here so this lets us really really easily Now notify neovim when we start up hey I can provide a hover to you and that will make it so that when we request that we're going to send that request so let's build a new version we close neovim and we're going to open it again right we get our messages that we're expecting our text document did open and everything and now if I press shift K I'm going to send hey look at this a text document hover oh there you go now you can see it right we're going to send a text document hover request now we're not currently doing anything to respond to that that's what we need to do next so we go back to our main function and we need to add basically a new case statement here but before we do that we say all right of course I need to figure out how to unpack this request so to do that we look here and we say all right yes there's our registration options and here's our request so we're going to make a new file again once again not required to make a new file for each of these I'm just doing this to separate out what's going on we have text document hover.to and we've got it in our package LSP we're going to say okay our request looks like a hover pram and text document position prams okay great great so we're going to do something like this type um hover request okay and that's going to look like a request and our pams are going to be hover pams and that's going to be encoded like this again and our type hover prams struct is going to look like this we need to figure out what a text document position pram is and hey we've actually already implemented this first type right so we can go back to our text document file we can go to type and let's go ahead and we'll scroll up a little bit again so you can see better text document position params struct and we have a text document text document which is nice because we already have this text document identifier uh text document like this and then we also need to have a position okay so we say all right what's a position position which will be a position and we'll encode that as position you really love repeating yourself sometimes and go don't you but that's okay type position is some struct that looks like this and we've got a line and a character okay that's easy so we have line which is some integer and we can say line like this and character int just character like this all right so now we've got our line and character so now we have all of the types that we need to go back to our our hover request our text document we first were adding our text document new types we go back to our hover right and we can now say that our prams are actually just go back here hover prams just is a text document position pram so we can just put text document position pams like this okay so now we have our request and the params that we need to pass so we can add our response our response is either a hover or nothing right if you couldn't find anything then you won't do anything but we can actually do this very easily we'll just say that a type uh type hover response struct is a response and also it's going to have a result of hover result is hover result of hover or of result sorry like this and a type hover result is going to look like a marked string or a list of Mark strings or markup content we'll just see whichever one looks easiest um I'm not sure Mark markup content versus marked string a mark string can just be a string oh well that's quite easy we'll just say that we have a string and that's all we need to do so we can just say we go back we go back to here we have a hover we just need to say contents is a string contents is a string that's the easiest version and like like I said each of these has somewhat more complicated versions of what's available but we don't need to use those for the educational version of what we're trying so we can just return a string directly and you'll see what the result looks like in just a moment now that we have the structures in place we need to actually do something with those structures which would be we're going to actually reply in the same main Loop in the same way that we did for our other notifications and responses this is going to be much more similar to the first time that we did initialize where we created some type we try and unmarshal it to that and then we reply so we write this back out standard out so a bunch of this is going to look very similar and so in a moment what we'll do is we'll go ahead and refactor just a touch to make sure we're on the same page instead of text document did change though now we need to respond to the hover method right so that's the method that we're going to get so we see hover here and this is going to be a hover request we're going to unmar this into a hover and we can delete this part here so now we need to create a response and write it back okay but this writing it back is very similar to what we were doing before so let's go ahead and pull this out into uh a separate kind of function here we'll notice that this part is going to be the same basically for every single function so we can go ahead and delete this and we'll go down here and write Funk uh let's say write response message any and that's not going to return anything uh we probably also don't want to sort of force ourselves to always be passing to standard out in this so we could just pass a writer like this and say that it has to be something that can write and we can import that and so that's basically all we need to do here so instead of using the writer that we had before we can go up to where we have encode message or where we had the initialize here and now we can say hey I just want to write response to the message and we need to pass a writer as well and we should thread this into our handle message here because like I had said before one of the things you want to remember is LSP is about sort of the set of methods and how we're going to respond it's not about how we do sort of this transport right so if we later wanted to allow our LSP to communicate over TCP or something then we would just pass a different kind of writer that still fulfills this interface look like this and we go back to here and we need to create a writer and in this case that's pretty easy because we can just do the same thing that we had before which was um we wanted to do os. standardout so now we have all of our things handled here where we can just write our response like this okay well that's pretty easy we can do the same idea down here for hover we're going to create a response so we have response and that's going to be an LSP hover response and for now we can sort of do the very simple things of we'll do an ID which is going to be request. ID right and we can have RPC is 2.0 just like we keep doing every time and then we want to say a Content so we can say hello from LSP so what we're sort of Desperately hoping for right at this point is that when we send this text document hover request here what we want to have happen is we're going to UNM Marshal we're going to create this and then we just want to say say hey let's write the response back uh of our writer and our response here and if this works what we should hopefully see is after we build and we restart over here if I press shift K which is our request for hover I should see this message that we said hello from LSP so Our Moment of Truth and there we go that right there that right right there our hello from LSP is sort of our first indication yes we can really do this whole life cycle we can connect we can handshake we can say I have a provider and I know what to do with hover requests and then neom says ah I see that you've replied with a hover I can display that now right now it's not very useful right it does the same thing no matter what it there's no updates let's make it just slightly more interactive to sort of prove that we're really sinking the states between the editor and the server so to make this more interactive what we're going to do is we're going to pretend that we have in our analysis package some idea of something like s State and we're going to say hover and that's going to take something like a URI a loc and probably which will be a string and then some location which or we'll just call it position because it is a position and we're going to not we don't need to return anything if we pass a writer but it's probably easier if we do something where this just returns a hover response oops hover response so what we want to do here is we can take sort of this idea here put this here and we could just return this this is not super exciting right oh and we probably need to also pass an ID here as well um just so that we can make sure that we're putting the correct response in so what we can now is we can say response actually equals and instead of doing this we can say analysis dot uh oh sorry state. uh hover we're going to pass request. ID we're going toass request. URI and we're going toass the position right so we're sort of unpacking this request into something that our state can handle now in real life this would look up the type in our type analysis code uh but we're not going to do that that's we're not going to do that right instead what we can do is we can just do something to show hey I I'm actually syncing the state here and so a really easy thing to do here would be we could say something like let's get our document and that's going to be S documents of the URI that we pass and then let's do something like this we can actually say okay let's just count the characters in here to show that we're able to see how much text is inside so we can say the uh let's let's format Sprint after this right we're going to do the same idea that we've done before where we're going to say something like file and then percent s and characters percent D and we're going to go URI and the length of the document like this and if this works what we should see is that after we build this we're going to go back here we'll restart again and when we press shift K we get our file right here which is great and it's says 251 characters okay but now if we type again we should see that that character number changes right the reason I want to show you this is I think it's pretty cool that we're able to see here that the state is getting changed we haven't saved the file yet it's it's not as if the file saved in fact we can even do something like pretend. MD right so kind of like how mathematicians like to call themselves doctors right like a pretend. MD and we can say hello and then we can say here and say oh look this this file here this isn't written to disk this only exists in the editor in memory in the editor there is no sort of file system representation we only know about this file and how many characters exist in it because we are syncing back and forth the state between the server and the editor and the next request that I want to cover is go to definition this is the one that maybe you control click cck or rightclick definition or press maybe GD inside neovim to go to the definition of whatever is under your cursor and what you're going to see here is this part is going to look awfully similar to hover right and so let's just walk through it and we'll see sort of where we come out the first thing is we need to say hey my server needs to be able to say I support going to the definition and so we need to basically say I'm a definition provider right and so if we go back to our initialize and we have our hover provider we need to add a definition provider which we can also do as bu and do definition provider and then once we have this we can go to hover provider and we can say definition provider we'll set that to true now we're going to be able to respond to text document SL def uh definition right which is what it says our method is going to be right here before we can do that we need to implement the params now we're in a nice spot here because it turns out definition pams is just text document position params so we can go back to our hover pretty much copy and paste this whole thing make a new file again for definition we'll just go here and we'll do percent s hover and we'll just go definition and we'll substitute all those and now we're done because that's basically it except we have a different type of result right so we need to say our response is actually just going to look like either a location a list of locations or a location link in our case we're just going to do the simple one again and we're just going to return a location right so the location just looks like some document again we've sort of handled this part before and a range so we've got to make a new thing in our text document here let's go ahead and make type um and we'll go ahead and make this location and this is going to have a URI which is just a string URI and we're going to have a range string range like this H sorry a range is actually going to be a range which we haven't implemented yet so we go ahead and look at a range and that's a start to end position so type range is going to be uh start and a position start like this and it's end right end position end all right so this right here here is now what we can send back as a response to the request so this is literally just a result here is actually just going to be a location we don't need to have a particular extra type because it says right here the response can just be a location there's no special type for definitions and what did we do last time we said okay I got to go and I got to handle this in our main function here and so we can just go ahead and yo all of this code here and we'll do do the same thing here now instead of hover we have definition and then instead of here we have definition request and we should just make a note of that being definition now if we were sort of making a production grade one we may sort of automate a few of these or maybe make some structs with interfaces that say what the me I just want the code to be clear exactly what's happening in this case right this is really easy for us to read and easy for us to think about we're going to switch on the method we're going to do something different on each method and then we're going to respond right and in this case instead of having a hover now let's just do definition right and it says I don't know how to do that that's fine that's fine we can basically copy this here we'll take a definition and once again it's actually just going to be some ID some location and some position here right and we're going instead of doing this we're going to do a definition response which is going to look a little bit different than the last one but that's okay so we say definition response here and this instead of being here is just a location LSP location and in our case right we actually we we don't have any real way to look up the you know definition you'd do something like making sure that okay this is a completely formed word under the cursor and then I'm going to jump to that spot right in this case it's starting it to look up position that's what happens when I use the go LSP for us just since this is sort of of for educational purposes we can just do something really simple like saying that our line here this can just be position doline minus one and we can have character of zero and we'll just do the same thing for both of these this is basically just going to say all right you asked for the definition we're going to just say that the definition is always one line above where you're currently ask at the beginning of the line the only reason I do this is that we can show that the cursor is like doing something right I I I don't really care how you look up what the definition is that's just how you would get something like this where you would say oh here's our definition location we would look that up right so in reality when we go back to our main function here and we have our definition that's already going to create that response and then write it as well and so if we go and build this again and we run over to neovim and restart here if we ask for the definition up right here we're going to pop back up one line and when I ask for the definition again I'm going to go up one line again so it's not the most exciting definition lookup right but what it does is it just sort of demonstrates once you sort of get this flow you sort of understand the checkbox nature of okay I I see the spec I see a new thing I want to support I tell them that my server supports it then I look at what the request is and then I look at the response we just write those and then we respond there's no extra magic going on there's nothing extra special happening right it is just literally we're mapping this spec into the code that we have sort of in our main function right all right let's do the next one which I think we'll do now code actions and that one will provide a little bit more sort of interesting interactivity so I've written code actions off screen because I want to first show you what it does and then have you sort of guess okay I can see what it does how can we accomplish this task so a code actions does is allows you to sort of say as the client hey uh server is there anything you can do to fix this file for me and in this case if we're on a word like here vs code we want to be able to change that to something that's better right and so if we ask for code actions we're going to get something that looks like this and you'll notice that neovim is telling me these come from our educational SP and I have two different options I can either censor this to vs code right and it will replace the text there or if I replace I can replace it with the superior editor obviously which would be neovim and now you're probably wondering though how did we do that right because what we just did was we it's a really complicated series of steps if you're thinking about about it right is first I made a request and I said I I want to be able to do a variety of different things and the server said yeah here's your options and then I selected two different options and two different results were sort of like apply to my file or my project let's get into how code actions does that and how it works so I'm really hoping the first thing that you guess is that we needed to say hey we are a code action provider if we go back back to our initialize you'll see that I added code action provider and now I'm responding in my initialize to say yes I can provide code actions right this same exact thing as we've done before and once again there's a ton of different options you can look at all the different ways that clients can apply them that's great right but the point is we're going to handshake on what we're allowed to do and then I say aha I have a request and it's got a code action here and this sends something like the text document some range and possi additional context in our case we sort of don't care about some of these additional context that can help you know whether you're supposed to provide like help for imports or just fixing small things or refactoring there's a bunch of different options that you can do but once again we're sort of going to gloss over these finer details and just say okay I have a code action request it's going to have some prams about the text document the range and maybe some context we're sort of going to ignore that right now and then we have a result which is a list of code actions right and if we scroll down in here to our actual response here in our result it says you can have a command or code actions and in this case we just did code actions which can have a title right and they can have some edit that they're going to apply and optionally some command now in our case we're using the edits because edits allow us to do something where we say hey here's a bunch of files right I have a bunch of files and I want you to apply these text edits to range where text edit is basically just arrange in some new text right so you could do something like taking a range and if the new text is empty it'll be like a delete if you take the range and it's just like one position it's kind of like inserting or if you take a range and you have some new text that's like replacing right so effectively what we can do is when we respond we're going to be responding oh let's check main first we're going to be responding right in this function here with our code action we're going to be saying all right I have some code actions that I want to reply we're going to go and check all of the lines in our file and we're going to look for this string if this line has this string we're going to suggest okay the way that we can do this is we can get a range right on this Row from basically the start of this to the end and we're going to replace it with neoven that's our first action right which is this joke about replacing VSS code with a superior editor right and then the other option that we have right we're sort of positing two separate changes that could be applied the second option is the censorship one which replaces this o here with a star right you can sort of see how we're going to take this range and we're going to apply this new text and then we add this to our action list and now once we're done what you'll see is we take these actions and we respond to them in this text document action response so it if you're not getting that that's okay just slow back down and think about this step by step here we're going to respond with a result with a list of code actions these code actions for us are effectively just text edits and neovim knows how to apply right these two separate text edits to the current file so that's up to the client to perform the edits right so the server only has to worry about saying okay you asked if I have any code actions they're like yes yes yes I asked if you had any code action says here you go here's your options you do with them what you want right and so the user can select one of these two things press enter in our case and then apply that text at it right so that's sort of what's going on here there's not that actually constantly going back and forth at least not for every case between the server and the client for each of these it's actually that the server sends all of the information that you need to the client to be able to execute those edits and there's just two more things I'd like to cover in this video the first one is completions and after this we'll be doing some information about Diagnostics which is how you get errors warnings those kinds of things showing up in your editor completions are what you get sometimes called intellisense or autocomplete these sort of ideas they show you possible things that could show up after your current cursor and in our case we're going to implement something very Sim simple but you can imagine that the compiler could give you something more powerful or your static analysis tools could give you something more powerful we're going to follow exactly the same method that we've been doing before which is we see that first there's a description of client capabilities if we were writing a more complicated server we'd think oh these only support these kind of kinds it's all good we don't need to do that we're going to just do the base support and so once again we see that there's a server capability for completion provider so we're going to head over to initialize you guessed it completion provider and we're going to mark that as true it's going to say I don't know how to do that you're totally right we say completion provider and this should be completion provider but it turns out actually this time it's slightly different so we are almost tricked but what we see here is actually property type here does not accept Boolean as one of the options instead it's sort of this map of okay we've got Trigger characters or commit characters or result but all of these are optional so for us we basically need to instead of passing a map or I mean a Bo directly we can pass a map and we can just say that it's like a string to anything we want and we can change this down here to something like string any and instantiate this this will make sure that instead of passing a Bool which is not a supported type in the protocol we actually pass a map and make sure that that is specification compliant now we're telling our client we do completions and so now that we've said that we do them we need to handle the requests and so the request looks like a completion params and once again it's great because we've already started doing some of these other building blocks and we can compose them very easily again so we go ahead and go over and we'll just grab our hover one again we'll open this and do text Doc doent completion. go and we'll change all of the hovers to completion requests and the only thing that we'll need to change here is there's also this optional context we actually don't care about this so we'll just skip it so now we have everything for our request now we need our response a response can contain a list of completion items or a completion list um for us right now really the only thing we care about doing is returning some some of these items and so that's all we're going to return so we're going to say here our completion result right here right here is just going to be a list of completion items right because if we're right here we're looking our result is going to be this and so we can just say uh completion item and we'll go ahead and go completion item right here and now we need to fill in what the possible values are for a completion item so the first thing that we see is that it has a label uh label is sort of the default text that gets inserted so we're definitely going to need that label string label we don't care about any details uh the kind is also sort of irrelevant for what we're doing here we're just providing a text completion but you can see that completion kinds are for saying oh this thing's a function or this is a class that's how you get nice little icons right when you're getting completions you can get tags which we don't need right now d detail is nice because this tells us some additional human readable information so we can do detail detail and that will help us sort of provide a little bit of nice popup on the side and a human readable string for a document comment as well so we can add a documentation and string documentation we won't handle the more complicated cases for a markup content right now just a string is fine and then the rest of these we really don't need these are for more advanced features these are things like if when you accept this completion you'd like to insert a import at the top of the file that's how that import can get automatically added the server is going to say oh you want to complete this package name okay well I'll complete the package and I'll add the import you may have noticed that this has happened quite a few times inside of our go files if we have example here and I write something like f s print F when I accept this completion it adds that import up here that's done by doing these sort of additional insert text information here occasionally those may also be done through text edits or replace edit but that's sort of the same idea right so when you want to do these more advanced features we're just sort of poking around in these areas right additional text edits so you can sort of see how the text edit support that we had just talked about for code actions we're reusing that idea inside of completions as well and so that when a client can support inserting text edits or applying text edits I guess would be a better way to say that then we can also apply text edits when accepting completions or when doing others so it sort of all Builds on top of each other in that way but for our case we're just going to stick with the very simple example of putting some detail and documentation so we can see that those are working and now that we have that we need to go back to our main file and we're going to sort of take the same idea that we had here for code AC and we'll go ahead and do another one but this will be completion now and this will be completion request and now we can do text document completion which it won't know how to do yet and that's totally fine normally you would also pass in the position here and you would be calculating this but we're not going to worry about that right now when we go to this state analysis here we're going to make another new function here and we can probably just basically copy the same idea that we had for code action and do text document completion and this is going to return a completion response now right now it says you are definitely not doing that that's true um completion response we'll update some of our types here the result is no longer actions but instead items we haven't defined what our items actually look like yet so we'll make items and we'll do completion item and here you would normally do something like ask your static you know ask your static analysis tools to figure out good completions right so you'll notice right when I do like a completion in goang and I write f s printf like this it still can even guess that I might be talking about some of these even though I wrote sprt which isn't any right so you're seeing like the LSP can do a whole bunch of really complicated analysis in the back end to sort of guess what you might want to be typing uh but the idea here is we don't really need to do that we're going to just put in okay so we did that analysis and now we're going to make some items and the easiest thing that we can do with this is we'll just create a new item and we'll put inside of here just something simple like neovim by the way and the detail would be very cool editor and I would say um fun to watch in videos don't forget to like And subscribe to streamers using it right I think it's a nice little tip perhaps and so when we have this completion set up we go back to our main we just double check that when we do completion we're getting this here right we're unpacking completion request yep we're getting a completion option and then we write the response back and so now after we go and build this item here we're going to be able to go inside of our neovim we'll restart it so we get the fresh version of that LSP when I type neovim you'll see that neovim by the way is a suggestion and we get our text and our description that we added so all that's happening is we're sending that request from neovim to autocomplete basically as I'm typing and when it sees that I have something that matches right it if I start typing hello world it's it's not going to suggest anything with neovim but if I start typing neovim like this it will accept that and I can accept the completion and now we're on to our last topic which I wanted to show something a little bit different because notifications and Publishing notifications is instead of being sort of a pull method from the client right I'm pulling information from the server publishing Diagnostics is generally a push notification from the server to the client we're basically saying here hey uh we did some analysis sometime after you know you told me about an update to a file I was either able to like run the compiler and get results out or something like that or maybe a linter or whatever it's going to be and I have all of these Diagnostics you should know about them you should see them in your editor and so what we can do here is we're going to implement the same thing and we're going to sort of have another little uh play with the uh VSS situation that we were joking about in code actions we'll use the same idea here to check if we see that anywhere in the file right and so we do we're going to follow along with the same rules that we've been doing up until now right and the the slight difference is that there's not going to be a request from the client to say I want Diagnostics there was recently some things added to pull the Diagnostics manually but in this case what we're going to do is we're going to do that push style basically notification and so what we need to do is is we need to go back to our text documents here and we'll go ahead and open up a new text document so we have something like this uh pams publish diagnostic let's do this and say pams prams and let's make this type here publish diagnostic prams so what we're going to put inside here is a URI which is just a string again so we've seen this before and we're going to do we don't need the version CU we're not doing version documents right now we're just doing full sync State and now we need to have some Diagnostics so we have Diagnostics like this this is going to be a list of Diagnostics and we'll publish that as Diagnostics so what does a diagnostic look like well it's going to have a pretty big sort of range again of the kinds of things that we can do but we'll do the simple version as we've been doing for the educational purpose right so the first thing we see is we need a range so we have range range which should look like this oh and this should just be a diagnostic it's a list of Diagnostics there's a severity severities are nice because we can sort of say different levels of Errors right and we have sort of one two three four you could use uh goes enums and things like that but for now we'll just do an INT for this and say severity we once we have the severity we don't need the code right now a code description we're not going to sort of deal with that a source uh we can put a source in which can just say that that's from our language server or something like that we can have a message which is very important this is what gets displayed to the user and then we don't need any tags related information or data this can be used sort of to do a few round trip Style Communications with the editor so now that that's all sort of the structure that we need to be able to start publishing these Diagnostics now like I said normally you would do this on some sort of maybe you've got like a multi-threaded server and it's doing some background job of analysis as it's picking up updates and when those things complete it's going to send something new that's a very good strategy for us we're actually just going to go to where we have this sort of did change status right here and we're going to publish new Diagnostics on updates and writing so you go back to our state and what we can do here is we can say okay when we update the document what I really want to do is I really want to return a list of LSP Diagnostics because I think that this is probably the easiest sort of way to then push this and for right now we can just return sort of nothing right we can say okay we're going to make a diagnostic list with nothing inside and we'll do the same thing when we open a document because we want to get these Diagnostics right when we open the file if we go back into here we'll sort of say Diagnostics is equal to this and then we can just write our message uh we can do write a response and we're going to do LSP dot oh sorry we're already in LSP here so this would be publish diagnostic notification and we're going to say method uh method right is uh the notification here is going to be a notification which has a method yep so we'll say notification like this and we'll add what we need here we don't actually need the original notification so we can Say LSP notification like this we'll fill the fields and we'll fill these fields to say RPC 2.0 method is text document SLP publish Diagnostics I think it has an S this is a classic one where you can uh you can pretty easily sort of mess yourself up yes it has Diagnostics with an S and the for the pams we're just going to put in the Diagnostics that we have here as well as the current URI okay and there's not a whole lot going on here to be able to do this um but we can oh we can pass writer like this so right now we're not really sending any Diagnostics there's nothing really interesting with what we're doing here but Diagnostics we're going to add those shortly okay so the first thing here is when we create these Diagnostics let's go ahead and check the text that we have to see hey are there any are there anything inside of this document that could be fixed okay right and in our case we'll just actually do the same thing where we looped through the lines here like this and we'll actually let's write this as a so let's do Funk get diagnostics for file and there's just some text string like this and it's going to return a list of Diagnostics ICS and we can go Diagnostics lp. diagnostic right so we're going to get our list and we're going to append to it so we can go Diagnostics equals append Diagnostics we're going to do that and then at the end we'll return the Diagnostics right so for each of these what we're really going to do is we're going to do get diagnostics for file text and we're going to do the same thing here how are we going to get those well we are just going to say if strings contains right uh line and we'll say vs code again right then what are we going to do we're going to say we want to add a new Diagnostic and the range for that is going to be can we find does contains just return a Bo I think we can do strings index yeah of line and here so this should be our idx right and this should be some number and we already were using before our uh range for line range for row idx idx plus length of es there the severity for this if we look back at our severities is going to be that's definitely an error that's not a warning or information level that's definitely an error level and source is of course common sense and the message is please make sure we use good language in this video right so once we build this again and we reopen the read me what you'll see is Boom we've got the error right here that tells us please make sure we use good language and of course if we wanted to add more things inside of here we could do something like this where if the string instead contains something like neovim then we can go and do this this and that could be probably a hint and we can say great choice smiley face right and you'll notice that once we build this again and we load this back up we're in the same situation that we had before if I typed this right oh it just says great choice right here right I don't have that I don't have that underlined in my editor because I don't like when hints tell me everything we could change this to something like two rebuild and you'll see that each of these cases we sort of have a different experience inside the editor right so that's how we're getting these different choices here so it says great choice we could change this to here or we could do this with a superior editor right and then now we get our great choice here that says okay you've got this Diagnostic and it's inside of our editor so that's actually like not getting requested by neovim right instead that's actually getting just submitted by the server telling neovim or whatever editor it is hey I've got some I've got something for you to display and you should display it last time I'll I'll say it but obviously if you were doing a real one you would do something a bit more complicated than uh than this string checking but that at least gives you a picture of how Diagnostics works and that's everything I really wanted to show hopefully even though maybe the video was a bit technical at times where maybe we were moving a little quickly through the go you can go back I guess and double check on everything to make sure you're understanding but the goal of this was to show really there's not any magic going on with what's happening with a language server and an editor they just pick a protocol they send it back and forth and if you send the right messages the right things are going to happen in the editor and we did all of this with no dependencies no additional dependencies just in go in only just over 500 lines of code and a lot of that is just really saying what the types of the different methods are right a lot of that is not really any of the difficult things going on if we look at something like how much how much how much code did it take to actually do the RPC itself only took us like 80 lines of code to really start communicating with an editor and if we're thinking about how much did it take for us to start actually like running the code and hooking that up to the editor that's only another 125 lines of code so I really hope that this demystifies what's going on with LSP if you like this video please subscribe please leave a like leave a comment if there was anything you'd like me to go over in more detail another time um I'm hoping to maybe do a video in the same style for tree sitter coming up soon and I really hope you liked it it's a new style for me a little rough around the edges maybe but I I hope you like it thanks everybody um and yeah I'm TJ and hopefully you like these videos cuz I'm having fun making them bye

Channel: TJ DeVries

Views: 66,188

Keywords:

Id: YsdlcQoHqPY

Length: 119min 54sec (7194 seconds)

Published: Tue Mar 19 2024