Understanding Language Server Protocol - autocomplete, formatting - Adrian Hesketh

hey everyone thanks for coming um so yeah I'm I'm here to talk to you today about language server protocol the thing that gives us all that stuff in the development environment so we're going to cover what language server protocol is the message formats the things the messages that go back and forth in that in that system um how you initialize a language server and we're going to build a language server show you how to build your own language server implementation from scratch um and go through things like request response handling notifications and then how you actually test and use your new language server inside different editors because different editors work in different ways how you can log and then I'm going to introduce you to the temple language which uses all of the features above and you can rip it's a code from and ideas from to make your own implementations and then at the end we'll cover passing which is probably one of the kind of challenging aspects of building a language server is passing the thing that you're trying to sort of produce autocomplete and so on about um so what is language server well it defines that protocol between your text editor and your kind of programming language so it provides things like autocomplete so when you type in your editor HTTP dot uh the language server sends a message so the editor sends a message over to the language server and provides a set of results back you can jump to definitions so when you go to go to definition you're sending a message to the language server which responds back with a series of potential locations that you're editing and then jump you to and then Diagnostics which tell you when you've written your code badly or produced warnings and so on and you can also do formatting as well um the language server protocol gives you one protocol for all the different languages so what that means is like prior to language server protocols introduction if you wanted to add say go support to a particular programming language you'd have to know about all the intricacies of the go specific tooling which at the time was go Guru and used byte offsets and all sorts of things and then if you wanted to produce the list that there was no sort of necessarily any shared code so language designers if they wanted to get the like auto complete another features into a text editor it's quite an arduous process to do that and for people who are making text editors it was also every different language was was sort of had a different interface and different way of working so this language server protocol was produced by um Microsoft and they open sourced it as a specification and it was introduced in vs code and go introduce go please a little bit later and that that's what you see that's what you're using when you're doing your auto completion you probably don't see it like directly as Bingo please because you in Visual Studio code you tend to use a vs code extension which provides that feature for you um language server operates over standard in and standard out rather than perhaps a HTTP stream like you might be used to if you're building rest apis so what is standard in it's when you do LS you're typing into standard in and when you get the results back that's coming out of the program from standard out and you can send data to a program over standard in in Unix type systems by using this pipe operator so here you've got this vertical bar and if you Echo your sentence in and do a word count on it you get the number of words out but you can actually send any data so you can send binary data through that and you can even send structured data like Json and pipe it for a program like JQ to produce like formatted Json in your terminal um in the go programming language the you can get access to the standard in through OS dot standard in uh and it implements the i o reader interface implicitly so it means you can you can uh use that in any function so you can take in a reader and read from send it in and stand it out is available through os.standard out so if you write to stand it out uh you can you can that's just an i o writer so you that that allows you to use anything any function in the go programming language that takes in a reader you can pass in uh like the OS standard in and standard out so here we can decode Json that's coming in um so put that into the map and then write it back out to standard out through this interface and that basically provides the same features that we saw in in JQ but in writing ourselves in go um so what are we sending over over the wire when like over the standard and the standard out well editors send can send requests that require a response so some of those examples of those would be things like the auto completion results when you send in a request to the server you you need a response back which contains the the pick list if you like of items that you can select from um the messages themselves are split into two parts they have a series of headers um these are mime headers much like you'd see in a HTTP request so they have a key value pairing here um the language spur the specification states that only one header is always required and that's the content length header so you'll always see that in language server protocol messages and then you have an empty line so if you imagine piping this this value into your program that's what the language server is doing the content part then contains a Json RPC formatted message so the the as the name suggests Json RPC uses the Json format so uh that's the that's the structure of it um you can see that it's Json RPC because every Json RPC message contains the this key called Json RPC and the value of 2.0 which is the the value of that spec and every Json RPC method message request message includes a method as well which says you know what is the what is the thing that I'm trying to do inside the language server so in this case it's uh you know get to the Declaration when the LSP when the editor is sending a request where it expects a response back it always includes an ID field and the ID field is then included in the response message and that's how the text editor knows that this request this response matches up to the request that I made and we'll see why that's important later um and the request messages can contain arbitrary parameters so in this case the specification details um exactly what's going to be in each request that's going out to the the language server but in you know it's pretty straightforward there's a there's a header containing some you know HTTP headers and then there's a Content part containing some Json the response messages we're going to send back are actually in the same format they all start with this content with this header section which contains the always contains the content length and then it says Json RPC it Returns the ID back that we got in the request and then it always returns a result property containing uh you know the result of that RPC call so in this case we asked for like where is this uh where is this defined and we'll receive back a range to say like that's that's its location in the in the in the source code so your editor will then bounce you to that location editors can also send notifications so there's notifications don't require a response so examples of notifications are things like hey the user has opened a file here is the file that they've opened here's all of the contents of the file that they've opened so if you're imagining your editor you might not press save yet it might be a new file so the language server won't know about that file it can't just read it off the disk uh the text editor has to notify the language server hey this is what the this is what the user is seeing on the screen um and service can send notifications back the other way so you might send back to the um to the to the editor um diagnostic information so warnings uh and those can come at any time and you can also send messages back to the user so we'll see an example of some of these notifications in a minute uh but you can you know the the they can be sent at any point through the through the cycle um notifications are basically the same format as requests but without an ID so they have the content length um and then they have the Json RPC field and the method field and then there's no ID field just parameters so that that tells you that there's no response required to that to that message so we've got the Json RPC field the method and the parameters um unlike uh kind of typical HTTP request flow messages can be interleaved so if I send uh if the editor sends in a request the next thing that comes back on the wire might not be the response it might be a notif like a notification back from the server it might be some other elements so you can send a few requests in Rapid succession to the language server and then you'll get them back and it's up to the editor to sort of like consolidate that um the specification says that they should be roughly in the same order when you get them back but it doesn't clarify on exactly what that means but in practice you you know the messages themselves include version identifiers which identify the version of the of the document that the user is seeing so generally the editor can use that to work out what you know whether they should discard the message or not when they receive it back finds the requests as typescript interfaces which is slightly awkward for anyone who doesn't know typescript um it makes also extensive use of typescripts uh extensive type system including a lot of inheritance so for instance these things here these vertical bars are basically so an ID could be an integer but it could be a string it could be either one of those types and the params could be an array but it could also just be an object and the question mark says yeah well it could also be null as well it gets worse so uh but for us in go we can say actually well for the ID we don't really care about what the ID is we we all we need to know is that when we return a response back the idea is the same as the request as the ID in the request and so we can use Json raw message to say like look don't bother just whatever value it is we'll just Chuck it out the other side and we can do the same with the parameters to defer the loading of the parameters so we only know what what type or shape the parameters are going to be once we've read the method on the incoming request so if we use a raw message we can defer the processing them to a little bit later so that's my go definition of what a request message looks like um so to read a message we can we can read from standard in which is as I said earlier in IO reader and there's a couple of things we need to do first we need to read those sections of headers those the content length and then the empty line and go standard Library includes a tool for doing it in text Proto so it's actually just you know one line of code to to do that and that's so that's relatively straightforward it takes in a buffered reader so I've updated the functions taking a buffered reader instead but it's relatively easy to do um then I need to pass the content length header so we know that the content length header should always be there and in the specification it says like if the content header isn't there it gives you a specific error response to to return and it's just not a valid response um and then we can make use of a limit reader in go which says which you can pass it a reader and say read this many bytes from from this incoming uh incoming stream of data and we can pass that into the Json decoder and decode the value and away we get our our message finally we can check for the presence of the Json RPC field to make sure that its value is 2.0 and if it isn't we can start complaining that you know I don't know what you sent me but it's definitely not this uh it's definitely not an LSP formatted message so yeah um the response messages are also defined as typescript interfaces and I did say it will get worse um and what this means is the result is quite flexible uh it could it could it could be null but it could be string number Boolean array or an object um so what that really means uh in go terms is any right so if you look at the Json uh into like the Json Marshall interface uh it returns like in any result so um so what we've got there is just in any we don't really mind we're just serializing the results back so we can um we can be quite abstract in what we're doing here um as I said earlier we don't need to process the incoming ID we just need to make sure we always use the same ID so it doesn't really matter that it could be a string or an integer or null you know as long as we put the same result back um and writing a message is even easier than reading one we Marshal our message into memory first so that we can work out the length of the message then we can write the content length header ourselves and then write the body out flush the buffer and we're good so at that point we've got everything we need to read and write language server protocol messages but we haven't got any custom deal with those messages so let's Implement a multiplexer just like the HTTP serve multiplexer and then that will allow us to wire up individual methods to functions that have an implementation so first we're going to map the method names to the handlers we're going to create the multiplexer and pass it a reader and writer which will be our standard and then stand it out and then I've defined here two functions one defines the like any function that matches this signature is a notification Handler and it's it's going to take in the parameters and it's going to return an error no result because if you get sent a notification remember you don't need to do anything with that with that you don't need to return anything back to the server and also a method Handler so the method receives parameters and returns some sort of result back to the language server client in this case the editor and so then it's a case of making a map between the method name which could be like text document uh you know or it could be did open so like you know I've opened up um I've opened up a text file so you'd add that to the handle notification and so that's a mapping then of that method name to the function that implements that behavior um we want to be able to send notifications back to the text editor so the multiplexer also contains a notify method um the notify method has a mutex on it so a mutex is a way of making sure that two things aren't trying to do use the same resource at the same time so the right function locks the mutex and then defers an unlock of it executes the the right operation that we saw earlier and that allows us to build up then a notification which says okay I'm going to notify back to the text editor this method with these parameters off you go if we didn't have a mutex we might have this sort of scenario where like two things are operating in parallel and go routines and they both try and write to stand it out at the same time you might end up with this kind of scenario so there's no guarantee in an IR writer that there's the that two things won't try and write at the same time um but with a mutex you can make sure that the the messages are ordered correctly so that you only get a complete message through um final step in on multiplexer is to process those messages so we need to start up something that's going to read from the standard input bring in those requests work out whether it's a notification if it's a notification grab out our Handler and execute it if it's a method let's grab the the method function if we haven't got one then we have to return a response back to the editor to say you've sent me a message that I can't deal with and in your text editor you'll see like a warning usually saying like you know the LSPs said they can't handle this um if you do have a Handler you can execute the Handler and if if that fails you need to return a response to the editor to again say like well something didn't go right in in whatever I was trying to do and if it was successful then you can return the result so the key thing about returning the result is you know including the ID that you got from the request and then passing the parameters um and because it's go we can easily multi-fread this operation by adding a go routine in this one so as soon as we've read the message uh we're then free to throw that onto a go routine and and you know it'll process when it's good and ready you might want to control the concurrency in a more realistic scenario so the sample code that I'll show at the end has some concurrency controls to limit the amount of concurrency that you've got running on right covered a lot in a in a in those 15 minutes there so let's like have a quick summary of that so LSP messages sent over standard in standard out the messages consist of those mime headers with Json RPC we've got that correlation ID so the ID of the request and the idea of the response have to be linked to each other so um the text editor doesn't need to make wait for a response they can be interleaved and we can send notifications in either direction the editor to can send a notification at any point and also the LSP can send a notification to the text editor and now we've got this mux type that you can use to wire up these uh these method types into something a little bit more uh you know actionable um so the language server protocol also like now we've dealt with the kind of wire Behavior like the the sort of how messages are trans transmitted we go and we're going to go to the next level up which is there's a sort of initialization handshake at the beginning of a of a session essentially so when you open up a new project in your text editor the editor will start the LSP server so it's going to start your program and then it sends an initialization request over to your language server that initialization request includes uh its capabilities like the things that it can do and the LSP returns an initialized result which says well okay I've I've heard the things that you can do here's what I can do and between it those those represent the capabilities of of both ends so as you can imagine we're on I think language server protocol version 3.17 and we keep adding new stuff all the time so like I think one of the latest things that they added was a sort of uh I can't remember the name of it but like in function parameters you can add a little bit of text next to the phone like next to a function parameter so there's like new stuff being added to the protocol all the time so what you might do is say well I can't handle that or I'm not interested in handling that and that means that you can write a fairly minimal language server you don't have to take on the whole of the specification you can just say look I can only do these one or two things and you can still get value out of a language server that you create um finally the editor says okay I've got everything I'm all set up and then it sends an initialized notification back to the language server and at that point you're in sort of free-flowing territory you can you know receive and respond with any type of message so after this initial handshaking you're basically good to go um more typescript as we know because it's all defined in in typescript so here we've got an inheritance of work done progress and and so on and I think this is probably the most frightening part of language server is the sheer volume of types and the size of the types but the key thing that we're really interested in the client capabilities which is it is absolutely huge as a message so it includes all of these things like um you know can I handle workspace edits I was in like if if the language server returns an edit can the can the text editor actually process that edit so can the LSP trigger changes to the text documents um so I've brought out some of the kind of most important Fields so things like the inlay hint which is the the that's the thing I was talking about inside parameters which you um gets it added in and also Diagnostics which is like compilation errors and warnings and that sort of thing so those are the probably the most important ones I think but again we don't have to take on the whole of the specification to provide value and to create a language server so we can just take out just from the client capabilities what we want so here I've just said I don't really care what the client is I'm just going to receive any you know anything I'm just going to assume make some assumptions about the capabilities of the language server so the the capabilities I'm going to assume are that it can handle some of the messages that I'm going to send back but if you're following the specification to the letter you should be receiving those client capabilities and then before you respond before you send a message back or a notification to the to the client you should check that the client can actually deal with it otherwise the client will just reject the message um the server capabilities are a similarly complicated labyrinthine set of typescript um but again we can we can reduce that to just the things that we're interested in so if our server supports things like uh completion we we have to say yeah by the way this language server you're communicating with can support completion and one of the most important things is synchronization of documents so when the text editor opens up when you open up a document in the in the text editor it'll send the um a did open notification to the language server containing the full text of the thing that you just opened and that's great um but then after that as you start typing you haven't pressed save yet right like it's just it's in a kind of unsafe State and the language server needs to be notified about constant updates by the text document did edit um notification um and there are two two modes that the client will do that one is that it'll send the whole document every time which is great because it's really simple for a language server then to say well I've just received the whole document or you can also do incremental updates which is better for performance so in that case the text editor will say oh the user has just added enter in this particular spot or they've just deleted this range of text so in my example code today I'm just going to use the the full text of the document because that's the simplest implementation but if you look at the language server that I've uh that I've shared that I'll share at the end you can see a mechanism for dealing with incremental updates because it was probably like the buggiest bit of my code was sort of dealing with uh sort of partial updates to text files so I took a look at some of the other implementations that were out there um like the JavaScript one and so on and came up with one that works really well and go and performs better so we've got our we need to implement the initialize Handler so we need so this is a method we need to receive these these requests in and return that response back so we've got these initialization parameters so we need to start up our multiplexer on standard and stand it out uh register our initialization Handler we can unmarshall those params and then return our results and say okay right please send me the full text every single time that a user makes any changes to it and by the way I'm called example LSP then we're just going to Loop and listen to messages until uh standard input is closed so very straightforward for us to then sort of add on additional messages um and then the final kind of bit of the handshake then after they initialize method is when the text editor send us the initialized notification and so we can do something interesting at that so the kind of hello world of a full language server implementation would be to handle that initial handshake and then when we receive initialize we can start looping around and using Windows show message to send to to make a notification in the uh in the text editor so um I guess the most useless LSP in the world would be like you know hey you spent 20 minutes coding uh maybe you should stand up or something um right so let's go again so we let's go for a quick summary then the text editor Center initialize request the LSP returns a response that has its capabilities and then afterwards it sends that initialized notification that's the handshake of the protocol before using the capability you should check to see whether the client supports it or not um and we can start sending notifications back to the client with that notify method and we've got like a hello world um so it's great but how do you actually run the thing right like uh you've got a language server uh that was actually one of the challenges that if they were implementing it because it talks a lot about the protocol but not about the realities of how you actually test these things um I think about 20 of the go Community using uh vimo or variants and it's actually the simplest implementation is actually to do it in Vim um or neovim because um near Vim as if I think 0.5 or 0.8 includes language server support by default um so you can you can add a bit of lure to your initialization and this says like if there's a file called example dot example LSP in your directory then start up the example LSP program and as long as that's in your path um that's that's all that's required for uh for neovim to provide basic LSP capabilities and you can start up neovim with them like this uh dash dash clean to get rid of all of your other configuration that you might have and then you can test it in near him so if you do that new event you see this kind of ugly thing at the bottom that that shows you what's going on but for you know people who are like near them that's like awesome all right Visual Studio code is probably is probably the most popular uh text editor for go users and yes you're not going to like it but there's more typescript involved so in Visual Studio code you have to create a visual studio code extension which calls into your language server but actually this is the entirety of the the code although when you when you create like if you follow the instructions from from Microsoft's team there on how to do it it generates like a big project but actually the extension TS where all the action happens it's only a few lines of code to get it get it going and it's basically the same thing right if you if you have a doc I've if I have a file with a cook extension um then it'll start the example LSP command and as long as that's in your path it'll it'll work great we can also tell it that we want to use standard input standard output for LSP because it is possible to use some other mechanisms to communicate we then create the claim with those server and client options start it and we're good to go in in vs code um you can also uh oh yeah the other thing you've got to do is update the package Json to add in the language client so that's the that's the code that starts at the language server implementation and then tell it to activate when you open a DOT cook file so that's the activation events and you can also provide like a language configuration and syntax highlighting rules inside the same visual studio code extension so the configuration defines things like uh closing brackets so if you open up a bracket in your language and you always expected to Auto insert a close you can you can apply that in there and that can help with passing later because it means that your your object model is more likely to be correct we'll go into that in a little while um so once you've created one of these things if you open up your um your your extension project and hit debug it starts up a new vs code which contains your language server which contains your extension and is ready to configure so I'm going to show you how to make a LSP for the cook language which is about making recipes because it was a relatively straightforward thing to sort of uh explain um jetbrains in last month just in our support for language servers as well so um prior to that I think they had you had to do your own kind of thing in jetbrains but last month they produced this so again you have to create a plug-in and in jetbrains the ecosystem is Java so it's kind of in Java but you can see it's pretty much the same behavior as what we're talking about with the typescript one right and indeed with the lure one it's just you know does does this file extension work if so run you know the language server executable and communicate over standard input stand out so basically you should be able to support all three major um major editors without without much fuss so let's let's build an LSP for cook Lang um cook Lang looks a bit like this you've got then adds salt and ground black pepper to taste and paste a kilo of bacon scripts on a baking sheet so what we've got here is if ingredients start with an amphora or at symbol and the ingredients end with either you know a word boundary or they go all the way to these uh curly braces and inside the curly braces we can have quantities like Burr quantities like so you might have app free eggs and then three inside the curly braces um or you might have a sort of unit or a quantity you can also um describe uh required equipment like a food processor and you can put timers in as well which so that they have a similar Behavior um somebody made a a passer for go already so let's like use that and that allows us to do things like look for really silly measurements like American measurements called Cups sticks what are these no one knows what is a cup um so if you can pass the recipe you can look through the steps look at the ingredients and you know uh complain if somebody's using non-si measurements although to be fair in Britain we are slightly on a you know a country that measures oh what is it fuel in liters and then distances in miles so who knows internals um it passes line by line but it doesn't have a structured error so it it basically includes the line notification it's just text in an error message um if you're building uh a parser and you want it to be used in the language server implementation it's much better if the error includes a range like a start and end of worthy where the errors are because that then allows you to to place those notes like to place those errors uh in the user's uh text editor interface so if you can include a more structured error and include the start and end positions of those it makes makes the job of writing the language so much simpler and also if you include the um like in your if you're passing into an object model if you include the range of where that is in the text file as well that allows you to sort of iterate through the object model and look for different positions and say oh is the user's cursor inside this object or not um helps you to do like autocomplete and lookup so let's let's oh like so we've we've got the handshake done earlier now we need to handle opening up the file and returning Diagnostics so when the editor opens up a file it says you know text document did open you can pass the document find any errors and then return well it's not a return here sorry you can send them a set of Diagnostics down to the text editor for them to display um so they did open uh notification wrong wrong one they did open notification includes the content lens um like we saw earlier and then it's it's basically the full text of the document you just opened um and then given the user might make some changes to that far right and those haven't been saved to disk so the did change will also then include the updated document as well and then we can push the same Diagnostics and the did change is basically the same structure as they did open except it's got an array of content changes so you you might receive more than one content change if you if you're receiving um like partial updates so like oh the user added enter here they pasted this section in they deleted this line you might you might receive a block but in our case because we tell the text editor hey we can't deal with uh like partial updates just send us the whole document we're just going to get the whole of the content in the in the text and that makes it much easier for us to implement this um so we want we now want to start processing this stuff and and dealing with these changes that we're receiving is a language server um and so we can start uh we can again we can make use of goes concurrency here to make a channel that contains a list of like that's going to receive updates to to the documents and then as we receive those updates uh we can start finding our Diagnostics so and then notify the client about the the diagnostic so that means like if we get a passer error or if we get um like quantity errors or other information we can notify back to the text editor and the text editor can display them so we need to register some notes some some method handlers so some notification handlers so like to receive that open text and change text and push those into our queue of changes so it's a case of subscribing like subscribing um a Handler that's going to send notifications and then also handling those notifications and pushing them into that queue the Diagnostics are themselves then are fairly straightforward we can start to do things like pass our cook language and if it's uh if it's our cook Lang era if it's not it like we can then start to look at the ranges of that and then apply those those errors to the to the text editor um so the first thing is to pass the text check the type of the error like is this an error that's coming from from the the parser or is it some other type of error if it's not we can't read the ranges and then we can start to apply Diagnostics so we can say right okay this this uh this error is happening in this location and we can pass the error message uh straight to the user um I also created a I swear word Diagnostics which just looks it doesn't pass the text it just uses like a list of known English swear words and says you know you've included a swear word in your in your document very straightforward to to do um and even that simple language server gives us the ability to do things like like this so invalid block comments so showing you straight away before you've even saved the document that you've got invalid syntax in your file and you also get timer syntax because these are already built into the past so even if you're if you're working like in a in a small language like D2 or like graphviz or something else that already has an existing parser you can provide like a lot of value um by making this minimal type of LSP and just shipping back the area straight to the user and then that just brings those errors so that they're not waiting till they've hit save on the file and gone to the command line and run it run an extra command you're bringing that to them a little bit earlier so yeah in summary it's possible to make a LSP for pretty much anything that has an existing go passer or uh and and you can pass that into an object model um and then use that as the basis of your like notifications back to the user about like where the where the errors are in the file if you include those far positions in your object model then it makes it easier to display the Diagnostics in the correct locations um and you don't need passing to succeed to produce some Diagnostics so like you know it's it's helpful if you support like partial failures so like have a good go at passing through the document and still return like a list of a list of Errors back to the user so let's let's look at autocomplete so it should be pretty familiar by now I think like you know you're probably getting the idea that there's methods there's notifications and each method has a name each notification has a name so it's basically the same process we need to add a new method Handler to our multiplexer in this case we need to uh tell an issue we need to update our initialization to tell the the text editor that we support Auto completion so in this case we can set the trigger characters so we can say that when you when you type in your text editor a percent uh like uh sorry um yeah percent symbol that should trigger the text editors to send a autocomplete request to the server and that's done at the initialization stage so when you set up the client server capabilities and then we can the other thing we need to do is update our change handling so as we receive updates to those text documents um if you when we get when we get to receiving the auto complete request we don't receive the text of the document so we have to keep in memory somewhere the the current text of the the file so that we can that we can look at the correct place for auto completion or we have to continuously keep a cache of um like where Auto completion requests would be successful within within within the LSP so at some point you have to essentially cache what's coming like the text changes that haven't hit this yet and then we can implement the completion itself um so the completion parameters include um like the name of the file that you're that the person is in at the time that they're asking for completion and then we can start to look for completion items uh in that in the position that we've received from the from the user so in this case if I look through each of the steps in the recipe and look for each other ingredients either can then say well if they're inside an ingredient there and they press percentage they've got to be in the unit of measurement area and then I can send them back a list of of potential measurement units so milliliters and kilos and stuff like that um the the passer needs to retain something useful at all times so if you're making a passer like as you're typing at pizza and you put in the curly brace you've actually got an invalid cook Lang document at that point because you haven't got the closing brace so earlier I showed that you can you can trigger the you can trigger the text editor to insert automatically the closing brace and that would give you a a completely valid document but you should also be able to handle the case where you know I've actually you know I've opened up a brace but I haven't completed the brace you still need to be able to produce Auto completion results at that point you can't just throw an error and complain about it so in this case it when I use when I use cookland.pass it returns an error of unterminated quantity because yeah the quantity isn't terminated it doesn't have a closing brace but it also uh returns a range that says yeah but it but we know really that there should be a quantity at this point so that that's that's going to be the range where if the user's in that and they and they trigger Auto completion I do want a full list of results coming back um new of him like always makes it very straightforward it's a simple control X control o to trigger to trigger Auto completion to to to to show up or you can introduce nvim comp but uh Visual Studio code has it built in so straight away now we're seeing um Auto completion happening there so when you it knows that you're in the the sort of ingredient section and it's then able to provide you a list of of valid quantities right so most of the time in my kind of day job of making apis rest apis or json-ish apis you generally log into standard out right but we can't log to standard out here because standard out and standard in is being used by the communication for the language server itself so typically what you want to do here is log like enable some sort of logging to a file so you can see what's going on um with the new slog Library you can create uh you can create a file pass that into slog and then use Json logging and then inside your handlers you can log the parameters that you're receiving from the from the text editor and that's really useful to be able to see what you know what you're getting from the text editor so that you can interpret you know what your results should be coming back um over the last few years I've been using these techniques to build a templating language for go so I started in 2021 and it's kind of like like jsx in the JavaScript ecosystem or Razer pages in C sharp but book for go um the uptick there is is a Reddit post basically if you want to publicize your stuff stick it on Reddit it seems to work um and it's strongly typed it's compiled and I wrote it so that it would support LSP implementation so you could get like auto complete and stuff when you're making HTML pages and go it's designed to work with htmx which is you know all over Twitter worth a watch um and I've been using it over an insurer to produce all of the PDF documentation that you get when you buy car insurance um so I think most people go well why would you buffer girl has templates built in right like uh like these ones here but actually uh this this go template will fail on program startup because I didn't register the two upper function as a function map into the into the go templating language um and also this one will fail at runtime because I re I referenced a price field which doesn't actually exist in the in the struct so the this is a problem because you know what we're trying to do with like the whole point of having things like you know editor uh implementations of language servers is that you get feedback when you're typing before you've even hit save on the file and if you're only failing when you when you're in production then it's much more expensive much more difficult to deal with even if it fails at CI CD that's like five or ten minutes before you've you've found out that you could have maybe dealt with that a little bit sooner and you don't get any syntax highlighting your intellisense I know that there's there's some work happening around like a language server for um for the for the go templating language but it doesn't seem to be working for me and maybe yeah it's also much faster as well but you know if you when you're talking in nanoseconds where it doesn't really matter I think for most templates so yeah the language itself is um is fairly straightforward to read it looks like go code but with HTML in it because that's what it is and um you can Define like the types that are passed to a template by by it looks like a function because that's what it is and you can then use the name in scope and it'll kind of output the results so inside those braces that's a go expression so um you know outside of it it's HTML inside it's it's go and you can call templates with each other like like function calls uh and you can use normal kind of ghost constructs like if States and so on the way it works is when you when you generate it it generates a go code so use Tempo generate the CLI and it generates a go like a go file that that outputs all of the HTML and does all of the interpolations and that gives you compare time errors and so on um that makes it relatively straightforward then to create a vs code extension so here here it's a were that the last name field doesn't exist on that person because the um the go code isn't isn't working right and the way that the way that this this operates is that it uses a it's it wraps the go LSP itself because if you think about it that that position there on line nine the go code isn't failing at that place right well the go code is failing at a completely different place in the generated code um and what Temple does here is it it creates an LSP implementation and then passes the the temple file works out which bits of that are go code and and well it generates then a it generates the go code and produces then a position mapping of where that go code is related to the temple code uh it then communicates with go please the go language server and Returns the results back and remaps those positions and what that allows it to do is make use of the uh the go language server so you can you can do that with other languages as well so if you have like um if you have a programming language that or or a like a templating language that then wants to drop into other other sections you can you can proxy in between different LSPs if you wish um and another debugging technique that you can use is although standard and standard out is is taken you can still start a web server so internally within Temple um it started it starts it can start up a web server which allows you to inspect what's going on inside your language server so in this case it keeps a mapping of uh all of the all of the on the left hand side you can see the temple code and on the right hand side you can see the generated go code and the green bits are the mappings in between the two things so you can see in real time like as you're editing text you can go onto the web server and kind of inspect it so that's a really useful technique if you're building your own language server is you know put stuff like that into your into your tool so you can see what's going on inside um so yeah hopefully from that you get the idea that the idea is to move the notifications earlier to the developer and you can also piggyback on other LSP so if there's an LSP that does what you want and you want to use bits of it you can probably proxy in between those two things and you can have a look at Temple temples LSP to see how to do that feel free like to feel free to start up a web server for introspection into into your tooling and of course P Prof can also be attached to get like information about what's going on in your language server um I guess the challenge then of of dealing with the language server then is passing the input in the first place so the temple language that I mentioned earlier uses the this passing library that I wrote and it defines an input and then the input has essentially a pointer to a location in that string so its initial position would be zero and then as you successfully pass a out of the out of the stream it moves the index to one and then as you do B it goes okay I'm in our position two and then if you fail to pass it doesn't move the it doesn't move the position it's like no you didn't get it um so with that you can then start to stack up these functions into sort of bigger passes so here's here's a passer that's going to look for four digits in a row um and you can say okay well I'd like to pass that year and here you can see it's returned the result of 2000 it's past successfully the first four D so it's equivalent essentially to the regular expression of backslash D with you know brace four um you can also then create like passing to objects so it uses goes generics so you can create a pass a function that like passes into a time struct instead so here I've got a year passer that will look for a four digits uh for for the use that four digits attempt to pass if it's unsuccessful it'll wind back to where it was at the beginning of the passer so it's a recursive descent style and then it will return a date if it's successful um you can you can this is what uh Temple uses to to load into its own internal object model so if you've imagined like uh the sorts of things that you're going to have in a HTML document you're going to have HTML elements and those elements are going to have a name uh you're going to have attributes on those HTML attributes those attributes can be of lots of different types so they could be um they could be like expression attributes Boolean attributes CSS attributes different types of attributes so there's an interface that defines those and then inside your element you might have other like a series of nodes so those nodes might be if statements but they might also be other HTML Elements which they then themselves contain a tree of elements so you want to create like an object model that contains this kind of document so that we can provide that LSP functionality um and so it uses this passing library to do it so a self-closing element is a like otherwise known as a void element uh is is one that has like a closing slash at the end so the password for that would style work out what its current index is and then attempt to pass the less than symbol uh if we're unsuccessful we can just exit you know like hey this is not uh at this location in the file there's no self-closing element to be found um then we can try and get an element name so if we don't like an element name consists of uh like a non-digit uh so non-digit like a a to z type character followed by a number of different other characters including like hyphens and and colons and such and then we start to bring in the attributes if we fail to get any of these we have to seep back and wind back our parser to where we started and we can return uh finally we'll get some optional white space and then we close up our our void element and building passes in this way lets you create unit tests of individual like blocks of of code so for instance the passing the the um the anchor tag here I can check that the object model I get back uh is is what I expected and that the ranges of of the expression there are in the correct place and that's what that's one of the reasons that's one of the ways that it makes it fairly straightforward to identify which is which bits of go code and which bits are sort of HTML code within the within the object model um you can then again build these up into higher level and higher higher level uh passes so that you can have like an element pass of that sort of again does the same thing you start like works out its current position tries to pass an element like is it self-closing or is it an open closed type element and then after you've passed it into that object model you can start to validate it like what problems are we seeing inside this like oh do we have any empty attributes do we have any other issues and we can return a passer error there um so the kind of this library is like a bit of an alternative to tools like antler or Peg tooling so if you've kind of seen pasta expression grammars and so on like uh you might find them a little bit intimidating and also they produce they tend not to produce extremely good errors to for users so if you're trying to build like a a little language and and provide really good errors for people it can be quite challenging to do that um so this this technique uses like a pass input which keeps track of that position and the pass error which returns exactly the position of where the arrows occurred and it and it helps to provide good messages to users um so yeah again this is a way of building up small testable passes that you can use then in your language server implementations and and this pattern lets you compose small passes into bigger ones which has allowed me to sort of maintain the project quite successfully over a couple of years like it's quite easy to add new things in with a really good solid test suite and when I get an error I can kind of see where where I'm going wrong so the beginning this is what we said we covered today so it's like what is the language server protocol so if you remember we went right down to the basics of like the wire protocol itself like the the headers the Json RPC body and and how the ID is mapped between request and response we went through notifications and and methods and how those are different we went through the initialization handshake how the LSP client has to tell the the server like what it can do and the server has to tell the client what it can do in return and that's how that that begins and then we've got request response handling of how we have to open we have to deal with the did open request and the did change requests and return responses back to that and we also showed a few notifications like how we send notifications back to the editor to include things like diagnostic information and so on covered a little bit there on how you use the LSP in an editor so in the oven setting up the lower configuration in vs code creating a vs code extension and then jetbrains uh you know using Java to create a plug-in for for go land or similar and then we talked a little bit about logging like we can't use standard input standard output don't just try and log stuff you'll just break the LSP you need to log to a file instead and pass a logger around uh then it covered a little bit about the temple language how it uses some of these techniques and and then went into some detail on how that how you can build up passes that that will allow you to sort of pass those object models and and build up your LSP implementation and so what I'm hoping is like uh is that people who are like using a language like the Nyx language or D2 or lots of other different small kind of text-based languages will kind of find it a little bit more accessible to to build up like these these text editor plugins that I think are really valuable for helping developers to be efficient um I've got some resources here at the end so uh this is like the slides for the talker available there on on GitHub in markdown format we've got the example LSP all the source code there if you want to play around build your own LSPs on that um and there's some links there to Temple itself and you can inside the temple LSP you can see some really that's worthy um that's where this partial update handling is so if you're interested in sort of in really efficient LSPs you can see how that works um and the passer library though as well and also the lsp.dev so there's a team out there who've built like you know all those complex typescript object models that I showed you earlier with the Brazilian parameters and so on uh this team here has built a go package which can contains all of those defined as ghostrooks already so you don't have to sort of tackle that yourself basically um so yeah thanks for coming along and uh yeah I think we've got five mins uh maybe for questions as well if anyone's got anything to [Applause]

Channel: GopherCon UK

Views: 7,496

Keywords:

Id: EkK8Jxjj95s

Length: 54min 24sec (3264 seconds)

Published: Fri Sep 08 2023