Creating Your Own Markdown Preprocessor Is Easier Than You Might Think

hey friends what's up today I have something really cool for you today we're going to make our own MDX so you might have used MDX in the pass so this is really an awesome pre-processor for swell you can use this to use swell components inside markdown and vice versa and if you're interested in just using MDX I have an entire video on making a Maron blog with SW kit and MDX but today we're not only going to learn how this works under the hood because you're going to recreate a bit of a mini MDX right we're actually going to learn a lot about how swell works and Etc and I think that this is the ultimate markdown experience that I want because I want absolute control over it so let me show you here in our examples as you can see this is just a regular Magnum block here I'm using a counter component inste nothing special here I have this viget which we can look here I have this marked on text I can change it so you can have interactive examples in your PO to make them more interesting right let me just change this color orange red let me actually show you something interesting so in this case maybe you would want to use a KN frame or something because here we're using this if we remove title and we press title we're going to Target the elements on the page and this is also kind of like really interesting maybe we can make like one of those blog cms's with live editing and Etc I don't know this is very interesting in my opinion but yeah you can also Target let's say for example we can Target the Pags on the side and you can see this looks like a complete disaster right so maybe we can revert this to titles just so this is affected as you can see here we're using a modern synx highlighter Shiki which is really awesome and I also have an entire video on using Shiki and why it's really awesome but as you can see this is a really a perfect example using this SW code snippet why Shiki is so much better than your standard syntax highlighter because we have something like this which is a swell syntax we don't really need to specify this syntax it works absolutely beautifully and the syntax highlighting is simple you're going to be surprised how simple it is to implement something like this and yeah that's basically it in the next part I actually want to show you how a pre-processor actually works so what is our goal and Etc how are we going to achieve this right just understand what the pre-processor is even though I have also an entire video on pre-processors in swel and why they so powerful all right so if you look at our markdown here is nothing special this is just a regular markdown file which is our post and this gets actually treated as a swell component I'm going to show you how that works so here we have this front matter nothing special but as you can see now we can use our script tag because this is actually just treated as a swell component so we can import our components here here I have a counter and vidget and if we look at them we're going to see nothing special can also show you the widget so we can open the widget and if you're wondering how I made this widget it's really just a cheat for a simple editor I just stacked a pre and text area on top of each other so I get this awesome syntax highlighting using Shiki again here very meta right but actually I want to show you how this actually works so what is our goal so you better understand the power of a pre-processor in SW all right so if I open a new pane here on the right let me just copy over everything I'm going to create a new file and I also even kind of don't want to even include syntax sting because I kind of want you to understand that this is just a string and whatever string is here we're going to feed to the swell compiler and before we do that this is where a pre-processor comes in because we can actually write a pre-processor that can turn this string into anything we want so any valid swell component syntax is going to be valid at the end and actually let me show you what I mean by this so actually I can copy the same code again this is a new file let me just use the pointing but I'm going to change the language mode back to S or whatever so actually our goal is going to be here to turn what we have on the left to an actually valid swell component so for example here we have this front matter what we're actually going to do is we're going to use something like gray matter to parse this thing and then we're going to have an object and then we can use context module so we can say for example I think it's context module or module context it's really not important and then from here we can actually say something like export meta and then we can export an object with all of our data right and also imagine what possibilities this unlocks like this isn't just for Mark Etc you can make useful pre-processor for anything you want for example let's say that maybe you want a table of contents on your page you can just export a table of contents right and now boom you're going to have this actually on your page not just inside of your markdown where you have no control over it right you can actually have a table of contents you can do whatever you want and then the rest of this is also going to be very easy so we're going to just turn markdown to HTML and we're going to use a marown pre processor right so everything here that's not marked down is going to be ignored anyhow so this components for example is just going to be turned into a H2 right so this is going to at the end of the day turn into something like this and this is how simple this is this of course going to be a pag and this is basically what we're going to do using a pre-processor that is how simple that is so anything that we can turn at the end of the day into a valid swell component is what you can do using a SW pre-processor oh and by the way if you don't know what context module is which I don't blame you for this is something that it's kind of rarely used to be honest if you have a prop drilling problem you should probably use the context API but in this case it actually works great because we want to export something on the module but what the heck is actually a module let me just explain this quickly so for example let's say that you're writing here something like function counter and Etc so this file where you're writing your JavaScript Etc is a module right and under the hood this is how actually components work because in spell five and moving forwards components are just functions right but for example let's say that you have multiple instances of a component and you want to share some State across them well actually instead of defining it inside we can Define it inside context module so as I showed you before we can just say cons meta and whatever we put inside of here is going to be available in each of these counter instances and that's how simple that is so now we can take that we can just parse out the front matter so actually in the next section I'm going to show you how I have this project set up because I'm not going to do everything from scratch but if you want to dive deeper into it and I'm going to have a link in the description all right so here I have a simple swell cre project I'm importing this swell down pre-processor from Source lip swell down this is where we're going to spend most of our time because we're just going to focus on making the markdown pre-processor right and we have to specify the Dos extension because we don't have really any pre-processing here from V because this is outside the scope of the swy project but you can publish this as a library maybe somewhere else and then you can use all your tooling and Etc whatever you want if you prefer to use typescript and Etc and the only important thing here is using extensions right so you can specify extensions do SW and MD but this can actually be banana if you want and this is going to work and maybe you can also associate this banana file with something else but you can actually specify anything you want and here what you specify is going to be treated as a swell file if you want to learn more you can watch my entire video on a swell preprocessors and here is the actually also important part here we're going to pass this pre-processor and it has to come before anything else so we want to turn this string into something else that we can pass to the swell compiler and this is also going to get changed by the vit preprocessor right if you're using SAS T script and Etc so you want all of this to be done before it gets passed to the V preprocess and here another important thing I have here since I have in the root here post so this is how I want this to work I want to have an example here and I just want to have post inside of here and components right and that's how I want my marown structure to work so I'm going to just dump post here and I want to include any component that I might want another thing that you have to do if you're going to have this post folder at the root in your read config you have to actually specify this option server file system allows this is going to go one step up to serve files right otherwise you're going to run into some weird we error right but that's how we can do it and for the actual project there really isn't anything else that you should do this is just really on routes I have this post route here I have this page SW and page. TS where you're going to get the data so here if open page. TS as you can see because we specified MD to be treated as a SW file now we can import it here as a module and you can see this is how this works so we can get the content from the module and we can also export the metadata right and this can also be table of contents Etc whatever else you want as you can see if we go to plus page. SW we can just get the data from the props and we can output it on the page and this is how easy peasy lemon squeezy this is friends but in our case we're just going to mostly work inside lib swell down. JS and you can actually see how simple this is this isn't actually any amount of lines of code it barely has any lines of code right you're going to actually see how simple this is to create something like this all right so let's start working on the mardom preprocessor and here I'm going to save us a bunch of time from installing all of these dependencies so you can do that yourself if you're following along but basically we're going to use UniFi which is just a tool chain for working with text so we can convert markdown to HTML HTML to markdown and Etc there's other formats but you probably heard of this and their plugin ecosystem which are remark and rehive plugin so for example here I have this remark GitHub flavored backdown so we have tables and a bunch of other things and then I have remark smarty pens so remark smarty pens is going to use actual typographic quotes instead of those fake quotes that you use in your post right and then we have something like Rey Shiki which is going to be used for the syntax highlighting and then I'm going to use gray matter for passing front matter so we can return back the content and the metadata all right so the first thing we have to do is actually declare our processor I'm going to name this swell down but you can name this whatever you want right and let's just say export default swell down this is actually going to be it there we go and now we also have to return now some methods so we can actually change the script section the content section which is going to be the markup and the style text section but in our case we just want to change the content and again if this is going to fast or is confusing to you watch my entire video on how pre-processor in SW works so this is going to be an acing function markup and now we can D structure two things we can D structure content and file name and that's it and now we can say for example console log content and you're going to see this is going to log everything so let's start the development server and you're going to see it's going to spit out everything that it open so let me just refresh this and you're going to see it's all of the components it interacts with once it boots up but this is the reason why we actually pick that do MD file but it can be whatever you want it can be banana and Etc so to only console log content if file name ends with MD in our case and let me just on log out and again the only thing that sucks here is we have to restart the development server when we're updating this file here what you're going to see now we only get the input for our markdown component because we're only targeting MD files and another thing here that sucks is this vague error message so for example could not find example and it's really annoying because where is this error here in our terminal right so how can we know this and basically this is a scale issue because that's a me problem so for example here in Plus page. Ts I actually have this error here could not find po but we actually should output something more useful so let's say for example in this case let's output the actual error and you're going to see now when we refresh this let me just see now we should get this error so this is more veros and when we read this error here it says post example blah blah a component can have a single top level script element and a single top level script context module element okay so this might s confusing but actually if you just go back to our post and if you just strip things down one at a time you're actually going to find what this error is and this example the problem is we have this Cod block we're using this script but this actually closes the script tag right so s is really confused and we also have this problem because these curly boys are going to be interpreted by spel as an expression right so let me actually prove this to you for example if I remove this cold block everything should work of course it looks like junk is but actually we're not going to get any error so you're going to see everything works as a expected kind of right but for now I can just return this and when we get to processing the markdown we're going to uncommon this but for right now let's just comment this block and that's basically it so now we're going to see if you refresh everything is going to work as expected now we get more useful errors okay so let's start with the low hanging fruit which is going to be front matter so I'm going to go back here to our code and I'm going to create here a front matter function so we're going to process front matter it's going to take in content and that's basically it all right so now we can use matter and we can actually the structure two things here so we're going to pass content to matter so we can get back two things content and data which we can return here we can return content data but the problem here is that we already have content here so let's rename content to markdown and then let just return that say markdown here and that's basically it so here I'm going to yink this and now we can dest structure from our front matter two things again we're going to pass the content so now we're going to get back markdown and data so if I console log this out console log markdown data let's just restart this refresh the page so now we're going to see that this is actually split in half so we can see here we have our markdown which is this right so we can see this is our plain text and here is the front matter so actually this is passed for us we don't have to do anything special so now we have title description date categories publish true right so this is from our front matter but to make this available here to our component we actually have to return this on the module same as MDX does so what you're going to do here I'm actually going to create a meta variable and let me just use template literals like this and again we just need to construct a valid swell component this is absolutely awesome so we can say script context module and we just need to export the metadata right so we can say export const we can say metadata and now we need to serialize this because we can just really pass data right this wouldn't work so we actually have to say soon stringify data and instead of data now we're just going to pass meta here all right so now we're returning markdown and meta so now we split our front matter and content so we actually don't even have to worry about removing the front matter right how beautiful is this friends and now actually this will be so much easier with tipt but we're moving blind here for so we can just actually say meta and let's just do this let's save everything and again since we made changes we have to control C kill it and start it again and let me just reload this just so we get everything here and now you can see here we get our markdown content and here we get the meta so this is just a string and of course everything in programming is a string which is absolutely beautiful but this is actually going to be a valid swell component and now when we import this Mar post as a module we can actually Destro structure this from it so let me actually show you that to actually return the change value here is what we need to do let me remove this we have to say return and we have to return code right so we can for example say code content but in our case we actually want to return meta plus the code that we have so for example meta plus markdown let's just restart everything now when we go here and refresh you're going to see now everything works because this is actually the front matter that we par and again if you lost let me just reiterate that here we get the data for the page from the post here we importing this from the module right and then we can just pluck those values so for example this can be content metadata it can be table of contents it can be whatever you want so now that we have this we can send this to the component itself so let's just go to the component itself in Source routes SL poost right and now we get this from data what do we get we have meta so we have data do meta where is our metadata from the front matter and we have data do content which is our content itself right and how beautiful is this friends all right so now we have the content but we need to par the markdown right and this is going to be very simple so I'm going to go here and I'm going to say ASN function I'm going to name this ours markdown which is going to accept content and the only thing we have to do is create a markdown pre-processor or processor right since we're using unified so they're called processors yeah so we can just say con processor and now we can say Unified actually let's await it so we can invoke unified and I'm going to go to a new line just so everything is neat and tidy all right so now we have to turn our markdown into an abster syntax tree so we can parse it right so for this and using UniFi we can just say use and we can say to markdown as and then we can use the markdown plugins so I'm going to say remark GitHub flavored markdown and I'm also going to use smarty pens but you can include as many plugins as you want all right so now when we're done with markdown we need to turn this into an abstract HTML tree so we can say do use to HTML as and we have to allow dangerous HTML because we're using things like script tag and Etc which otherwise would get parsed out or replaced with HTML entities I'm not sure all right so now when we have this HTML tree this is the perfect time to use our rehive plugins so now we can just say recap Shiki and boom they can also pass it some option for example which theme I use I always get as this I use po mous and this just happens that this is a team that chips with chiy if you have some more esoteric team you can of course load it all right so now that we processed all of this so now we have a tree right we just need to turn this into HTML string that we can render so we can say use to HTML string and again we have to set allow dangerous HTML and that's going to be it and now once you're done you just have to say do process what we want to process we want to process the content and because this returns a processor we actually just want to return the result of this to do that we can just say return and you can just say processor to string and that's basically how simple that is awesome so now here where we have our markdown we can just convert this or transform it from markdown to HTML so we can say F HTML equals we can say a wait what we want to do we want to par the markdown so we just pass the markdown and that's how simple that is and let me just console log it out that you see that is true so I'm going to to restart everything just restart the development server refresh the page and boom now you're going to see all of the markdown is going to be transform you can see our component title went from markdown to HTML exactly as it should but actually you're going to notice that you're going to run into some problems because for example you can see how we're passing this counter component with this curly boys which is perfectly fine this is a regular SW component but remember what I told you about Expressions this is where we're going to run into a problem as you can see for example here where it's on click and it has this curly boy SW is actually going to try to interpret that as an expression so here for example we actually don't want to return code meta plus markdown we want to return meta plus HTML so when we save this let me just refresh this we're going to run into an error as you can see we get an internal error and you're like okay what is this count is not defined and Etc and then you just start looking at this this massive a right which might look confusing but actually it makes perfect sense because for example let's just go back to our blog post again I'm going to remove all of this as you can see it works fine again that pesky component right so what we have to do for the last part we actually have to replace those characters with HTML entities all right so going back to our code beautiful all right so now let's leave it as is and now we can actually create a new function here and I'm going to name this function Escape HTML this is going to accept the content and we going to return the Escape content and let me actually close the terminal and for this part actually stick with me because this might seem a bit weird but there's a method to my madness so for example what I'm going to do I'm actually first going to replace all of the entities in the file then I'm going to specifically Target the components with Rejects and I'm going to return them back from HTML entities to the regular curly boys and I'm doing that because you actually run into some weird issues and annoyances with how you convert that using Rex right so gets really complicated but actually found that this is the easiest way and of course you can do this your way so the first thing we're going to do we're going to reassign content I'm going to say content replace so we're going to actually pass this Rex here we're going to use the opening curly boy let's just make it g or Global or ganga whatever you want cool and now we only have to replace it with this HTML entity which is going to be m% found side 1 2 3 and then colon and let's also do the same thing for the closing curly boy so we're going to say replace it's going to be closing and it's not 1 12 3 but it's one 125 okay so now that you converted all of the text we just want to revert the components so for this we're going to use a component Rex I'm going to say cons component Rex and this is really simple so here we're going to create a Rex we're going to say it has to have this opening pointy boy it has to start with a capital letter so we can say A to Z and then we can match everything after that of course you can be more concise with this you can Target only the components that have the curly boys in them but that's up to you I haven't found a problem with this approach yet and of course let's add the global flag all right so now we just need to find the components Loop over them and replace them so I'm going to say cons components and then we have to say content match and let's pass it the component rejects that's basically it all right so let's say components and they might not exist so let's use question mark for each so now we're going to Loop or the components and first we need to replace the value of the component so we can replace the old value with the new value so we can say con replaced and we in this case need to do the opposite thing for the component right so swell treats it as a regular component so we can say component replace and here we're going to first start with the entity so let's just copy this over so we don't make a mistake and what we want to replace it with with the opening curly boy all right so now we can just copy this lower because we want to do it again but for the closing curly boy so let's just close it and instead of one 123 it's 125 all right so we just need to return what we matched with the updated or replace component right and that's how easy peasy lem and squeezy that is all right so we can reassign content we can say content replace replace the component with what with a replaced one right and then once that is done we can just say return content and that's actually it so now we can go here to our pre-processor below HTML we can actually say cons code and we can just say Escape HTML we can pass in the HTML and now instead of returning meta plus HTML we can actually return code all right let's save this I'm going to restart the development server and let me actually log this out just to see what's going on Al a Cod let's do that and now we're actually going to restart and now if you look at our component everything looks right here we have the script again we're just aiming to make a valid swell component nothing special as we can see here this looks right this is our regular component so we did that right and let's actually see when we look here this should be replaced with our HTML entities so we should find no cly boys here so remember how it was complaining about count or something because it couldn't find it well here it is count and again it's not going to be interpreted as an expression by cell because we're actually replacing this with HTML entities how beautiful is this friends all right so now when we go here and even if I refresh it for good measure you can see everything works beautifully so now now we can also change this and do whatever else you want right all right that's basically it regardless if you just want to make a Blog with empty SX or if you want to learn how to do it yourself and have complete control by making your own custom Maron pre-processor I hope you at least learned something all right so if you like what you seen don't forget to like And subscribe and you can also support me by becoming a patron thank you for watching and catch you in the next one peace

Channel: Joy of Code

Views: 5,827

Keywords: sveltekit, svelte

Id: UFSmNnWPvrM

Length: 24min 24sec (1464 seconds)

Published: Fri May 24 2024