Scala project from scratch #1 - rough idea, setting up, design decisions

hello everyone uh this is going to be a completely new thing on this channel so i'll tell you about the idea first so i wanted to make a larger project something that you could follow along and watch me you know solve everyday problems in implementing such a project this will be built with the type of stock using a lot of the functional stuff that i've been showing around on the channel in other content that i've been posting in my talks so i just wanted to have something more complete uh something like a full-blown project uh where we will actually go through the face of implementing every every layer of of the code of the architecture and yeah this will be it so this is going to be quite a long series i predict this will be about well at least 10 20 episodes and each one of them will be about an hour maybe an hour and half long and i will not really edit you know the silence out and so on because it's just too much too much video to edit to go through uh so this will be kind of like a stream but recorded offline uh maybe if you if you think it's going to be fun i'll do it live uh a couple times uh i guess we'll see so yeah i think i think we can start so the project i wanted to make i i briefly mentioned this idea on twitter the other day uh it's going to be a docker clone um but it's not going to be like a full blown docker clone in the sense that it's going to deal with like file system images something like that we're not going to have virtual uh well we're gonna we're not gonna do containers we're not gonna do anything like that it's actually going to be uh well stealing just a couple ideas from docker like having a cli that connects to a demon running somewhere but it'll be mostly like a compiler with caching i guess i guess we'll get to that later when i'm actually implementing this kind of logic uh but yeah uh actually i'll show you a demo once we have set up the project so this will be like a complete walkthrough with like all the like there will be no behind-the-scenes work uh i'll show literally everything that i do to get this project to happen and i realize there's music in the background it's supposed to be free so i hope i don't get a good great strike so we'll create a project uh with sbt uh i have made some decisions already like technical decisions on what to use in this for now we'll use sbt maybe in the future we'll use something else uh so yeah here's my empty window in vs code also i'm not entirely sure about uh whether i want this to be watchable on like mobile phones tablets whatever it would be better for me if i could you know make the phone smaller and just work as i would in a normal project but if enough of you are going to watch this series on devices like that then i'm going to keep using something that makes it readable on smaller screens uh otherwise i would just use something that's that can be watched on a desktop so the project name is going to be steveter stevedore it's apparently a synonym for docker so i decided to use it and just prove that there is nothing in the project yet uh so there are several ways to start an sbt project uh i'm going to do everything by hand just because it's going to be a custom crafted project so let's open vs code in it okay uh that's my vs code setup uh first i'm going to show you like the general idea so when you have a docker file uh the font size yeah hopefully this is large enough uh so every docker file starts with like a from and you have a base image so let's say you have alpine uh optionally you can specify the version like this is a tag or you can probably put push well put a hush of the image that you want to start from uh then you can put like a couple of comments like run i know echo fo into a file like build result and then you can copy files from the current directory or a different image um i'm not gonna do that i'm just gonna do this and then you can specify like the command that will run when you actually start uh a container uh so like my nsh uh let's see yeah let's let's show this file that we printed and when you when you build this with docker i'm just going to show this to have a fair comparison of what we're gonna do like how many ideas we're gonna take from this so when you build this uh this is an image uh okay this felt because we don't have a directory that's fine uh we're gonna create the directory okay so the image was built we can optionally like tag the image when building this is going to be important um later i will just do like demo 01 and it doesn't really rerun anything all the commands here are cached which is which should give you a hint of what i'm going to try and accomplish in this this whole project and we can refer to this image either by this hash this sha 256 hash or by the the tag while the tag is this so now i can run it i'm going to use the rm plug so that it removes the container when it's done and this didn't work because i'm not good at this why didn't it work um let me just try this well the business 8 should be available uh let me try with more white space here i don't think it should matter but not this i think we can just to cut like this maybe it will work i don't deal with docker that much to make my own i own images that often so yeah it doesn't see that file which is strange build result oh i see this is cut it needs to be a separate uh argument to this when i need to build the image again okay so now it works so basically the idea of an image like this is that you start with some bass uh i don't really know how these are created like how the the empty image can be defined but that's just a detail that we don't need to concern ourselves with here and then you can do a couple of commands and maybe we can copy like uh let's uh it was supposed to be a file like we can additionally do like copy uh [Music] rest press and maybe we can print both of these files so still the the first run command was cached and it should reuse the layer every every command creates a separate layer which i'm not entirely sure how they are built but supposedly when you have a run it will basically diff the file system and like sort the changes don't quote me on that i'm not a docker internals expert but the most important thing is that when i build this again everything that i haven't changed like before a change like it's if i change this the previous things the previous layers are going to be cached and we're going to reuse them uh they will be cached in my like local docker registry and each of them has like a hush so the the registry will store these hashes and map them to these layers and then uh you only have to rerun like the change commands or the commands that uh show up after something you've changed uh so i think this should still be the same so the idea is that we're going to steal from this uh we're gonna have like a similar like file format at first we're gonna use something very uh easily machine readable like json later on we might actually get a custom file format like this one uh similar to the dockerfile uh we will not have a from i think maybe we will later but we will mostly have like run and so the file that i was thinking about is going to be like pseudocode uh we're going to have like from okay maybe like a hash here uh this might be optional or from starting with like empty or default and then we'll have like a key value store so we'll be able to do like put key name value this will be stringly typed just strings and strings in the values we'll be able to delete king name and there will be no command at first i think we don't need that uh essentially what we need is just like a sequence of commands and then at the end we will do uh when we actually execute something that was built with this we will just show the whole like all the keys that are known this will be like an upsort actually can pull it that way so yeah just two kinds of commands at first so we'll actually have a file at some point um maybe a json file in the beginning and later something custom as i said um and we're gonna have a command that will allow us to build an image from this so essentially it will be like a compiler but all of this will run at build time at compile time you could say so we'll have one command that will take this and make an image out of this and we'll be able to also run them which will just print the result and at some point we're gonna do you know add a form of caching so that we have a bunch of these well it thinks it's sql now which i guess makes sense uh like we're gonna try to do the same kind of caching mechanism uh mechanism as docker uses well in a simplified way because docker has this problem that it has file dependencies we will not uh our only inputs are going to be the code after some normalization because i wanted this to be the same as this so like whitefist will be a real event and so on we'll have to transform this into a data type and sort of calculate a hash of it might be a hash code i i'm not sure if hashcode is going to be problematic here like a jvm hashcode or we're gonna just use you know something better i guess we'll see it'll also depend on the kind of storage uh system that we'll use for this registry of of layers uh but at first we'll do we'll do everything in memory with no external integrations whatsoever at first actually we're not gonna even like expose any interfaces uh like hp or tcp or whatever we're not gonna have that we're just gonna do deal with the business logic uh so called business because there is no business uh we're gonna deal with that first uh do some testing do some you know focus on the code quality and the design of that and eventually we're gonna add layers on top i think we'll have first like an http server uh i also think docker works like that it's an http server but it doesn't ex expose like a tcp socket but a unix socket uh forgive me if they're if a unix socket is a subtype i'm i'm not an expert on that level either uh i you know i try to discard that's most of what i do uh so so i think docker does that uh so we're first gonna do like a real you know usual hp server on localhost and that's going to be our demon of the of of stevedore and actually you know what let's just call it steve uh i'm gonna rename it uh we're gonna have that and we'll have a cli at some point in the future let me just close this and reopen when i have renamed this it's going to be just steam much better so steve is a clone of docker perfect um yeah so we're gonna have a server for sure uh at first http uh later on we might use a linux alien unix socket uh i hope they work on mac i think they do it because i guess that's how docker works here as well actually not gonna close docker because they no longer need it and it might be taking too many resources uh on my machine okay uh so i'm gonna remove all that we don't really need the files to be there increase the font and yeah uh so i'm just gonna start this this sbt project uh we're gonna do everything from scratch including um like the good setup the git gun r uh i might actually create the repository on like while this happens i guess i will um so i'll start with a built sbt file and this should trigger metals to try to yeah metals is going to start uh some metals again if you're not aware it's a language server for ska it works with vs code and several other tools editors um [Music] we're definitely gonna need a project build properties file this will set the version of sbt i'm going to use 155 and i'm not going to import the build yet because i want to set up some basics if you're just starting with sca i do not recommend that you use this setup you probably just should use intellij and it will just help you in so many things so i'm gonna start we're gonna have a root project this is how you usually usually create it in the directory [Music] um we'll have some settings later on okay we need the scala and t configuration so it was created now i know the latest version is probably three or four and yes we're going to use sk3 at least as long as it's available uh i'm actually going to use like the cutting edge version 310 rc2 because supposedly metals works with it and otherwise you would have a lot of problems with completions okay so we have skeleton tea okay so i decided to steal the config from another project that i had uh this is going to be version 304 of scarf and t and we're gonna set run our dialect to scale three i think that's how you're supposed to do it uh it might complain later so let's see okay i think it did work like the formatting works so we're going to just stick to that so we have spt we have root so i know for a fact that we will have a server and a client or cli so i'm going to create projects for that as well eventually we're going to have like a shared module between them i want to reuse the sources so that the formats that we define like json codex or whatever or the case classes for communication uh we will be able to reuse them between projects uh so i just defined this empty settings but because i know i will be changing that very soon and i'm going to do aggregate on both of these projects thank you github co-pilot for completing that for me and now i can import the build so i think it worked in the meantime i tried to go some background tasks because the performance isn't really the best when i'm recording uh i'm still trying to fix that okay actually i just switched to another version of obs which i think works better hopefully that's gonna work like that so i'm gonna create uh common settings that's a very common pattern in sbt projects uh you see around the world well i have this i have had this at probably every project at work and most of my open source projects as well so i think you'll see this quite often uh we're going to discover in here for now it's going to be the same everywhere so 310 rc2 let's hope this works and also i'm going to open this bt uh again well i'm going to open it by hand just to see like validate my changes more quickly because with metals you have to open the project every time that you want to to make a changes that you make changes uh metals will like start the sbt program so this will start the first jvm and then update the project and actually actually get some some like generate the blue files for example and the process that takes by far the longest in here is like reloading the project and then there's the uh overhead of starting a new jvm so to avoid that by just using the update command to uh to make it go faster oh sorry that's not update that's not what it's doing reload is the one so this is going to run faster than the first time so if you're making a lot of changes and you don't really need them to be available in metals immediately you can use that or you can uh change your metals uh setup so that it uses the spt client command instead but then you have to reload yourself when you make actual changes which is kind of a pain so what i'm going to do here so we have the skull version let's actually prove that it works we're going to work on the server first i don't know why it created a scattered 12 directory so let's try so to prove this to scalar three let's define the command i will delete this immediately we'll have a get and delete well that was supposed to be put uh let's have this apparently it doesn't work source mainscale yeah that doesn't oh i guess i didn't use the command settings so yeah and here's a piece of trivia like normally a settings accepts a variable amount of parameters which are supposed to be settings like uh whatever this returns um as setting yeah but and you would have to use this kind of syntax but it also has an overload that works with sequences or there's an implicit conversion to one of these like this i'm sure we can see it we can okay so yeah so there's an implicit conversion that's quite nice so we said that uh i think we are i'm not sure if we should be doing this for root uh probably will will not have any kind of sources in the root project uh i don't think there should be so i'm going to say publish this i think that's how you're supposed to do it eventually this would have been a pain like if i didn't set publishing uh disabled publishing in this project so this will check if we actually have this color version apparently we do let's try to compile in sbt so metals will still not work because we didn't refresh after adding these common settings to projects but sbt did when i did reload so this reminds me that skat3 actually takes quite a while to compile and i'm not entirely convinced if we that we should be using it maybe we'll be using scale 3 to define just the data types and because enums are so much nicer in scattering i guess we'll see so i'm going to add some plugins to the project in my project slash plugins sbt i'm definitely going to use sbt to polkadot so um let me actually look up the version because i don't remember okay so this is how you add it but i'm not sure if it will work with this rc of scala 3 1. okay apparently it does work we're also gonna get fatal warnings i think uh which i don't think will be a problem but i'm still going to remove it so i don't use uh i don't really like fatal warnings i have explained why in a different video which is not out yet by the time i'm recording this uh but basically i think when all your warnings are compilers you get too much of a slowdown when you try to fix them one by one or fix actual errors you get too much noise and you are not able to see clearly which issues are actual problems at the moment so i i try to avoid that uh so i think we are all set for now we just don't have any libraries uh and i know for a fact that we will need some so i'm going to put some shared libraries into into my common settings we're definitely going to use um cash effect i already mentioned that i will not add uh like hp for us or anything else like that yet uh we will do it later uh for i actually think we might even abstract from communication so that we would be able to set it like in configuration whether we should have an http server or a unix socket i guess we'll see and also i forgot to mention this earlier uh when we do like a unix circuit or hp you know tcp server at some point we might even move away from the http model because i think that's what docker still uses but i want to you know use this as a as an opportunity to like you know optimize and for example just create our own binary codec using scodec uh something more efficient or something i know maybe not scholarly but like avro or something like that uh but for starters we'll just do json over http and that should be good enough and it's still going to be super fast uh like i think even lsp servers send like thousands of messages a minute uh with well it's i think it's json rpc it's still you know json so we're going to get the whole of mass effect and i think the latest version is three two nine but that makes sense we're gonna use that also we're definitely gonna use some config but maybe not yet so i'm just gonna stick to cats for now i'm wondering if i need anything else here we might need cuts mtl so i'll just add it but it will be commented out i'm not to make sure this is the right artifact it might be core whatever so i'll just put it here and let's update the build again in sbt and actually i'm going to check if this is right artifact of kasemtl it is okay so i'll still disable it for now just to have fewer things to worry about and let's import the build into metals again in the meantime uh yeah let's just wait for this we're not going to use spt that much well it's going to be is every time we generate stuff for bloop for metals but most of the time i'm just going to use blue so this is server it still doesn't compile because it hasn't imported so let's look at that some errors in here i don't think that's a problem like there will always be some kind of exceptions in these vlogs um it just takes some time to notice which ones are fine and which ones are not also i guess i guess i forgot we need the testing framework uh so i'll get m unit that's recently been the one that i like most uh so i'm unit cuts effect that's what i think we need the confect 3 variant i said that as well it's taking some time to generate so this is the speed that i get when i use sbt while recording normally it's faster i have to be honest here it's not that slow i remember the times when sbt was like super slow uh to resolve any kind of dependencies back in the day when corsier wasn't a thing uh i don't miss these days when i was starting nesca adventure in like 2016 2015 and using spt as a my first commercial project ah that was a pain that was very slow the compiler was also quite slow back then that was even before 212 so we've gone quite a long way since then like 213.6 is very fast to compile uh scout 3. i've seen it work better before the stable versions sometimes i also heard that you can configure like how long it takes to how long it can give uh your code well how long it's supposed to search for like missing imports you can configure that but i changed it to like zero one time or like to one millisecond and it didn't really help at all so i'm running the import again wait that's the second time so that we get the testing framework but this already compiles uh our skull three code already compiles this guy three which is nice because that's what we wanted uh so we're not gonna have a main file actually i'm just gonna rename this to something else i think we can actually start from the model so let's see if i can yeah the test will come later so i'm going to define the model first maybe that's not how you should be doing stuff like maybe we should be doing the design of the architecture first but i want to have the model you know not set in stone but i want it to be ready before we start doing that so yeah so we're going to have like a build i think that's what we're going to call it and it's actually going to be a case class and also like i have a like a feature set in mind that i want to implement this part of the series but when we're close to the end and we are satisfied with the current result uh i think it's fair to assume that we can extend the functionality to any kind of feature set that we want uh so this can go on forever for all i care if you like this by the way if you're still watching this i appreciate this and i'll let me know like how how long you you've made it if you watch the whole thing i am gonna be really happy if you're gonna watch that um and also i mentioned that i'm not gonna do like any behind the scenes work but this project is going to be open source it's going to be on my github i'm going to have a link in the description uh i cannot put links on videos like arbitrary links yet so you won't see a card here although when i give if you folks are going to watch this i'll get enough few time to become a partner then i guess i'll be able to post links anyway there's going to be a link in this description for the project if you i i think i'll have a couple issues always open uh so that you can actually contribute to this and help me and i will review all the pr's in the beginning of every next video if there are any and you know make sure that nobody misses anything if they don't watch the pr's themselves we'll go through all that in the beginning although i don't expect a lot of that to happen i don't have that many followers or subscribers but if you want to you know be part of this and have your name in these videos and actually help me i really appreciate that i'm just gonna leave some some tickets open later on oh and we're gonna set up skull stewart in the meantime as well uh i might actually do it on this on the video uh set up scott stewart which is a tool that will make pull requests updating our dependencies but i'll make sure that we review these well we at least look at these changes these updates and maybe like review the change logs of the libraries every time we start a new video uh for now i just use the latest scott effect and uh that's all we have and m unit uh i mean it's kind of cut effect and that's bt bulk which doesn't really change much in the in the releases but i guess it's all right um i don't think i have any flags that i'd like to use besides that i guess source future might be one let's actually add it so this was x source three one maybe although i guess we don't need it if we are already on three one i wonder what imports look like here if i can do import scala collection maybe this map or all like that so this works but the underscore shouldn't but it still works okay i'm not gonna complain about that so we're gonna have a build i'm wondering if we should have a name here probably not so this is just gonna be like commands a list of command and a base let's have a base and the base uh of course this is gonna be just a string but i'm wondering if this should be uh let's have an enough for this for now just a placeholder um [Music] it's going to do i'm wondering what the name what the base should be because it can be either a tag like you know we had alpine alpine in the docker files here we would have something like default or empty or hash so i think we could be treating default default and empty as a keyword kind of so then base would be like an option of bass you know of a string of a hash or we could make it an enum so that it's either default or something named i think let's do that um this is going to be fun so build bass all build base uh so object build we're gonna have a num base also about sky 3 syntax like uh the the braces i'm gonna stick to braces just because i think that's the popular choice uh we could be using the white space based syntax and then in in case we have to run back to 213 which i think might happen uh we would just uh reformat the code i think skeleton t is able to change uh whitespace-based syntax like indentation based syntax into braces i don't really want to do that though so because that would be a large diff as well we'll have enough changes that as it is with like enums uh but i hope that won't happen so if i ever get frustrated with how slow scout 3 is or like other issues with it uh we'll go back to 213 i hope not and also the client uh this is just random information about the future of the project the client is going to be compiled to a native image i'm going to try that at least um it'll depend depend on what kind of dependencies we use i guess i hope it will be seamless or quite easy i'll do some research in the background so that we don't waste time here but all the work will be done on these videos uh the point is to have instant startup like like the doppler command it shouldn't start a jvm every time we run it that would be just way too slow uh for a client like this and eventually we might have a front end you know i think uh when we expose the interface for http first uh we will use stop here and we'll also get uh we'll also get swagger uh so that we can actually call the api from the web from ui without writing one so the base of an image will be either empty or well it's empty or a hush and this would be a sha it's kind of hushing will be actually know what this is going to be so that's going to be a different type so like steve dot hash for now this is just going to be value string uh the implementation like the detail of how we hash stuff to a string or maybe like an area of bytes that would be more generic uh like the detail of how we hash i think that we can leave that for layer uh so i don't want to hardcode it here and i guess it just just makes sense um so because you know we'll have to choose a library for hashing and so on maybe it'll already have a type for it we probably don't want an external type in here though uh it's good that i don't have any dependencies here yet in the model because the model can then be a scale 3 module and everything else can be a sky 213 module and still use this but if we had any kind of dependencies would have to use the scout 213 dependencies uh and then kind of like if you had any kind of macros in here that just wouldn't work in the whole project at all so i'm glad we don't have dependencies yet here so we have either an empty image or an image hash maybe let's go like that and then i don't need to do a fully qualified name here and now the commands so i mentioned i want to have put and like absurd key value and i want to have delete of a key and i think that's all maybe just uh we will be printing like the command that we are executing and the result of it uh like what the final state of this key is or maybe what we are transitioning from we will be doing this on during the build so that uh it will be clearer that we are actually you know doing some caching when we actually do caching and maybe we'll introduce some like artificial delays uh to show how fast the cached bills will be uh that sounds like fun so we have the command and everything i think we i can actually introduce the the model or the shared uh sub project so i'm going to do that it's going to be shared uh proceedings something like this and will depend on it here i'm still wondering where to put the uh like the codex when you actually have them i guess that's gonna be a problem because if i want to have this so what i just said about having a scout stream module and dependencies from sky213 it's probably not gonna work if we are gonna use any kind of macros or if we're gonna use codec because codec between versions that's not not gonna work i think it's not gonna work it's gonna be very difficult to deal with it if we try to do this kind of cross version so the moment we get into you know problems with sky 3 we'll just migrate everything to scout 2013 i think and let's also put it here so this aggregate here it shows that when i run tests for the whole project it will also run tests of all these projects as well and the same goes for compilation and so on uh but i mostly care about testing also we'll need to set up ci at some point maybe even now i guess we can do it now that's gonna be fun i still haven't set up the repo so let's get a repository and i'm going to create an empty comment so there it is um and i'm gonna move this to the shirt project when it's created although i guess i can just rename this to shirt and that'll make sense so now this is the shirt sources and uh we'll have to create the well fbt will create the directory for the the server code that's a lot of stuff okay i don't want to check all that in so i will have a good ignore i usually make these myself but you can use there's plenty of tools to generate this for you like extensions for gas code or whatever your ide is um and you can i think download these from like github or some websites by the way i can see that we have a very low frame rate here i'm recording this like everything together using obs um my screen and my face because it would be just too inefficient to record the video from the camera and then convert it to a common uh like a frame rate and merge that it would be just too too much time to render i mean it's doable it's absolutely feasible it just takes a lot of time and i don't want to do that for uh videos that are going to be this one so i think we have almost an hour of content uh so when i see an hour on the current recording i think that's when we'll wrap up so about half an hour left uh yeah so we definitely want to skip all the targets i always do this it might not work for your purpose so i'm not sure why this about that okay that's better so this is what we have in the project and for now that looks fine i also do yes stored i think that's what it's called uh so mac creates this kind of i think it's mac and this kind of files are created uh i guess for thumbnails or something i'm not entirely sure so this is all we want to check in in the first comment and the shared sources are oh i don't want to call it that that's good let's call it model okay and i think that's it so i'm just going to create this repository okay supposedly it's been created let's just make sure it's pushed as well and it is so we have the repository set up i said we're going to do ci i think it's the right time to do it um because why not like as early as you can so we're gonna use let me try this first uh spt github actions as far as we're gonna use that maybe at some point we'll use something else okay so it i guess uh this plugin this codex extension doesn't work with sbt plugins yet uh so here's a hint for lva who probably isn't watching this but i will send it to him let's hope if i remember this let's see if i remember it so this is going to be calm code comment this is linus b works uh organization and i think it's zero 110 now it's time to check oh it's 0 13 000 fine i was close so i'm gonna work with spt again directly um just because we're gonna have to change the file a couple times i think so first of all the sky version needs to be global and it needs to be the same for every uh supported i think so just build conversion um [Music] that actually might be it for the setup like for the basic setup uh we definitely don't want to publish yet because we cannot uh i haven't yet thought about distribution of this i'm not gonna spend too much time on it though because i know nobody will use this this is gonna be used as just a technological uh demo kind of like a demo of the libraries that we're using and the techniques that we're gonna be using uh so i think we're gonna just set up publish local and we're to use uh you know and we're going to use uh crocie to create bootstraps for us um the same way that i've been doing with spotify next which is another one of my projects um i only distributed well i do publish to male central uh but i can very well just publish locally and then use cosia bootstrap from the local repository and it will create a file that that is runnable and runs the program yeah there's a bunch of unused classy options whoa so get projector is apparently built in by default that's that's cool and explain types as well or that might actually not be implemented in sky 3 yet i'm not sure so this is what uh the compiler plugin called splain used to do now built into sky 2013 i'm not sure which version introduces it but i think it's released already and sky 3 either does it on its own like it already does like explain types uh or explain or maybe it just doesn't have this yet uh but i think it does some explanations like that so we have published and then i can you know i can create a booster but i don't have a runable class yet uh like a main class that's all in my project so it's not gonna work okay so we're going to do ci again i've been distracted uh we need to run github workflow generate and let's see what that did so we got a new directory called github.github workflows and we have two jobs and that's the standard so i just want to make sure that we have everything we need so we have some sort of java i would like to use uh java 11 but i don't remember what the exact name is for it but i think it's fine if we just used 1018 for now i think it should be working fine so oh wait it does publishing it does publishing always i wonder what that would look like let's see okay it doesn't work because we don't have a version scheme oh god uh i'm gonna do summer i guess although i don't think we're gonna i'm not entirely sure we're gonna do like any kind of versions uh so i'll definitely create like a tag uh zero zero one at this point maybe and now let's run spt again uh the tag is necessary oh god i'm not even sure if we have a version yet we probably don't because we don't have a good integration so i'm gonna also add another svt plugin uh sbt ci release by olaf but i'm only gonna add it for these plugins not we're not gonna publish the song type or anything yet maybe we will at some point but for now i just want this i want the version to be set because for now version yeah it's like a default version and this should be zero zero one plus something snapshot whatever so when this reloads i'm gonna see what the version is so i know this is a little boring because there's a lot of weight i'm going to try to skip some of the like the longer silences from the video but uh there's still gonna be a lot of boring stuff like setting up as pt if you're not into that uh i think we won't be doing uh a lot of that in the future videos this is probably gonna be the most that you see of it but i guess we'll see uh there's still some work to be done in the build in the future uh like adding new dependencies and so on hopefully that's gonna be faster and maybe we'll migrate to another tool at some point which will make this easier as well although with these modules i'm not entirely sure um yeah i wish i could tell you what i'm talking about but i cannot not yet i'm also going to add skip and publish true i also wonder if this could product like a special meaning in all of tasks now only some tasks supported it so i think we need both of these things uh to actually skip publishing in that project and we need to use different kind of syntax like this i don't know if we need the parents the parentheses i've been learning how to pronounce that because i've done it wrong in the past it's parentheses well yeah okay i think i did it right so i'm gonna try to publish again and i i don't really know what it's going to do whether it's going to try to publish this channel type or or what hopefully not because if it does i'm going to have to set up credentials in the project and that's asking for trouble kind of i don't want to do it yet it's taking a while second it's sweet old time and there are no sources in here to to publish so why sorry directed too many times so it's definitely talking to someone type uh that's not good so i'm gonna disable the publishing in in the whole project for now i will go back to daniel's project i want just skip the publish part okay this goes to an empty seek so this should ensure we don't run the publish job i'll set the like leave the version scheme anyway and make sure that publish local works although we're not gonna use it this year either so now we do github workflow generate also i want to try something if i connect to this using an spd client and now i stop the server does it actually yeah it turns off everything that's not cool so i think i want to use a server here because i've been doing some some changes to spt all the time and then every time i have to exit so when i exit sbtn or svt client when you start sbt like this without the server running now if i go out of this after it's imported loaded the build when i just disconnect and i connect again it's almost instantaneous so now i need to reload which is going to be much faster than the first time first time it took about 28 seconds to just start it and disconnect and now we did it in half the time with some waiting so it's much faster uh so what i was saying uh we got this i don't think we need to import into metals again we didn't change anything related i'm just gonna add this and hope that it runs in in github so this is the repository it's not documented at all yet uh i'm gonna add some stuff in here so by the time you're watching this video there might be content in here at the description or whatever or some tickets to implement probably not this time i think we'll have tickets to work on after the next video because in this one i don't think we're gonna get a lot of code for you to work on and it's dark again nobody told me this so i need to change some settings again that should be fine for for now it's getting darker and my setup for like light lighting is not very good i should also get a better microphone i think uh one that's i guess this this one it was made more more for like podcasts and so on when you don't have to like show your face you can actually hide your whole face behind the microphone and be very close to it i cannot do that because i need to put my head on something uh back to the work yeah this is a lot of distractions okay so we have the model uh i'm wondering what we should do next uh probably to find some interfaces right so this will be in the server i just had an idea that we could support like two modes of building stuff like client side or server side and have the same kind of algebra or you know have the same kind of code and the same kind of interface so that the client will not even know whether it's being done locally or a remote uh but we can always do that later i'm just gonna do it in the server for now so one implementation would be like you know based on whatever the server does like whether it's in memory or talking to a database with all these hashes of images and one of the implementations will be the client side which will use http or tcp or whatever to talk to the server but yeah there's always time later for that so we're gonna have well i guess it's time to define the commands actually uh like not the commands for the the build because these are build commands so i think i will actually move this here into the build so this would be a build.command and we also have commands like in general so that what you can do with this client and this will be either build so like docker build and this will take an optional tag we'll also have run with a required tag or like we should have something like tag like you know so that you can use a tag or a hash um and we'll use it here as well so maybe tag like tag will be uh yeah a type makes sense also like in docker you have these namespaces right so you have like cubicles slash steve uh colon tag or you can just have like a because steve and here we have just the empty image so image hash maybe let's call this a reference we'll have actually you know what i think we'll let's make the hash required for now so the tags yeah let's actually skip the tags we will not have any tags at all in the initial phase uh it's just too much complexity for for now we can always utilize tags later as a kind of alias for for this so let's pretend we don't even know we're going to use that so having either an empty image or in reference that's fine and we'll have to build this out with a hush and we'll run a hush i think that makes sense so we'll just pretend like we are not planning to add tags later the refactoring will be a little more difficult because of this i'm aware but i have to have you know some content to make so this will take me some more time on the on the recording uh no not really but yeah i think it's best if we do that we can think about how we want to do namespaces and so on later uh maybe we will extend what docker allows you to do at least i think it doesn't allow you to do things like you know nesting namespaces in other namespaces for example you would have i don't know you would have a cubicles organization and then a steve namespace and inside that you would have images like model client server whatever like you know uh i'm thinking like more than two levels so you have more nesting allowed uh we can always do that later uh and that's gonna be an interesting thing especially when we try to build like a front end and a way to view all these images under an organization that would be sick but yeah it's it's gonna be very complex let's just stick to the simple thing just working with hashes so it's a command this is what we can run as a client uh at some point i'm gonna move this later to a different file later but we're gonna have like a command eval command command engine let's call it executor very very generic name we can always rename it and it will take a command and return f of output so now the question you need to ask when we have this kind of algebra is whether we want the output to be you know type safe when you make when you run a command like build do you want a build result and that's probably the case you probably want one like wanted this kind of specific type so we probably don't want this kind of type we will have a separate one for every case uh and i'm not i'm not even sure what to gain by having a command uh like this yeah i guess let's do it separately uh so we'll have a build this will take uh a build actually to this and actually think like this is a cli kind of command but this needs to refer to an image somehow so in the cli we would build like a directory but this is not going to be cli related this needs to be generic and needs to contain all the information that you need to build something so we'll have the whole build description here so build steep dot build this will refer to that to to this all the the base and the commands all need needs to be being the command here so we'll have this and this will result in a probably like a stream of events that's probably gonna be the final result but for now for starters we'll just return the hash and the events will be just logged uh but we want to be able to you know uh have a stream of events like what is being built and maybe what is being printed during the build like in docker you would have all these commands that are executed and we would like to in this case see their outputs i don't think docker shows you that but i want to show it here like the state transitions that we do in the in each command each command like like this right i want to show that so these to show them on the on the client will need some sort of view and stream uh but for now just a hash just the final result and we'll log this in the server i think that's that's reasonable yeah or we could do like the client every time we run the client command it will connect to the server and just subscribe to the events sort of in that background thread or fiber but then we'll have to worry about synchronizing like which notifications to send to which client and maintain like a separate connection i think it's better to just build it into this later uh into the single like executor command yeah let's just call it that i can always rename it later so we'll be able to build and then we can run and this will output i think we can output like a stream in here this is going to be easy enough that we could do a stream but it's going to be more difficult to serialize so i think we'll do like a similar thing we'll just log on the back end immediately uh on the server or the daemon but we will return well there's something we have to return or not yeah let's just skip it like return unit i was thinking we should be doing like a stream of strings here but then it also makes the mix everything complex so run doesn't take a run it takes a hash uh so a hush responds to like a program right and it's not really run because we will run this during the build this will basically get the output maybe at some point we can do you know change the file format for the command so that is up for the build the build so that you can also do like a cmd like the final command like what what gets run when you actually execute this image right like in docker that's uh that's the run that's the exact command i think or cmd that's a cmd here uh i don't really remember if there was run or exec or what the difference was anyway this would be like what what gets called you know the entry point whatever what gets called when you run this build image or this build program so bill will be like yeah but for now and this would be essentially what would get printed here and instead what we'll do this will just print the final state after the that hash and if that hash doesn't exist uh we will fail and as for the exceptions like the any kind of errors in here are handling i think every command will have its own error type but it will be kind of implicit in this f effect uh i don't want to do typed errors in here although at some point maybe we'll do either you know builder and hash something like this maybe um or not i don't know it might be just you know fitting in the effect and we'll just catch it uh using like a recover with or something like that from cats so that's basically the interface that we'll have let me add the boring stuff i'm going to regret using this like the using keyword uh if we ever have to go back to schedule 13 but for now i don't care this is going to go to the server so let's make some folders and i guess that's it now i wanna use metals does it work oh it works that's neat uh and i just now yeah i i know just have to move the news into the different module by the way i haven't focused so much in this during this video on my pronunciation i hope you can actually understand everything i'm saying uh i know it might not be that easy we have about 10 minutes left i think until we stop for today uh i wanna define some more stuff still uh but there's so much stuff we could be doing uh i guess we can implement like a fake instance of it and just do [Music] something that only works for the empty file so that yeah i'm actually gonna write the test although i first need an instance so uh we'll have an instance i don't know what parameters it's gonna use uh and it's gonna be an executor of f okay does this work already it doesn't like metals doesn't have completions for whole methods yet but uh thankfully github go pilot can do that so yeah um i thought i had these off by the way like these super references i think i i thought i had to move doesn't really matter so uh in the build uh as i said i'm gonna fantastic shortcut yeah although i wanted this the other way around um oh you can barely see the code oh i need to yeah i need to look at my recording screen so that i don't cover it in my face and this screen this should help hope the font is large enough so uh we want to support the build of an empty image also we need some sort of hash for the empty image i was thinking uh yeah i guess for now we can just come up with some kind of random value or use a you know an empty array or whatever uh for now i'll just use an empty array but this will be an implementation detail here and we will not rely on that anywhere else uh empty hash i'm just going to build hash array sounds good to me um yeah eventually it might be you know like a let's say a shot is 52 characters long i'm not entirely sure but that's the length in mix of the next hashes uh this wouldn't work because it's just an empty array um yeah we probably will have like a fixed size array for for the hashes or maybe this would be a special case i don't know actually let me make a test for that first before we start implementing it so source hold on steve okay i'm just checking if it fits on the screen let's make it larger this is going to be a i mean it effect sweet i guess i think that's how we're supposed to do it uh also have the executor io for now it's io but i think i want at some point actually you know what let's do let's do id i want to i don't want to have too many tests with actual effects so i don't know if it's gonna work like can we do an yeah i think we can still do like a test without io inside the classified suite and still it's going to work so build empty image okay that's almost that's almost right let's open the executor alright let me just check my messages okay nothing's burned yet so when you take the executor and we build it and we build um well we built an empty build so starting with build base empty image and no commands i was wondering if we should make this like a non-mtls because does it make sense to have an empty build but for now we'll allow an empty list i guess that would be a way to create like an alias but i don't think there's a point in this anyway building an empty image should be almost as sustaining as like the moment we noticed that you are using an empty image this should not incur any kind of network calls uh assuming the empty image is like you know the the the server knows what it is um and no commands so this should be like immediately returned even after we're done with this prototype when we are actually in a real implementation it should still be like this this is not dark outside so i just turn on the lights and we can proceed with this uh i think that's gonna be the last thing we do today so this should give us a hash and i'm not gonna really check what the hush is uh build and run so i'm going to test the interaction of these two commands when you build something and you run it you should have some sort of result that is based on whatever you build um i think that that makes sense and eventually maybe we'll have like like an integration test so that when you build any build from the client you should have the same result as if you just built it on the server um i'm sure how useful it's going to be because it's essentially going to be like a just testing the something that's already going to be provided by a different library like probably by top here or if we don't use http it's going to be you know codecs but i guess it will be an interesting test in the future that's that's way ahead of us so that's the hash and we're gonna do exact uh run this hash that doesn't have a result yeah that's that's actually kind of bad so maybe instead of a unit what would we expect to get this is gonna supposed to be like the state of the system like the key value store so maybe we can just get a map of all of that i guess that would make sense um but it's going to be internal to the the server so i will make this by the way are these cases final yeah i'm not sure if they are final in scala 3. maybe at some point they are going to be final by default i hope they will be because in skype that wasn't the case so i always added um otherwise you can sub classes okay stuff which doesn't really make sense uh it scares me so we're gonna have the state um of the system but like tv state i'm kind of worried about this i don't i don't think we should be i don't really know if this should be you know public or part of the executor or like an internal you know type here but then we wouldn't be able to to do anything with it so okay here's an idea this is gonna be a trait and we're gonna describe or like a bit all yeah something like this i think makes sense and this will be our private uh implementation detail private finance class perfect wait that's not allowed what why the nested yeah that was just wait here so uh so this is not exactly the same as the case class uh let's just call it system state you know we can always rename this later i don't want to spend too much time on it and that's going to be the output of running a program and so here you get all should be well that's not that's the wrong wrong framework assert equals this is gonna be an um an empty map that's always going to be the state of the empty image and actually i'm going to special case this as a value in build so when you go to build um we're gonna have it here so when we build and build it empty we want this kind of result let's start running these tests they should be failing immediately um okay it's empty uh i mean it's it's failing uh so let's implement it so as i said this is not gonna do a lot of stuff uh we're only gonna return the hash for now so we're definitely gonna need an applicative here that's for sure oh how imports work i remember the times when you would have to you know type everything in uh in scala 3 and even with metals during the rc phase of 300 i had to i had some fun with that it's much more fun right now so then when we run the hash uh by the way no this cannot work like that we need to ensure that build is the same as the empty build normally you would this is where we would involve you know like the cache mechanism and evaluate every layer i assume there's gonna be like a fold in there or a scan something like that or not unfold or traverse who knows uh there's definitely gonna be a traverse somewhere in this application i can guarantee you that uh but for now i'll just check for equality like is it the same build that we know um and yeah let's check so if build is the same as build empty um if it is we'll do empty this is going to be an option so then we do this to f let's just fail with a running ruble and support it i'll need a bunch of syntax for cats it's going to take me a lot of time to get used to this kind of syntax for imports but well somebody has to do it so this syntax the guard syntax is from the alternative type class by the way we can't do this or not uh we don't have cuts deriving yet like at the renovation library for scal3 and cats so i'm just gonna work with that uh what do you mean you don't have an applicative you do of course you do oh so that's a wrong error this shows up because i think it shows like the first well the wrong uh implicit not found message so when you go to monad or locative error you'll see it doesn't have an implicit unfound annotation what applicative does and it says this so scar 2 would show you i think all of them i i'm not sure if this would be inherited in sky 2. anyway the real problem is that we don't have this and we need an application for throwable so this is called applicative throw in cats it's just a type alias this should compile and not not work yet uh because we haven't recommended run let's do so um so we're gonna do an like the same kind of thing so uh hush empty let's go to empty hash oops and this is going to be this is well actually let's call this get all and make this extend system state just like that as kd state not on a team that should do it and the test should be uh passing except this doesn't work this needs to be empty hash yeah i definitely need to work on the performance of the stream because well this video um the frame rate on the on the on my face isn't really the best and now we are missing some more stuff okay there is no applicative throw for id so the tests don't compile at all uh we're gonna do either and i don't know what the right syntax is right now in sky 3. okay it doesn't what that's not the writer okay so i think that works so this kind of syntax and kind projector i think we're supposed to do like underscore maybe eventually that's supposed to be supported no or is it maybe it should be a question mark i don't remember what the target is yeah let's just use the the star the asterisk that works so this is an either we're gonna flatten up well let's do it for comprehension actually no we're gonna flatten up uh so this flatmap exec run we'll just put this in here and then we'll get him up to just get all and this is supposed to be right not empty okay this is green that's good so we have our basic test setup and apparently it works it runs and we have a our first like interface we also made changes to the names in the base like a build base can now be an image reference not just an image hash we happen to use a hash for now but in the in the future it might be like a a tag as well oops and we defined the commands well they were there already that's just a weird diff we moved the commands inside the build which makes sense i think and we made some cases final we got the empty build value and we have that test so actually let me check if the project if the ci works already because we didn't check that yeah it's green apparently it works i wonder how long it took one minute that's that's quite fast even for scout three i'm just going to commit that because i i trust everything will work well we did run the test so i think uh now that we have this like we have the logic for the empty build like i i can see that it's clearly very far from the end result but it returns something and we could already build the client for it uh i think that's not an unreasonable decision to start writing the client now so defining like a probably a protocol for communication uh so i think in the next episode we will uh like pull in some kind of http library so that we can communicate with these uh systems the the server and the client we'll start writing the client logic so that we can maybe run a scala class that will just send some command as a you know testing uh method a method of testing in this sense method um so we'll start doing work in the client using this shared model and i think we'll also try to share this that'll be a client implementation of executor so we'll have to split this and move this instance on the server instead somewhere else um and we will probably not be defining the cli yet um [Music] like using you know defining the main class that will actually parse the arguments or whatever the commands uh although i can i can already tell you it's going to be uh we're gonna use the decline library it's probably the best one i know for this kind of stuff and yeah so we're gonna focus first on the logic inside the client or at least the communication from the client and reusing this interface and yeah because the client isn't gonna have a lot of logic it's mostly gonna be just decoding the commands from the um the cli parameters at some point we're gonna start reading files um so the files will have to be parsed decoded into a command sorry into a build right that's exactly what the file will be so we will reuse this model properly and that'll be a file so we'll have to decode that from maybe json or in the future a custom crafted format uh still has to be human readable so if we made a custom one we would have to define a parser and eventually the client will also have to well the client will also have to just serialize this kind of stuff in some way and communicate with the server uh i don't think we're gonna do like several discovery yeah it's gonna be really simple well the communication method maybe the protocol will also be like defined in the share module so that we can switch it in one place i don't think that's gonna work although if we did have like a you know switch between is it a unix socket or is it a tcp connection i think we can put that logic into the uh the shared module anyway uh that's it for today so i'll add a readme that's that's gonna be the the only thing that i do off-screen i'll either read me put some information in the repository uh probably no tickets yet uh there's nothing really to implement that i don't want to show live well as i go through it um sorry read me uh i hope this is an interesting format and that somebody actually watches these till the end if you do i appreciate you very much uh let me know leave a comment uh do all this stuff like the usual like and subscribe and yeah let me know what you think about the format like if you have any ideas for this project how to build it you know what you would like me to would you like me to do so that you can learn um how to do it and yeah whatever you you would like to see in the future videos in this series and maybe i should focus more on uh the architecture like this how i come up with this design uh or maybe i should focus more on just coding and leave the you know the design decisions for the background so that i can think about this in the meantime i wanted to show you the whole deal so that whenever i actually think about stuff you can actually see what options i'm considering so uh i hope you appreciate that and that actually helps you um somehow so yeah uh i guess that's it so thanks for watching again and i think i'll publish these weekly maybe bi-weekly so expect another episode in at most two weeks that's all i have for today so thanks again for watching and i see you in the next episode

Channel: Jakub Kozłowski

Views: 1,005

Rating: 5 out of 5

Keywords:

Id: EIE-6gx_qi0

Length: 87min 17sec (5237 seconds)

Published: Sun Sep 19 2021