Nx After Dark: Finishing implementation and tests for a data access library

hello welcome to nx after dark and right now it is definitely after dark in denmark it's past midnight but it's friday night so we're going to have some more fun we are going to work on the energy insights app angular app again and as a reminder see if i can find it here we're going to be visualizing these the forecast of the co2 emissions in the danish electricity grid and let's see what it looks like right now so all through the night it will be very low um co2 emission levels it's well below 80 kilograms per kilowatt hour so that's great um there will be about 50 so uh wind energy or there was within the past hour it's definitely not the sun i can tell you that so let's get back to the app we have in this branch here we have one app project we have a library project a data access library for co2 and we're working on these two services there's the co2 forecast store which is a component store or it's at least using njrx component store and we're working on this co2 emission prognosis http which is named after this dataset on the energy data service the api we're using which is public but we left some to-do's here we're using we're missing aware clause so we need to filter and to filter we need some parameters because we're working with time series so we should be selecting a slice of the time series like a from and a two part ah so what would that look like let's just go to the store yeah there i created this query filter maybe i can make that more reusable we're going to need it in more than one place right yeah probably where am i using it here down here but i'm not really using it yet okay let's put it here eight filter name and name is heart right h strange free bigquery yeah let's call it the date query we will put it into this file export it and then we're using it here so import it and now we can rename it query okay okay that's good so now we can start to reuse this this data structure so let's oh change to this log file a lot apparently uh yeah i was running pmpm install lock dependencies before i started this stream on this branch so it should be right let's see here extract update query this refactor commit following the conventional commits format for the commits and now here in the this get method on our http client we can have let's see hmm well we actually need well first of all we need the date filter so let's put that in here it's called now uh yeah jquery okay now we can filter and we're filtering on this or sorry this column so it would be this is a postgres sql query so it's very similar to standard ansi sql where minutes 5 utc is after so that would be i'm trying to think here i'm not doing a lot of sql these days uh it's after here we put in our should be probably like so eight time string so that from and then to iso string will give us a iso date time string and the formats we've been using for our test data and and when minutes five utc is less than two yeah this should be this seems right there is one more thing like so we have this price area because we can like we can select a specific one there is the two to select from dk one with which is western denmark and dk2 which is eastern denmark with copenhagen and new zealand um let's just get all of it or what yeah we can we can add that clause later if we need it like maybe we'll display maybe one yeah maybe one chart with two lines in them but i think they will be very similar or actually i think i know no maybe not for this forecast yeah there is some something about the dk1 and dk2 errors in this data set or at least in some of the co2 data sets they are actually the same right now so it's not split into dk1 dk2 the values would be exactly the same so it's rather the whole dk control area so so all of the danish high voltage grid but let's just load both of them and we'll we'll figure it out okay so that's the filter here and now seems like we might have some breaking tests wallaby is telling me down here see where that is oh right so now we have to pass it parameters just put in dummy dates for now good are there more yeah okay here okay now they're all passing again yes so now i'm thinking that we're not really seems like we're not really testing uh the query that gets sent and that that's becoming important now we're rather where we're testing what is returned or what is emitted from from this client but not what it's sending to the server and well we're yeah we're checking the url and that it's a get method but we're not checking the the sql query that gets sent um but yeah like it will it's not really maybe we should split this out into a separate file so that it's easier to test because it's just a tech string but we're going to have to remove the new lines and then it will be a bit more difficult to to test what's being sent there or i mean we could do it here but it's just it's more complicated to to use the htp testing api to do this hmm decisions decisions i will go for easier testability to forecast equal rates your two forecasts equal query we're going to create a function that can generate this thing this query create 0 to 4 cast sql query and it will take a big query query it will return a string this string actually yes and then rather than having it here we will say great co2 forecast sql query bigquery but now we have this file this simple function a pure function which is more easily testable so let's create a test suite for that and we'll put it over here and expand the views like so now we'll do describe the name of this one and get filters uh paid range yes it's an important test so um eight furry so now we'll have a proper date here let's go for right now we're at the 22nd of may and it is yeah we're past midnight so we're at the 22nd in denmark at least and we're at daylight savings time 2 and it will be uh see this represents the beginning of this day this day we end of saturday and we'll be looking two days forward uh but this yeah this represents uh the 24th that would be monday but it's uh as we can see here yeah so we want two days like want to see the saturday and sunday but this is monday it's because it's it's an it's not an inclusive uh end range here there's not an equal sign so this is based on how the time series are set up here this will if we go to but not including the first time series after of monday then we'll get what we want time series we want yeah so see call co2 forecast cooperating update query expect sql to contain um um filters using a lower eight range lower limit mid-range start ah minutes five utc bigger than or equal to and then the string and then take query a string inside of the sql string and then from and that to iso string yeah let's just we can see now that it's passing let's just verify that it it can break we'll take the utc string instead yeah now we can see the output here this is definitely not what we want we want the iso string let me just add a character here then we can see this is the output of that to iso string on a javascript date object so this will this should be good for most databases including postgres it filters filter filters at it that's that way going to start filter yeah that's update range and filter and let's move it to the next one one day into the future so we have separate test data in the test now we'll say it's if we want the records with minutes five utc before the two filter okay so now we have these two tests and uh what's more what's important it's important that they are sorted sorts with oldest records first oldest i mean first in this time series earliest records first yeah great then the bigquery and we'll in this case we'll just pass it dummy dates doesn't matter it's equal to contain and then order by ascending let's make sure that it can fail if the query is not correct again and we'll be showing me why so we expect that order by minus five u to see descending but we're getting all this back which includes order by minutes by utc ascending which is what we actually want so let's change the expectation assertion to what we actually want and now it's passing great great let me drink some water okay what more what else is important here the filter the sword the collection this is the table name for this data set so it because we're we're always using the same end point where did we store that the url here it is we're always using the same endpoint and then a query string will be ended like so but it's the same endpoint for the same endpoint url for all data sets and we might be adding more data sets at some point so it's important that it's selecting from the right table or the right data set it queries the co2 emission prognosis this is data set table name expected table name is co2ms croc for co2 emission prognosis um sequel to contain from and then the name expected table name like so let's make sure it can fail it can good so now we have a passing test so now we have tested for the four most important aspects of this sql query and because we extracted it into a pure function it was really easy to test we're passing a parameter we're checking the output of of that function call uh excuse me it's getting a bit hot here um it's at night so there shouldn't be too much traffic let me let me open the window just give me a second i am back good so what do we have here close all of this close all of this let me see what i've done hmm add baked curry yeah add the query to hp client and well eight range filter to two emission co2 forecast very best there's lots of things getting mixed together here sequel http [Music] stores http clients request responses okay let's have another look now at the http client okay so this one i will delete now i have implemented that remove comments yeah so what are we missing now we're missing to remove the new lines of this sql query because uh what we get returned is it has some additional white space here and it has new lines we want to remove those just to make sure that it doesn't break the query sent to the server because i don't know how it would handle that in a query string parameter oh there's also like since this is a query string it needs to be it's a query parameter so it needs to be html encoded but we're passing it here to the angular http client and passing it as a query string parameter so i trust that angular's http client will take care of the the url encoding of the of this parameter but we should strip the new lines here so let's create a function for that sequel um sequel parameter yeah m sql parameter again i'm putting it in a function here so that is a pure function so it's easy to test part of the class a sequel which is string returning a string now let's get those regex skills out we'll return the sequel but we will replace something with just one just look like so here uh placing that with a single space let's let's look at some what would a query look like let's copy paste this and i usually go for this tool regex101.com now let's pause in this test string this is an example of the query we might get so let's see how we can remove those or match those uh additional spaces and the new lines so new line is black class n well if it's windows machine it might also be x r backlash n but i think this repo is set up for [Music] only unit using the new the unix new line which is backslash n so now we're matching the new line even though we can't see it here and but let's also match the following white space yeah now you can see the first white space being matched here but we want all of them plus what if there was none let's let's use this use a star to make sure that means zero or more white space characters and what if we had some trailing white space here also now we're matching one of them now we're matching at least one but now we're not matching the other lines so we need a star here as well okay so now we can match all this line breaks with with with the space characters so we can replace that with just a single uh space so that would the result of that would be so and so on this is the regex the regular expression we need let's put that in here and the g is for global or something so that means it will match more than one occurrence so it will this replace all there is there is a replace all function or method for strings coming to javascript i think because this has been a bit of a pain point for years that you have to use a regex or some looping mechanism to make multiple replacements in the same string but yeah this is what it looks like with the regex but there will also come a method i think it's called replace all where you can just pass a string as the first argument but so let's add a test for this one at least one test and i need the name of this nice little function print sql parameter it replaces new line with a space character so if we have um if we have raw raw sequel and then look at that did i still store this one yeah use this one again now that's actually the one for this test replace this white space after a new line and following white space ah blue line follow by white space with a space character now sequel prince equal from the sequel from sql to be now let's yeah let's remove this aliasing just to make it a bit shorter and we can remove the filters as well to make it shorter let's do them let's do the manual replacing here this is what it should do one space one space one space this is where it might be important to have that space rather than stripping facing it with empty string it seems to be passing but let's make sure let's add random character here and now it's failing on that word good very good um this is the big test base here um replaces new line a space character let's take a simple test as well hello world from simple parameter text hit text to be hello world with a space not two spaces one space okay but yeah that's that's good enough for now see here wallaby says that we have a 100 test coverage now but again 100 test coverage doesn't mean all cases are tested like this is a regex so you want more more accurate or better indication of like how good are the tests we should add something like mutation testing like striker js really really great tool that would actually modify this regex to make sure that we're covering um more test cases if someone accidentally alters this regex or this one or whatever so that yeah our tests would catch that but that's not for today so now we have this trim sql parameter let's use it here let's use it let's import it but so this is a class and usually you would put put logic like this in methods of a class that's the class way of thinking but yeah i took them out of here just to make them easy to test these two outside of this class because an http client or a service wrapping an hp client is not very easy to test there's a lot of extra setup with the http controllers and test requests and whatnot so now that i've tested these two i feel confident using them inside of this method because they have been tested in isolation good good let's add a commit repeat print sql query string parameter so now i think i'm good with our http client but now we have a lot of files here so what could we do we could group it like say everything that's htp goes here is there more this one and everything that's staked specifically goes here and this this one this one this one they are also for http so they go in here so we still have a lot of files inside of here but at least now they're all they're all related and they're separate from the store and this date query is it's it's um yeah it's a shared interface we might find uses for others places in this application then we can lift it up into shared libraries for now we'll leave it here all renamed good that's good for the git history factor split into separate folders push our work now i'm happy with the http client that it will do what i wanted to do now let's go back and look at this state hmm [Music] saying that yeah there's an angular strict rule here the strict injection parameters is not happy so maybe this one doesn't have a provider oh oh i oh okay i broke some import paths when i moved the files that sucks there is probably an extension for that i mean i used to have an extension for that but i forgot the name so okay i'll go through that files one by one i'll say add all missing imports and check the paths look all right they do oh that was the co2 forecast store but what about yeah the individual ones should only refer to each other and they shouldn't depend on state but what some of them could have this date query reference let's just go through the imports this looks good what is red here probably one of them missing like or having a broken import this one yeah so let's move up the folder the parent folder yes same here same with this one okay that looks good what about the tests hmm let's shut down wallaby started again it's been confused by some files changing this one these changes will be amended okay sheet okay the spec oh down here this is where something is wrong then cannot find http servers oh ah i see i see i see so now we need to pass this query filter let's call it the date query instead curry oh oh i need to properly unpack it because we're actually getting two parameters here in an array the first is the value of this observable the second one is the value of emitted by the timer which will be a number i think it's one two three four and so on so now i'm unpacking here using array destruction or what's it called array destructuring there we go good good test is still failing strange oh okay so there was some compilation error so it didn't show me the paths here that were broken so it would be in http like this at missing imports good just our passing okay yeah also i meant that and now i need to forge push to my repo this branch okay good that was a bit of work actually all of that http client work and now the store so now we're actually also go away angular compatibility compiler um yeah where's this one coming from this is still uh like this is still some dummy dates so right now this would be bad this would be bad because it it wouldn't return anything the http called because of this yeah so this is one thing we can change now we can add a service providing these or an updater maybe but the other thing is we didn't test the timer of this effect but it's way too late for that right now so i'll deal with this one instead uh so two options here we can add an updater for this date query filter and put it in our internal state which is this one but we could also put it in a separate service and we're definitely going to need we're going to need going to need a date library of some kind and i don't know if i'm up for that today yeah i'm i'm i'm thinking like should i put it in the internal state of yeah that might be a good first step i'm probably going to share that state in other places at some point but let's let's do it here for now so hey [Music] and the initial state yeah so we might like we might be able to statically determine or like when it's created determine the current date that will be a good start that will probably be good enough for a lot of of cases so let's do that uh yeah from start with the dummies here the dummy dates now rather than having it here you can take this that the query then we need to select that out and do that in a private observable so that we're not exposing it to consumers of this service state internal state and the date query property and remember to finish notation very important or observables will stop to work no they won't okay so here here we can see how we're actually connecting like the effect to an observable before we had an object here but it's also possible to pass in as we can see then by the name observable or value so before we had a value an object now we have an observable and then that will be the observable that we are passed here so we combine that with a timer that emits every minute and initially to so that either every time the date query changes which it won't right now because it's only ever this initial value and we still have to fix the dummy dates here but every say that changes like if we go past midnight it should change but that that's for another time but say that changes it will trigger this effect say the timer emits it will every minute it will trigger this effect so an http client respon or request using the date query and if we get a proper response back we will store it in our internal state and if we're getting an error we will clear the records in the internal state yes okay so one thing left to do here these dummy dates okay so let's see uh can i do this like without how could i do it without a date library because i don't want to add that right now and not today let's see you know what i'm opening up my oh sir that was not what i wanted to do opening up the trusty browser console let's see what we've got from a new date here our date for now now and then to get to iso string and we have this but it's in utc time i need the danish time is there any way to get that probably not utc date yep so utc is still on the 21st what is the danish the pace is 22nd okay good good good good so it will make an implementation that will let's see oh well it's good enough for now it's good enough for now it will have some bugs but i'll introduce a date library later now oh i'll just do it here oh i need to do it like um turn this into a function great initial state we need a now and we'll return the state okay we'll call that in the constructor create initial state and now has to be the first call we have to pass it now here okay now yes yes yes so now we can try to manipulate here from is let's see it will be now get that full year wait what could we do now set oh this might actually be better today is new date of now so we'll get a copy and today but look it's local today set hours zero can we chain no set minutes zero seconds zero milliseconds zero okay local paid off we're extracting it here so it makes sense what we're doing the dates are mutable and i don't want to mutate at a date object that's passed to me so i can make the copy and do this operation on the copy and then return the copy how will i do two days from now local day after tomorrow is the local date of now value so we take the ticks of now and we add milliseconds in two days that will be two days about 24 hours 24 hours of 60 minutes and 60 minutes with 60 seconds and a thousand milliseconds per second today's ticks mess yeah okay looks like this could work maybe if i'm not way too tired um yeah yeah looks like this is actually something i should extract into a function a function that's easily testable create to forecast bigquery sure we can use it for other forecasts at some point but let's start with the narrow scope so let's start inside of this very folder and we'll lift it up to other folders or other libraries as needed we'll get it and now update and it'll return a page bigquery a query a fun word it will have this local bait off it seems like a useful utility as long as i don't have a real paid library hey if everything you need is nine lines of code why bring in a library right but i'm sure i'm probably introducing a lot of blogs when doing date manipulation myself so i'm going to need that data library at some point [Music] local paid off by why is it not happy oh it's not returning it okay turn from local today to local day after tomorrow yeah but where is it getting this function from how did it there's no import statement something's broken i must add this import statement oh well front-end tooling oh it's not exported that's why yeah so now it has no idea where it is but before when it wasn't exported it was like yeah i can find that i know where that is strange okay s function i could probably test that and we will um he will now co2 forecast bigquery faster now yeah look at that look at that and all the tests are passing again of course we didn't add test for this logic even though it seems like we still have 100 coverage and this goes to show that hundred percent coverage is not covering all like it's not covering all the different cases that we might test you know this is a really this is a code smell right here like having a new date right here because i i would be unable to test that except just actually has some like you can patch the date constructor using a just utility which i don't remember right now but there is something it was introduced in just 26 or 27 see what we're using at least that i think 26 yeah but yeah it's way too late i'm not worrying about that right now but either you would like you would use jest to [Music] like inject the new date um or you would create a date service that can give you the time right now so a new date because then you can replace that date service inside of your tests but yeah new i mean it will create a new date as soon as this uh store is constructed and when is it constructed well that would depend on when it's provided or where it's provided what's the scope of its provider which then follows its life cycle of say a component so that's good enough instead i'll focus on trying to add a test for this one i might also add a test for this one yeah you know if we add a test for this function we're also implicitly testing this function the local data off so let's start with that uh but where do we want to put it let's put it in a bait util folder move it out here and update this import statement that's not what i wanted to test this is what i want to test local data under test hmm it sets the [Music] hours yeah this is a bit tricky this is a bit tricky because if i output the yeah there could be some difference between my local machine and denmark and the ci server wherever on on github actions so i'm a bit worried about that but you should always be when dealing with dates and times uh because it really it shouldn't be measured the app shouldn't be machine specific it shouldn't be times zone aware like that at least not so that the functionality fails in a different time zone hmm so what can i test it clears the hours i'll do a test per property we want to clear because then it will be clear which which one of the parts is failing if any one of them fails clears the hours okay so near now or now is the new date of um [Music] and it's 48 something and it's danish local that's what i'm doing finnish local time daylight savings time so returning local eight it's a big now hours to be zero zero one nope minus one no zero will make it pass good we just have no i want to reuse the fake now uh i'll just add a minute here every time every test it clears the minutes at minutes zero about one no that breaks it okay good and it clears the seconds now i shouldn't have zero at seconds or it would be defeating the purpose so let's have 40 and then 15 seconds seconds so it's not 15 nope it's zero good what else is important oh milliseconds as well um okay milly milliseconds looks like so three milliseconds p123 now should be zero after passing it through our function here and it's getting late now it's 20 minutes to two long after dark sure so what else is important well it's important that it is not modifying it's not mutating the date object we're passing in it does not mutate the past eight times eight instance so fake now pick now um yes matter or does it yeah it's been a proper proper one that we so that we reproduce the same result every time it's run okay so right now it is 42. uh that's not even stored a result let's just see that well now takes belly off will give us the milliseconds since the midnight january 1st 1970 utc the unix epoch okay expect pick now got so if it's untouched it should still be the same good what if we didn't make a copy with this test fail yeah it would exactly what we want exactly what we want good unit test good unit test oh so now i did actually test this one separately but how about so now we know that that will not be a mistake if there's an error inside of here uh but what about this function yeah let's also let's also do that this is the one of the nice things about splitting code into smaller parts it becomes manual manageable to test them there's not too much going on in each one so we can start to reason about them it what does it do it ends today at the start of today as from bigquery or the victory start oh a passing test i wonder why okay so save our fake now is new date okay 105 22 146 a two hour ctc offset actual inquiry is great to forecast now expect actual equator from to equal and just as good at handling my comparing dates using the value of vertex so it should be this good and not one millisecond pass no exactly the beginning the first second of the beginning of the first second of the start of this day yeah good oh this start of today is the query start forecast of the forecast getting better aiming i think i don't think too clearly right now hopefully it's good enough okay the start of the day after tomorrow is the end of the forecast good so let's just use 48 right now and so two days additional two days that's the 24th this start of that day so from 5 to 2. let's make sure it can fail but the other one make sure that can fail as well good no oh i forgot to reset it how does the danish daylight savings time time zone good good good good good good yeah now i'm yeah does it does it run on ci i don't know and what's worse nx cloud will cache it if i run the test locally like i mean wallaby is running the test but that doesn't count it's not included in nx caching but if i run a nx test now it will cache the local result but if the ci server passes that result will be cached and when i try to run it locally i'll get the cache result there is a parameter to skip that or like bust the cache that's called uh dash dash nx dash skip dash cache so nx skip cache with dashes between and two dashes in front of it and on top of that just has its own cache so lots of cash invalidation going on sometimes or necessary sometimes okay so this one is also tested we tested what we're getting in both of these properties very good so [Music] what are we doing here what we're doing here we are that forecast of the curry good good are we getting close i think so i think so so now we see actually that this date query object is an internal piece of the internal state but it's not being it's not something that's admitted to for a public property except that whatever comes out of this records observable will be records based on that query because it goes through the http client passes that date query and then comes back with the records matching that date query we're putting them in the internal state and notifying the services and components interested in in this forecast the records and the date query is now calculated here yeah in the initial state and then we're passing it in through to our effect our low records every minute effect and we are passing it in from this property the date query property because then we can see all the parts of this store that are using the date query state internally and if we didn't want that traceability we could copy it we could inline it down here like so that will also work see the tests are still passing um but let's keep it this way then we can trace the different places inside of this store using this property or subscribing to it at least you can say oh that's used down here for this effect great yeah okay so i'm actually getting to the point that i'm happy now with this forecast that does what we need for now we can't really say if it would need more until we add a a feature on top of it and the only thing i'm not really happy about in case of or in regards to testing is we haven't tested this timer well we might have tested the initial time at the initial trigger somewhere we kind of see that um somewhere here yeah the the initial timer emission is what kicks off this http client um call which then updates the see here that we have to return it something from the http get method then it is the records are emitted or re-emitted through this records dollar observable property skipping the initial mission taking the first emission after that so that we can use that to promise okay so i'm missing a test here so maybe i'll go here and add this to do because not that i won't do it [Music] before merging but now i want to set up the pull request just to make sure it runs every the tests that i have passing locally are also passing in the ci server the github action runner and just review the code changes oh and one more thing one more thing let's export something here that's to do let's now export and what do we need to export do we need the http client no that's used by the store so the store is actually one the only thing we want to export right now let's do that state store good expose expo exposed exposes youtube forecast store so push that change and now let's go create a pr learn this is the branch oh uh in another episode we created we merged a pull request for what was it i think we extracted some special root level libraries from the app project to turn it into a tiny app project so i want this branch to be rebased on top of those changes and we have a ton of merge conflicts oh the joy of configuration compare the changes definitely want all of these root these new root projects so now this is my project that i'm creating in this branch and let's add the other ones here good what is this oh we'll keep the current change here one down this one compare changes yeah copy co2 data access and accept the current change and to sort them alphabetically will be like so one more conflict resolved nx json compare so i will take my project configuration from this branch i will keep their changes here the main branch changes and add this one good last one hopefully here we can say accept both changes and just sort it so uh let me open up again the workspace json and see if the projects are sorted here energy insights app should be after co2 data access i usually add a small script for sorting these so then i can run that as part of the it's part of the yarn or the format script we'll do that another day right now i'm doing it manually so hit rebase continue didn't i wait edit i'll merge conflict and then mark them as result using github but that's what i did oh here okay rebased first push let's see if we can still build seems that wallaby is still running good oh no here we go buckle up angular compatibility compiler good thing i didn't add angle material it has a lot of sub packages that needs to go through the angular compatibility compiler and it's waste of time build okay build work so now let's go and look into adding that pull request add c2 beta access well forecast store and co2 it's called the emission truck noses atp client create well this should still be a draft because i want to add make sure to add that test remaining test lots of commits here checks are being run let's look at them build lint test running in parallel good and i'm worried about the test but there could be some lint errors as well tests are good lent built everything is good great great that's nice okay so just missing that final test of triggering that http request every minute um and i left it to do for that so hopefully i will remember i could also add a comment at test marked two note to self yeah okay okay so soon we will be able at least now we have the data access and layer um hopefully ready to query for data like this let's refresh this page oh a clean green morning in denmark nice well below 80 grams per kilowatt hour of electricity produced and this last hour seems like there has been quite a bit of wind 72 percent wind and solar power and i can tell you it's not solar power so those green data centers are 72 green right now but that's not a hundred percent i'm just saying okay well that was a good session even though i'm very tired by now it's 2 am danish time so i'll be going to bed so i hope you learned something anything that would make me happy let me know and i'll see you some other time bye for now

Channel: Lars Gyrup Brink Nielsen

Views: 93

Keywords: nx, angular, ngrx

Id: r8JrLm9NJUQ

Length: 97min 13sec (5833 seconds)

Published: Sat May 22 2021