justforfunc #19: mastering io.Pipes

hi I'm Francis came for him this it's just a part welcome to episode number 19 of just a funk today we will be using the i/o package for grip we're gonna be trying to use three types that maybe you've never heard about before but they're not worried they're actually not that hard to understand in order to solve a problem now the problem itself is pretty cool I don't know if you've ever heard about item 2 it's a terminal available on Mac at least that allows you to do among many other things it allows you to write images directly into the output by incoming them into base64 and putting some header and footer at the end now I wanted to do that as a package in order to do that I need to use the i/o package I want to expose kind of a writer or sometimes like an i/o copy that will copy an image into the output in order to do that though to write some good curve we will need to use a bunch of types that may be the values before those three ones that I tender the coolest are il pipe reader your typewriter and finally IO multimeter if you know perfectly about those three types then you could move directly and skip the introduction I'm going to do to this point which I don't know yet cuz I will have a bottom editing but if you don't know what iota are your pipes are or move to reader then just keep on watching it's gonna be a quick introduction but I think it will help you understand better what these types are able to do and why we can use them you know to solve that problem that's not some fun but get started okay so as always I'm gonna start with a completely empty directory and this would be open sourced and it just found repo github.com slash campos let's just for fun so in this problem what I'm going to do is I'm going to simply say that I'm gonna call IO pipe and IO pipe is a function in the standard library in the i/o package that returns true elements true values the first one is a pipe reader and the second one is a typewriter so I do is I define them as P R and P W now these two values are really cool because what they are is something that allows you to do so you can write into the writer and whatever is written to that writer will be able about to read from the widow so you can do something like let's say f printer land into P W of hello that will write hello into that writer and then I out copy of into ossd out of PR IO copy now this could fail so actually so if that fails let's panic and if that fell let's panic okay so we have now this program creates pipe writes hello into one n and tries to read that message into the from the output from the reader and then copies that into something happen let's try to run that so go and these doesn't work and it doesn't work because we get a deadlock which is actually totally normal because if we go and redo the commentation of i/o pipe it says pipe creates a synchronous in memory pipe it can be used to connect code expecting an i/o reader with a code expecting either your writer the important part here is that it says it's synchronous so in order to be able to write something we need to have someone breathing on the other side does that sound familiar it's exactly how channels work in go to right if you try to send something to a channel and there's no one on the other end trying to receive it it will not work okay so how can we fix this what about we move this part of writing into a new cover tip then okay so we move this part writing into a go in a new girl routine right so now we have a cover team that will write into the writer and then a different origin which is the main go routine that will read right into the output and then finish that's right that now and nope not much better well I mean it says hello which is something it's better than before but we still get deadlock and why is that well if we continue with the same metaphor for channels if the same comparison for channels in channels if you are ranging which is what copies doing it just reading until we get and IO dot EF and the file if we are doing that in with channels ranging until we range until the channel is closed here is the same we will check we will range until the reader is closed but we were not closing that meter so we should close that reader let's say let's close that reader okay so let's try to run that get we're getting closer but still not there because we're closing the widow and what happens when you try to read from a closed from a pipe that has been closed well it fails so what we want to do is not close the reader we don't plus we don't want to close up the reading and we want to close the writing end which will then notice the pipe and it will make it so it will return and gof in the file error when you're reading which will cause IO copied to return without any errors so the whole difference is that instead of closing the reader what you do is you close the writer right you get a pipe you get the writing in and the reading and you're writing to the writing and you close it then this will will be able to read the whole message that you wrote and then realize that the pipe has been closed because it will receive an EO f and the file and IO copy will finish successfully let's try that cool it works nice so that's everything there is to know about IO pipes that's it and i hope i well this one more little thing that we will see later Deb is super cool but for now it's enough to understand the basic concept of pipe what about multi reader let's say that I have a let's create reader from a string so strings that the reader have hoped and then we can do io copy interest on the output of that reader this is what it does is basically just write it reads from that string gets the whole thing and then right into the output pretty simple right now let's imagine that on top of doing that you actually want to to have like a header which is the same thing but it says message let's say it's it's kind of like XML and writer a little correction there I actually meant footer not writer message now in this case it seems kind of simple because we know exactly how long those messages are but imagine that you just given three readers and what you is what you told is well start with the first one then once it done right the second one and then what you done brought the third one and you could do that with we range over our reader Heather reader writer so here what I'm doing is I'm iterating over those three and I'm using i/o copy which is not working Heather because why is this not working oh I think that it's call this body to continue with the same thing it's still not working what's going on oh I think that not executing the good program that might be why yeah I'm not executing good primero okay government okay so get hello mess the message hello and message okay cool so these our way of doing it but actually to be correct you should always check for that error let's say panic here but normally you should return it so you need to do something about that well instead of doing this loop what you could do is to simply say I'm gonna create a multi reader that it's given a bunch of readers Heather party a writer and then we can simply copy into the summer up put that reader and of course check for that error only once so this code that's exactly the same thing as before but this is actually pretty cool because we're basically saying if you have a bunch of readers you can merge them and so you can concatenate them that'd be the correct way of saying okay and what about multi writers well it's pretty much the same but the other way around and we're gonna call io multi right now I am NOT a writer what it does a given a bunch of writers it returns one single writer so what would these be useful for well this is actually very similar to that team leader that we saw on the episode talking about lock pipe so in this case we're gonna do is actually instead of just getting to like just making a tea out of this we're gonna make a little multiplexer will get word because we're not sending it to one or the other but basically we're gonna create a bunch of copies so let's say that we're gonna copy something is under standard output standard error and also to a buffer and that buffer is gonna be bad stuff and that's gonna be our multi rider I can do print the land to the multi rider say hello and then say from buffer so what we're gonna well it's gonna happen here is that when we write hello into the multi rider that is going to be brilliant built into standard output send that error and the buffer so every run now the program it says hello hello and multi buffer for them from the buffer this first solo is into standard output this one is from two standard error or the other way around I'm not very sure but it doesn't really matter and then finally we print the one that we receive in the width that we found in the buffer that's it okay so now that we have a better understanding of what IO pipe and multi be there most writers do let's try to use them in program but before let's write the program without being fancy right like without actually having a package just a program that works I'm gonna show you first that it actually works and then we'll show you how how to have it stopped so I have a program already written it's called image cat and if you pass go for the PNG it shows it right here cool so this is what we're gonna be running first it is pretty simple let's say that we can print as many images as the one so we're gonna say if the length of the arguments is less than 2 then we should say missing paths of images - cat - standard error and then always exit with true otherwise for every path in the argument starting from the from the second one so we're gonna have omit the name of the binary we're going to call a function gonna click cut that off path you if there is no Neil we're going to have print F could not cat this image and this is the reason so path and error okay so now let's write that function cut the given a path string returns an error and for now it's gonna do is we're going to open the file so we're going to open the file OS okay no path look it's a fun error if the error isn't new we're gonna return that error well actually let's use takes Chaney errors package it's pretty cool if you never used it I like it it's pretty much the same but it simplifies the formatting of things so okay if the air is so if there's an error could not open couldn't open image okay not let's make sure we close it at the end then what we need to is create a base64-encoded because we're going to encode actually before I do this so now that we have a file brain and everything how do we make it so so item to understand what we're trying to do it's actually quite simple you need to go through the meditation and the documentation says that these the protocol so you you start with an escape character which is 33 in octo then these characters here which is a kind of a pun I guess it's like Clete it's their specific code to understand that these something for item to then file equals all of the arguments that we won the arguments are in here we're gonna be using only one which is in line we're gonna always set it to one in that will actually print the image show the image inside of the terminal right and just now I'm loading it and showing a notification in then the base64 encoded file contents and then we end up with with this symbol here and this emotion Isabelle character which you can read it as backslash a it's it's a format I don't know exactly what they do with this rather I'm finishing with escape at the end the same thing put whatever we start with escape we finished with bail and then all the things in between so we're going to copy this here so we want to train into the output we want to frame the escape character which we can write as 33 that is an octal character then this then 131 1337 file equals in line 1 then now in mind equal 1 then the column then to base64 so base64 content and then we're gonna finish with a which is the Bell thing at the end cool so for the basic C for content what we're going to do is we're going to create an ear encoder so base64 new encoder we're gonna be using standard encoding and we're gonna write into the standard output directly and that will return a rather closer right clover start so we're going to use IO copy differently to copy the content of the file into that encoder that could fail so if that fell we're gonna return here's the wrap could not write could not encode image otherwise we need to make sure that we close the the right closer that's why to write closer because we need to close it and once you close it what happens is that you flash of the of the buffers inside of that encoder which means that we will actually encode the whole thing otherwise the encoder might be waiting for more content and never get to write the rest of the output so it is important to close the writer clutter so we could do it here and it's actually something that like of errors it's if you wrap something which in this case we're gonna wrap clothes it will return an error there is no nail only if the error double past is no deal so then you can omit that if errors Mandel I'm actually not sure if I like it or not I think it's cool and it's nice to use but it also hides a little bit the fact that there is check in here but okay so we're gonna close it and could not close fix it for in cover is not any error okay so that is our whole thing we write our file we so we we start with the header then we're at the basic C for encoded form of that content then we put the bear character idea so let's try it so if I do go bill that will generate an image CAD right here and if we pass go for the PNG it kind of works but it does something we're at the end I'm not very sure what that is so let's try to debug it for a second oh I think the problem is that at the end we also want to print an offline so this actually understands that that is what we want to do let's try that now that didn't fix it we still getting T T equals equals that looks like the end of the basics for encoding I wonder what we're doing wrong in this to became the basic C for encoder we're running into a standard output oh we're closing too early so okay so that is the problem when we close we buffer we we flush the buffers but we've written this before so actually we cannot do it this way so we're gonna do is so if closing that fails take an ax return the error then we're going to actually print the head the the footer and then return you that needs to be in that order otherwise has its own problems okay let's right now again go bill cool so now I worked we got the basic program working okay so the next step is I want to expose this not as a binary it put as a library as a package that anyone can use in order to do that I'm going to define an IO copy function that it's basically just simply going to do that you create a copy you can copy from a reader whatever it is in this case be the file image and you give it a router or a standard output it will do the whole thing the basics for encoding and all of that trans friendly so it's very easy to use let's do that now okay so what I'm going to do is I'm going to create a new directory here I'm gonna call it image cat which I cannot do is already exists so let's delete that binary I don't really want it I'm gonna create my new binary you might my new directory match cat and instead I'm gonna have image cat dog go and the package image cat it's going to expose a function that scope col' copy and give in and now sorry not that rather in a reader you returns an error similar to the io copy except that I'm not going to return how many bytes were written okay so we have a copy Jameel copy copies the given image reader and encodes it as an I turn to image into the writer okay so what we're gonna do is we're going to replace this whole thing with a image card copy we're gonna ride into the Sun up output from standard wheeler from F and we're going to return that error so now we we basically keeping the binary as simple as possible just open the files and stuff and we just give it where do you want to print it and the file that we want to use okay so now this is not happy Oh kiss yeah actually just return that oh okay so this should work well I mean it doesn't work because it's not implemented right so let's paste and then start work from there so what we want to do is we want to change these four reader does not have to be this and then return nil okay so these still works which is cool let's try it okay so these still working okay so before we continue there's a little bit of a mistake in here which is we're actually printing everything to the output to stand up put rather than the writer that we were given so let's change that by using F printf so that's gonna be that the copy is gonna be done to the right closer but it's gonna be running to the writer and then what else this is going to be written to F themself to the aperture now that we do this we need to check for the errors right because when you're writing to start an output you can ignore the errors really but now we don't know we're Bharani maybe we're encoding this into a connection what if that connection fails so we need to check for the errors return error and we're gonna have to do that here we're gonna have to do it also here F printer here you're here so we have reader here then IO copy here and then okay so now that we have all of this we we see that there's a lot of like error going on and I'm not very happy with this right so what I'd like to do is what if we try to use a multi reader right so could we do it so we have our header which is just these a string so strings the new reader the string the right the footer is this here strings down the reader of this and then want to do something with the rest of the content but then be able to do their I Oh copy into the writer of all the things we got Heather the body and the right now the from here is that what is bad we have not defined it also or it's not defined anymore okay so body is not defined body is all of the content from here now we giving a writer but we want to get a reader hmm how could we do that well I Oh pipes right so we can say I want to use a pipe PRP w io pipe and then rather than writing into the output we're gonna be writing into the writing end of the pipe and then we're gonna be reading from the reading end and that should kind of work except that this of course is a multi reader of all of this this is not happy because writer is not defined off sorry okay so now these this might work what do you think do you think this will work let's try it so if I run this bad luck why well because as before now what we're doing is we're trying to encode into the pipe and then we will be later on but that is not enough we actually need to say these needs to be done both at the same time and now this causes some extra problems because all of a sudden is okay say if you're doing this here the coding how do you return that error hmm not that easy right so what we can do here is to use that extra thing that I mentioned that we had not mentioned before for the i/o pipe which is one extra method which is not closed but closed with error now close with error what it will do is it will close a pipe with an error and that error will be the one return when we try to read from it so rather than closing this we're gonna do is PW close with error of this error and then which are you done with that and similarly we're gonna do here close with error that close so good nothing cut that image couldn't unequal be 64 and now the air is not defined so we need to return that there you go okay so now what we're doing is we're using closed with error in order to pass the air that we got from the reader from the writer sorry so whenever we trying to write at the end so if we are copying in for some reason it fails we get that error and give it to the pipe so when you read from the motor either you will get that error that will be returned by our coffee and they will return by copy so it's a nice way of actually passing the whole error of all the way up this is kind of confusing and order to explain it to myself I actually had to ride a couple of diagrams so it's probably worth taking the time to read these lines of code a couple of times and try to understand where those errors go let's try to run it now okay so it doesn't work at all right we're getting one and does it so we are in deep closing the encoder so the WC is close in here but the promise that we are not closing the pipe itself so we will exit with that closing everything correctly so in order to close the pipe we simply do PW close which just in case we can actually do here good for if we want to close that in any case when I run it not everything works okay so to finish with the episode what I want to do is I want to provide one more API to this image cabaret which is I want to provide our router and the whole idea of this is that when you have a writer you can change you can change it to auto rattles with copy here it's not really that easily done because image copy so image get a copy is a different function to i/o copy so basically what I want to do is I want to do so I could create a writer which is image cut the new writer that will write into the standard output and then I can do the IO copy into the writer and return that error so that doesn't exist yep but this is basically what I want to do it's pretty simple pretty clear right so now let's let's try to do that so if I come here and I define my function me rather given a writer we will return another writer so what we want to do is we want to somehow to copy and we're gonna be copying into the writer but what we're gonna be reading from a reader but that reader is actually the result of so we want to read the content of whatever is ruining - this rattle so we were to connect the reader in a brother how do we do that I hope I've so let's do PR PW it's a new I up ID so we're gonna be reading from the PR for now let's say that if your nail we're just gonna do log fatal we'll pick this later obviously because it's a very bad idea but then we're gonna be turnin PW what is the fun with this oh that actually returns on one error cool okay so let's document it me rather returns a new image card writer okay so would this work what we're doing is we're creating a pipe we giving the writing end to our program that to program calling our library and then we are giving we're getting the receiving the reading n and we're using it to copy into the standard output well into the router that we were given so that looks pretty good but will it work let's give it a try okay so it is not working clearly it's just not printing an image instead just prints one M something what could this be well let's see we are indeed creating a writer and asking to copy but the copy function what it does it creates the reader it then copies from reader to the writer and now we never even writing to these copies so this doesn't make much sense let's see if we actually do this so now works exactly the same way which is not much okay so we're copying we cutting from the pr2 w if that fails we'll look fatal also at the same time we are returning the PW so we're gonna write into W that will be meaning to the reader oh I know okay so the problem is that we never close P W so we never close P W which means that P R will never be closed either which means that copy will not finish so no if nothing will be flushed so the problem is that we're not closing things when should we close it actually we don't know we should close it once we're done with the image so we need to allow the color to do that so instead of a ride of a writer we're gonna return a rag closer now happily P W already is a rag closer so we can then say these are W see because he's riding closer and so if there is one you know we do that otherwise we're gonna return the air of closing that okay let's write again okay better but still not there what is going on now well the problem now is that we are actually closing the pipe but we're not waiting for anything to be flushed so we're returning so when someone closes cause close that pipe will be closed so P R will be closed so at some point I'll copy will finish but we don't know when and we're not waiting for that so we need to make sure that we wait for it how do we wait for things well there's a couple of ways to do it but in this case the best way is to use a channel and this is the typical idiom that we use with the channel called done when the channel is closed all of the people are waiting on that channel will be notified that they can continue right that way also fixes that if we closed down if we called clothes for multiple governments they all should be released at the same time so in order to do that we need to create a new type that will contain that channel so let's create it type rudder it's a stroke that contains the channel done it's empty because we don't need anything and also we'll need the pipe rider because we're gonna be still running into it okay so Ron file here in the image cat pack package okay so what we're gonna do is we're gonna be returning our rider that contains that PW in a new channel of empty structs now this Billiken part isn't saying this is not a writer it doesn't have method right it also doesn't have the method method close so we need those so right given a bunch of pies which ones I'm in turn an error and this one's simply what it's going to do is it's going to call right in the pipe right that's it that's pretty oh it's called data let's call it data too just to have the same name okay and then close and this one is the interesting one what it's going to do is going to close the what should it close it should close the writer so and that returns anything else other than ale we just returned that error and otherwise we need to wait for Dan to be done and once we're done we return yeah now this will wait until the channel is closed so now we know that calling closed we'll wait for something now who's closing that channel no one when should this channel be closed it should be closed as soon as soon as we're done with the copy so here what we're done with copy we should exit so let's do it with the defer so deferred closed oh okay so we need to move this writer to write closer here so they can return it so that way otherwise we cannot refer to the to the top okay so now we know that when we call closed here this will actually block until the copy operation has been done all the flat all the buffers have been flushed including the one for the encoder and then we actually done for real so let's try that nice it works cool there's one more little thing which is what happens if the router we're using is is wrong right like let's try to create a very bad bad writer but brighter is a struct this would be more something that you're writing a text but and then when you try to write into a bad writer it returns zero and flat rather it always return an error returns an error we don't need to name any impaired meters we don't need to name this parameter know the receiver because we're not using them I'd rather not write them so it's very clear that we don't use them so even now instead of writing into the output we what we write into a bad writer which I don't know why I'm passing a pointer because there's no point when we call this it just a slot fatal I don't want it to do log fatal what I want to do is to actually get that error somehow it's ended correctly now that log fado is coming from here here we're here this log fatal will we'll just do log fatal error right and this is not what I want to do instead what I want to do is say one the close operation to return it how do we do that well it's actually incredibly simple and it took me a little bit of reading documentation to understand why this works but what we can do is we can close the PR so the the reading the reading part of the pipe so it's saying I know you're working with the writing side of it I'm gonna I'm gonna close the reading saying these failed in these they are you you should return so it's the simple as this now when we try to write into the pipe or close it on the right again we will get that error instead of any other error so let's try that there you go now it's cool okay go cat go further kenji because of that brother so now we're actually going through the correct error path that I wanted so we go back to this and we move OS TD out I'm gonna try to do one more test which is going to try to run this program with an image and it works perfect yes I turned to also supports gifts and my program will also work with that cuz gives are just basics before I call that too so that's awesome I hope you enjoyed the episode it was harder to prepare there was a lot going on a lot of documentation reading I know that to make this happen thanks to rakia on Twitter JBD Jana for being so nice and helping me with this you review my code and make it made it so much better I hope you enjoyed it find the code on my tools repo on github you can also find the code that I actually showed here um Joseph funk on my repo on github and leave your comments let me know what you think on Twitter as always thanks for watching see you all in two weeks and now I'm gonna go have a beer with Dave Jenny because I'm visiting in Sydney goodbye [Music]

Channel: justforfunc: Programming in Go

Views: 29,385

Keywords: golang, justforfunc, programming, io, multireader, pipes, multiwriter, advanced, coding, screencast

Id: LHZ2CAZE6Gs

Length: 41min 23sec (2483 seconds)

Published: Sun Sep 10 2017

Reddit Comments

[u/AndyNemmity]

Really getting concepts on io.Pipes in a way that I hadn't before, as well as other simpler concepts within it.

Thank you. A lot of the other justforfuncs somehow find a point they go over my head, this one hasn't, and has been the best of the learning for me.

[u/dadleyy]

awesome video, thank you! Those Multi types look really cool. I have found myself using chan io.Readerin the past and I feel like there are use cases for io.Pipe in those places. Are there plans to make those buffered like channels, where maybe the write is not blocked if there is nothing reading yet?

could you have simplified the writer type you created by embedding the *io.PipeWriter inside it?

type writer struct {
  *io.PipeWriter
  done chan stuct{}
}

edit: you'd end up having to override(?) the Close method to send on the done channel anyways so nevermind.

[u/chewxy]

Also, Fransec will be talking about io.Pipes tomorrow at Golang Syd!

[u/forfuncsake]

Yet another great tutorial, with an opening from a great country! :)

The writer/footer correction makes me think of this classic: https://youtu.be/Ub7MkK-a0hU?t=13