justforfunc #14: a twitter bot and systemd (that runs free on GCP)

hi I'm Francis employee and this is just a funk welcome to episode number 14 of Joseph funk today we're going to build a Twitter bot and this through the bug is going to have two important things one is it's going to help me personally to retreat all the other tweets that contain the hashtag just a song from the chester song account and also the second thing that is going to do or the second aspect that i care about is the fact that it's going to be free to run and that sounds hard because there's actually not that many places where you can run through the bots for free but i actually figured out how to do it the group up so we will be able to run something all the time for free some fun let's get started cool so as always i'm going to start with an empty directory I called it Twitter bot and I'm going to create my major book now I mean the girl we're going to have as always like a train songs main and now we want to do something using Twitter how do we do that well there's actually great place to find libraries for go which is called go down the road so if we go together the Lord and I search for Twitter we will see that there's a bunch of them but this one on top that looks like has seven hundred stars one hundred twenty eighteen port so you know people actually use it and he provides structured functions for accessing version 101 of the Twitter API perfect that's exactly what we're going to do on top of that Kimmy red color is actually a good friend of mine that's his aditya and i think that if you wrote a library it's probably really good so let's try it going to start with running go get of that even able to do copy/paste because apparently another good at that go get of that cool and now we should be able to use it now how do we use it let's do the commutation so authentication we need to provide a bunch of things so if we run this code okay so this now compiled cool but these will not work let's for now leave it as this so it actually compiles we need a consumer key a consumer secret an access token an access token secret that's a lot of things but these are the things that you need to use to be able to use the github the Twitter API so where do we get those from it's actually another heart and you get them from a period comm let's let okay so if I go to absolute 2000 you'll see that you can create a new account I'm already sign in with my with my Joseph's own account so I'm going to create my account I'm going to call it just for funk why not Twitter bot which is a funk the website is just for funk comm HTTP just for fun calm and the callback URL is not needed because this for when you're doing authentication all over the web we're not doing that so we don't need that and yes I have read the developer agreement and I agree with it that is actually true I actually read most of it cool so now we have now we have an application that has been created we have key and access tokens and we have a consumer key and consumer secret let's define some Bibles here some constants and say consumer key is consumer key which is this one here and so we're going to use consumer key consumer secret is going to be this one here so paste it there and then we have access token and access token secret so we're going to use consumer secret access token and an access token secret cool we still need the values for those two and to do that we actually need to create a token so now we have a token right there the access token is this one and the access token secret is this one cool okay so now we have something that will allow us to create an ApS and how do we use that API well no idea so let's go back to the commentation and okay is that there's a buffer example here it will give us a it will search for tweets containing the word doling and then print off them so let's just copy paste that search result I'm gonna call it r s and press these two it works well okay so if I run this and everything works work correctly then we should see a bunch of tweets with going somewhere so go wrong in there go I could that I'm totally not in a good place so just refined tweet a pot go round may not go cool okay so we have a bunch of things people talking about go in somewhere another hashtag ball go like many places cool what if we say hashtag just for fun will that word yeah cool look at that well it's it's mostly me though oh that's that you should also talk about just for fun please okay so we have a little problem that works but now the problem is that this gets a bunch of tweets and are they not what we want to do is we want to get a continuous stream of tweets every single time that someone tweets anything about just for funk I wanna be notified how do you do that well there's actually a thing called public stream filters so it's a public string you get all the tweets from of Twitter all of them which you know it's bunch of them per second and you feel to them to say only the ones that contain these things we do so we're going to find the public public screen filter that's the function and you pass some values I still don't know what valid what those values are but it's cool API dot public stream filter and it's going to be URL values your values a map of strings to the slice of strings so there's going to be something and then something else here I still know what exactly but that will return a string cool well there's no documentation so that doesn't really help but if we do public stream filter to your API we should be able to see what are the primers or get parameters there's follow a comma-separated list of G's or IDs indicating the users to return status is for in the string also we could say I want to which is only the ones that I say for instance but let's try let's just say that any any tweets at all are good and know no matter who they were who they wrote them but who no matter who wrote them by track ok keywords to track phrases of cue or keywords phrases of keywords are specified by comma separated list tracked the tweeter and the tree and to your anterior okay so so try to the name of the parameter and you just pass a list of things cool okay so normally we should be able to say track hashtag golang and the stream contains stream contains two things two methods one if stuff so we'll stop at the end of this program the other one is is actually not a method is a C which is a channel is a channel of empty interfaces I'm not a huge fan of this API have this means that it doesn't tell me if I should be receiving from this or sending into it which you know of course it's not sending but it could be right like a little bit confusing but whatever okay so we're going to get some stuff it's gonna be empty interface so we don't really know and since I don't really know I'm gonna start by simply printing the type so person capital T will print the type of the value of Island value okay so let's run this oh I probably we're not getting any because people are not tweeting about goaling that's that but at least you can see that these not stopping this just keeps on waiting there so it's actually blocking here in this string in this range of your channel okay so let's change it to something more common as a glove that should work okay so we're getting and I counted a tweet that that's the type of the of the elements will receiving what we could do is we use a type assertion which is basically like a casting if you know by the name or a type conversion but is that conversion like crude fell right so I can say I want to use V as an anaconda dot tweet and that's going to be in menu viable now if for any reason we see something that is not an anaconda treat these will panic so there's another way of doing business which is comma okay now okay will be true only when the value is of type and acronym treat so if it's not okay then I'm going to use lockers and say this a warning f it sort of thing yeah warning F received and expected an expected value of type V and continue okay otherwise what we're going to do is we're going to print the T dot text let's print the text of the tweet and then just continue let's try again cool okay so we can see how people are tweeting about stuff that that's really good what if we move it to just a func now I'm spinning about about just funk right now but that's fine so what do we want to do so what we're going to do now is whenever we receive one of those streets we want to retweet it so that is actually very simple so it's API retweet of the tweet that we want to retreat so it's T dot ID and if we want to trim the user or not I'm going to say no that will return a new tweet because the trees are actually tweets that point to other tweets I have in between it slightly confusing API but basically when you retweet something you're actually creating a new tweet that points to the original one that's the whole idea you're creating three of tweets in an error so if the air is o'neil it means that we were not able to retreat the thing so let's login error could not retweet the street so T dot ID and eighty air and we don't really continue because there's nothing else to be done here actually yeah let's say lock where's info F recruited T dot ID okay so now we have we look for all the just for fun tweets we check if it's a tweet or not if it's not a tweet which is loud a warning because you know that's probably weird thing that's not really a problem really we retreat the tweet then and then if we were not able to retreat it we login error otherwise we we love anything for saying hey this actually works cool so let's run it again okay so now to make sure that this actually works what I'm going to do is I'm going to go to my personal account oh look at that people trying to be in the video if I reply will I show up in the video yes Tom Hudson you will same for you Ted come kill me can they could I don't know how to pronounce your name chat okay but I'm going to tweet from here this is just one more just for funk tweet for my next episode okay so now I tweet this and we got an error we're done an error saying oh so we treated the thing and then we tried to recruit again and it says oh you could not we cannot retreat this because you already retreated it and I feel that the farm is that these are actually both the same so we got the tweet we retreated it that creates a new tweet we received that one and then we try to retweet it but you can repeated something that you already retreated so it fells which makes sense so what we want to do is we want to check whether a tweet is a retweet already and if it is we should not be treated that is very confusing way of saying that we will Google only retreat original tweets and not retreat okay so how do we do that there's actually pretty simple way which is when we get the tweet we know which what are what we are returning and that will be near for the things that are original tweets right so if this is not new then we know that these are retreat so we just continue okay so let's go round Manor go let's try it again so I'm going to D the tweet and say just for fun that way only people that follow just for funk should be able to see this thing just find the next Joseph funk is almost here cool so now we treat this and I just received a notification on my phone we saw that you retreated just once and didn't try to reach it the retreat therefore everything work and everything is working not cracking nice cool so what do we do next we open sources no we do not because I do not want to open source these credentials and by the way if you're planning on trying to use this to hack into my account it is too late because by the time I publish this video these credentials will have been invalidated good try but still no okay so what I want to do is I want to put this I want to make this something that will appear that will be outside of my code and following the 12 factor app - for two-factor authentication the - a factor app I'm going to make this part make these values part of the environment so what I'm going to do is I'm going to replace these with Bibles and I'm going to say that all of these are OS that get environment of something and there are we called Twitter something okay so these two are going to be consumer this one is going to be key and this is going to be secret these two are going to be access token and this one is going to be access token secret cool so now I can hide all of this and for now I'm going to put it in a tweeter dot and then it will not be sharing obviously for obvious reasons and let's say that I will delete all of this as you can see I'm a huge fan of using multi cursive now and it's multi cursors for life okay cool oh okay so now we have all of these variables so if we try to run the program now go on man don't go it doesn't work and it doesn't work because somehow the authentication is not good I know very happy with this because it fails silently but that's maybe topic for a different episode now actually let's fix it so the problem is that anaconda provides a way of logging stuff but you need to give it otherwise it will not log anything which you know as a default is a good thing but for these errors it's kind of I know very happy I think that this should be an error be turned by probably either need to your API should return an error or public stream filter should return an error but we have API dot logger set logger and we're going to pass our logger now what is the logger so I'm going to create log a block gross new so that is a logger and the problem is that we're missing critical method so what we're going to need to do is we're going to need to create an E type type lager which is a struct that contains lagers lager inside and we will then use so there's a lager of sluggers it will pass that one instead we don't need this to be a pointer but whatever so and now we can do so given lager critical will actually do the same thing as love Dodd critical I'm gonna call it error so that is arks interface is a very attic argument and then we're going to use it also Bradley so okay so we have that we have critical F who's going to call air F and this one on top of that has a format which is a string okay that is compiled now cuz we're missing notice notice let's just say and notice is the same thing so notice and notice F is going to go info and influence okay okay cool so now now this seems like it works and to make it a little bit simpler we're going to replace this with log the warning of log that a lot of influence so Lockridge should appear only up here up here and up here cool okay so if we run it again okay now it actually fails because say the three thing authorization is required so what is the front here well the fun is that the environment variables have not been defined so these should not sell suddenly we should fail very loudly so I'm going to replace all of those calls to always get em too close to a different function I'm going to be fine get em the given name most viable returns the value of the Bible so that uses a light socket end of the name but if the value is empty it will panic saying missing required environment variable and the name of the variable which is next otherwise it will actually return B okay so now we want this code it should sell faster it should ok it sells and it says missing require environment viable Twitter consumer key which is cool that's exactly what we want it now what I'm going to do is I need to define all of these environment variables in my terminal to be able to run the code so what I can do i thing is I might be able to do ok so it's just going to do some batch magic here so for line in cat Twitter m to export line done ok so now we run the method we run the program this should actually work go listen to the circuit and if we treat again and people are going to be very tired on me at this point but if we treat again last just for funk treat I swear and just so people done I don't want to know too many people so I'm just going to attack just for fun too okay we retrieve the thing and everything works cool ok so now we have a program that works correctly that gets all the secrets from the environment so which means that I can safely open source this code without being afraid of people hacking into my account but this is running in my machine I do not want to run in my machine because when I close my laptop it will stop working so what I want to do is I want to deploy it to the cloud now it's important to know that I am a very cheap person and I want to run this for free so how do we that let's see that next cool so next let's try to think about where we could run this thing so in Google cloud platform there's four places where I could run this thing Club functions App Engine container engine or compute engine going from like the most the more the most automated thing to the more you do everything by yourself now for App Engine and Club functions you actually react to something and here we know we're not reacting to something it's not like we're getting web hooks from the API we're actually calling the API so that will not make it so we can either use App Engine or plug functions what else well we could use container engine or compute engine and before add juice container engine for the previous episode right the the little sake command was actually using container engine which compete with kubernetes but in this case since this is something that tiny and it is not something like it's going to scale because there's only one job running at any time it feels like it's a little bit too much to start three instances running in a class where plus the master plus everything to be able to run out this so instead we're going to go with content with compute engine now the cool thing is an in compute engine you get one instance for free per month it's a tiny instance it's what we call f1 micro which is doesn't even have a full core of CPU but who cares I don't care if there's going to be a little bit slow it's going to retreat things as they happen right and people are not let's let's be honest people do not tweet about just a song that much anyway so I'm just going to run this code directly on an instance so let's see how to do that so I'm going to start by going to to the Google cloud platform console and go to the VM instances section in the compute engine side and I'm going to create a new instance now when I create a new instance you're going to see that you can choose many things so for instance we can choose the name let's call it Twitter bot and let's choose it on US central one for instance and now here you can say f1 micro and now the cool thing is that it says evacuative hourly rate is zero to zero so zero that six cents per hour which means that it should be four dollars per month which you know it's pretty cheap but also the to hunt the first 720 hours per month are free which means that actually they don't pay anything huge test 130 hours per month into the 123 so maybe you actually need to pay maybe you need to pay six cents I'm not sure I think that this is just like because some moms are longer than the others and that's why in purchase three per month but at worth you know pay six cents you know we're going to be choosing the tiny tiny instance and for the rest we can I believe it is and we're going to create the instance so I want to click create now - is being created and in like a couple of seconds normally they should have here that's one of the things I really like that it is incredibly fast designer instance even these ones that are basically free you could also do this from g-cloud but I decided to do it from here today okay so now I can connect to this now I have an instance that is a Linux instance running Debian Jesse oh there's nothing in it so I need to send somehow my go code there and install the go compiler and compile a bear or even better cross compiled from a machine and just and the the binary already for Linux right so let's do that so if I go back to my code here I should be able to do goers Linux go bill - OH - debug actually that that was a default because the directory is already called to the board but just in case so that's going to generate a Twitter bot so it is ready to run on our computer how do we send it there it's actually really simple g-cloud compute s coffee - France s add to your file and let's copy here Oh what what do we copy scuffie Twitter bot Twitter bot okay so that should automatically manage the keys and send it directly there and it sent it now so we can go back to our instance make key bigger and we have Twitter bot and now it's running but we're missing also depend all the old environment variables right so now we have a job that it's not running that is deployed right like we have it on a cloud instance and it's not running because it's missing environment variables and there's a couple things we could do for this one is define those environment variables another thing would be to just store them in the metadata service but what I'm going to do is actually I want to do something so whenever made printer crashes it will restart it again and after reading a little bit about this and I tweeted about it basically people said that I should be using system D so we're going to be talking about system B real quick so system B what you allows you to do is to have demons or services that are running jobs running in the background like this one you literally interact with them naturally and you can start them labeling stop and then disabling them and whole idea is that if you start one it will be started whenever the machine starts now if I run an compute engine instance it will never be stopped unless something really that happens but even if that happened the instance would be restarted automatically so it very very quick you would have that thing going up again now if you do that whenever it goes back it up back up again you should restart a job in systems you will do exactly that so now we're going to understand is how do we use system D and it's actually not that hard so let's go check the documentations okay so if we check the condition for system B and we see how to create a success an example is exactly we're going to have so I'm going to create a new file and then I call it a tweeter bot service and I'm just going to paste all the content that we have before I'm going to change the description is my just for fun Twitter but after these means what does this job depend on and we're going to be needing the network so this is a way of saying once all of the network things are ready and running then we will start this thing because we're going to be connecting to something over the head put subtitles requirement the wanted by basically lets you know it should be not want wanted by something if nothing once this would be installed the service will not start so that's why you need this at the end at some point for the rest we're going to say type is simple user I'm going to use my user that I had before the working director is going to be the same the xxxx start is the binary so I'm going to call it Twitter bot which is already there with the home front set to the body is exactly where we had this thing right here so home front desk through the bot that's exactly the one we want to run and we start on a board so basically whenever the job stops just restarted directly that's exactly what we want now the cool thing is that there's also one more section which is environment and you can do food equals bug so when you delete your decline a new viable foo with value bar that is accessible only to these service so now we can do to get all of these things get them here and then simply right environment environment equals and with these we should be able to run our job as a service with all the vironment variables that it needs which is pretty sweet okay so let's also copy the Twitter bot service to the instance now we go to the instance we should be able to see it and now the street about that service needs to be in a specific directory for system D to understand world that this is actually a service so we're going to move it to to the bottle service we're going to move it to XE / system t / system and then now we should be able to do sudo system control or system cuttle enable Twitter bot and now it created a link so now it knows about the surface and we can check for things like the status so system CTL status of Twitter bot it is loaded but inactive so we can start it start to the bottom and if we get the status again we'll see that not only it's working but also we can see all the logs so it's already listening to the socket so normally everything now is very correct and everything is working so to make sure that these actually the case let's Street once more everything is ready for the next just for funk including Twitter featuring Twitter but running on system D and celebration time now let's hope that this works otherwise going to be very sad so now I treat this if I go here and I see the there you go if I see the logs I should to the retreated and if I go to my notice patience I should be able to see that these both action retreated which is awesome that's exactly what we wanted now I can disconnect everything skills is still working and if for any reason this machine went down it would restart and systemd would restart the job automatically awesome so I'm going to need this job running for quite a while to see if there's actually any issues if you want to test it everything you need to do is treat something with hashtag just for fun that's that's pretty much it that's as simple as that and you will see if it works and if it doesn't work let me know and I'll go check it out and maybe we'll do an episode on how to monitor these kind of things also I want to do an episode on how to test these kind of things I think it's a very interesting thing too great so with this we got our Twitter bot running for free or for maybe six cents per month I'm pretty sure it's free but I need to check it out on Google cloud platform ring and go and using Twitter API fun and therefore this is the end of the episode thanks so much for watching please subscribe and tweet about just funk thank you all and see you in two weeks [Music]

Channel: justforfunc: Programming in Go

Views: 18,047

Rating: 4.9656653 out of 5

Keywords: justforfunc, google cloud platform, systemd, google cloud, cloud computing, twitter, twitter bots, bots, programming, coding

Id: SQeAKSJH4vw

Length: 36min 21sec (2181 seconds)

Published: Sat Jun 17 2017

Reddit Comments

[u/[deleted]]

Usually I use EnvironmentFile instead of directly putting the variables in the service file. It has two advantages: less clutter in the service file if you have a lot of variables, and you don't need to daemon-reload systemd if you change a variable. Doesn't really matter in this case but as an fyi

Nice video by the way, I'm liking the channel.

[u/mrfrobozz]

Been watching this series on YouTube lately. Really enjoy it and I'm learning a lot. Thank you for making them.

[u/__func__]

Great video! And what about the scheduled tweeting? Does it worth a video?

[u/wiiittttt]

Does anyone know what extension he uses to replace characters in VS Code with unicode symbols?