Run Postman Collections in Jenkins with Newman

jenkins is the most popular ci server out there and it is no wonder that you may want to automate your postman tests with jenkins in this tutorial i will show you how to do that by learning how to use newman generate reports and integrate everything in a jenkins job if this sounds like what you're looking for let's begin [Music] to get everything up and running inside jenkins there are a few steps we need to take first of all inside postman we need to make sure that we can actually automate this collection run so i'm going to use the collection runner to make sure that we can run this collection in essentially one click after this when i try to find a way to get the data inside a collection and to run it using newman which is a cli tool allowing us to run postman collections and once everything is working locally with newman then we can go to jenkins and get everything up and running there with that being said let's first jump into postman and this is the collection that i'm gonna use for this example but essentially any collection out there should work without any issues this collection has a couple of endpoints and it is essentially building a workflow so you will see here that i'm having a status endpoint and i have here an endpoint getting for example a list of tools and every request also has some basic api tests inside so i'm just case in this case i'm just checking for the status code but essentially every request here has this test i'm getting a single tool then i can create an order and i can get all orders i can update an order and delete an order it's not really so important what this api does but what's important is that it works and that it has this test so just to make sure that we can indeed automate the execution of this api gonna go here to the context menu click run collection and simply start the collection runner so if everything passes here it means that this collection is essentially in a state where we can take it to the next level essentially and start automating it and in order to automate this collection run the next tool that we need is human human is a cli tool it means that we need to open a terminal to run it from there if you don't have new one already installed first of all you need to have node installed and i highly recommend that you always install the long time support lts version of node and this is available for windows macos or linux and you'll find in the video description detailed instructions on how to install each of these tools just in case you don't have them installed locally already so what i'm going to do next is i'm going to simply go ahead and open a terminal i'm going to write here node dash dash version i'm just going to make sure that currently have node installed and that is working so i'm going to see here the version and going back here to the newman documentation just in case i don't have newman installed i will have to run this command npm install g newman this will install newman on my computer and after newman has been installed i can simply run here newman version i'm gonna get the current newman version if you're getting here some warnings don't worry about them as long as you see here newman printing out this version it means that newman has been installed properly and that you can continue with the next step so now we have new one installed locally but how do we get access to the collection that we have here in postman there are essentially three ways on how you can do that you can simply go ahead and right click here on the collection go to export and export this collection as a json file you can put it inside the git repository and store it there it has the disadvantage that you have to do that all the time so anyone working on this will have to import it make a change export it again you will have to import it again so so on so this is one way how you can do it the other way is to simply again go here and share the collection you can go here and select with the json link and what this will do here is generate this link that you can see here you can simply copy the link you can go here write something like newman run and simply paste a link to this collection and now congratulations you have managed to run this collection from newman the problem with this approach is that this link that you see here it will not always contain the latest changes so whenever you're making changes to the collection you have to go back and click update link so for most use cases this is not really the best way on how you can ensure that this is working properly but anyway just doing this initial step is very very important now some of you already at this point will have some errors we'll have some issues running the collection we'll get here some tests that are failing or maybe some requests cannot be executed in probably 90 of the cases the problem is with the variables you have defined either you have an environment and you'll see here i don't have an environment but you could have here an environment which contains some variables and you haven't exported that environment or maybe you have exported the environment but you haven't really specified the initial value so let me show you for this collection i don't have an environment but i'm still using some variables so i'm going to go here on edit go to variables and what you will see sometimes is that the initial value is not being set for that reason when you're getting some unresolved variables and things like that make sure that you set here initial value otherwise it will not work and also if you need to export an environment all you have to do is look here inside the newman documentation and find a respective parameter for specifying an environment for example i'm going to search here for environment and you will see here you can specify an environment with dash dash environment and then specify a file or an address now i was talking about another way of getting to this collection and that other way is called the postman api so essentially whenever you have a collection in your account this is being synchronized with the postman servers and you can use an api to get your collection and in order to use that api you have to know about it and you also need a key so let me show you what i'm talking about this is a public workspace from postman and this is a collection describing how you can use the postman api so essentially um side here collections and i'm gonna try to get a single collection right so just gonna go ahead and copy this address go back here paste it here and you'll see here we need a collection id right so in order to get the collection id simply go to your collection and you'll find here on the right hand side this info button and this is the internal id of the collection this is what you need so i'm going to go ahead and copy this go back to my api request and add this here as a path parameter i'm going to try it again and what i'm getting back is a 401 unauthorized and i'm getting here an authentication error telling me that the api key is invalid now i haven't specified any api key but we will specify an api key going back to the documentation we're gonna open the documentation from here and you will see here that says requires api key and we can send it as a header or as a query parameter so i'm going to set it here as a query parameter these are the query parameters and let's type in something like foo okay so this is still not working now in order to get this api key from postman all you have to do here is go to account settings and from the menu that opens here go to postman api keys and generate an api key and let's give it a name so that we know why we created this tool i'm gonna name it jenkins tutorial and generate it i'm gonna simply copy the value make sure you store this value somewhere then i'm going to go back and unfortunately what i had before disappeared so i need to start again from scratch again i'm gonna copy this let's put this in a variable called pi key and the url will be this one and we can also replace this with the id of the collection i'm gonna go here to info copy this id close it and now i have the collection id so now i can run this request and what i'm getting back here is essentially my collection it's the information that i have inside my collection you can see here all the tests and everything and what i can specify in newman is exactly this address here so i'm going to simply copy it i'm going to run here newman run and now you will see that this collection is failing entirely so the exact same collection has worked previously and now it is not working anymore and probably wondering why is it not working it was working postman like seconds ago and if you're looking closely at what's being sent here you will see here that the address of the api is missing right so we're not sending this request where we should send them and it says here invalid url uri so it says here http and then slash orders so this is not our api so previously i explained about this variable so you see in this case these variables are not being exposed previously in the other link they were exposed but they are not exposed here so i'm going to edit this collection and go here to variables and i'm totally free to specify any other values here so whenever i'm sharing this collection i'm sharing here the initial value the current value is not being shared so what i can do here i can go to each individual value and if i just want to persist the same value i can simply go here and click persist but if there are some secrets maybe i want to use or share the screw secrets or maybe i don't want to share any secrets at all so i could write here something totally different to be shared but in this case i'm going to simply click here on persist all and i'll go ahead and save this and this is a bit the beauty when you're using the postman api we can go back to our request run this again and now it is working so any changes that we make to our collection will be automatically reflected when we run newman and what we're getting here is essentially a quick summary of what has happened how many requests have been executed how many tests have been executed and so on did any test fail in this case everything is working without any issues so super important always think about the kind of variables that you're exposing and if you're unsure like what does your collection contain simply use postman here use postman build your request here and take a look inside your collection if you're searching for let's say my case base url i will see here where this variable appears but i can also see if it has been defined and what is the value where this has been defined right so if you cannot find your secrets or your address or whatever you're looking for so i should look for something like https column forward slash forward slash so i can see it inside here and if i don't see it it doesn't exist so that's the main idea around using this api and exposing these variables so right now you'll see here both variables are empty if i'm running this again you will now see that both variables have a value so this is why previously it didn't work and now it is working so make sure you check that inside postman most of the time when we're running newman we also want to generate some reports and and actually one of the most popular reports inside the postman community is the html reporter from danny dayton and this is the html extra reporter it generates a very very nice report so what i'm gonna do is we're gonna also try to use this reporter now because this is an additional package we need to install it as well so simply gonna run this command and again if you're getting any warnings during the installation process don't worry about it in this case you cannot run newman reporter html extra and get the version this is not how it's going to work but at this point it will still be installed so don't worry about whatever you're seeing here in terms of warnings we'll see in a second how that works going here through the documentation you'll see here that in order to specify this reporter we can do something like minus error or dash dash reporters i generally prefer to use the full name because i know exactly what i'm doing so inside here simply going to run the previous command so i'm running newman against this collection and then i'm specifying reporters dash dash reporters and the previous reporter that we used was a cli reporter so i'm gonna manually add it as well additionally gonna generate here the html extra reporter now when you're specifying the reporters is very important how you write this this is reporters space cli comma html extra it is not reporters equals cli and again after or before the comma there are no spaces so if you add anything that shouldn't be there this will not work properly so let's run this you will see here right in the beginning using html extra on which version you're gonna get to the exact same report as before this is the clr reporter and additionally if you're looking inside the current folder where we have executed this postman has automatically created a folder called newman and inside this folder newman will get the html report so this is how the html reporter will look like we'll get here a summary we can look into individual requests to see what kind of data has been sent so for example get tools i will be able to inspect all the headers any data that we have sent what is the response that came back how were the tests executed and this doesn't only look very nice but it's also a very powerful debugging tool so in case something is not working properly you can look here and see oh am i using the right url what kind of data am i sending are all postman variables getting resolved and so on and so forth super super useful to have this reporter be looking inside here so if i go inside newman you probably have noticed that this reporter has this randomly generated name and we can also decide how we can name this reporter so this is the next step here i'm gonna give this reporter a proper name and as with any commands the best way to understand which parameters are available is by quickly looking over the documentation of the respective tool so you have the new documentation which you can look over and you have also this reported recommendation so for example here we will be able to specify an export path so by using this command i'm gonna copy this just to make sure that it don't make any mistakes i'm gonna simply display the previous command that i had add this additional configuration here and i'm gonna save it inside newman slash report.html so if i'm going inside the folder and i'm looking again at the reports available i will see the previous report and now i will see the report.html the purpose of running um locally is to get familiar with newman and all these reporters that you may want to use to understand how it is working before we move to the next step so if locally it's not working properly there's a huge chance that it will not work properly inside jenkins but at this point we are pretty confident that this is working properly the collection execution is working without any problems we have used the postman api to get access to this collection so now we can look into running this in jenkins jenkins is also available for various operating systems and you can install jenkins locally you can use docker to install it you can install it natively or you can run it completely somewhere else there are so many ways on how you can install jenkins so i'm not gonna get into that but the most important thing is at one point you should be able to see something pretty similar to this i already have a few jobs here but initially more or less after the setup process this is how jenkins will look like so the first step is to create a job so i'm gonna go here and click on your item so a new item will create a job and what we're gonna do here is create a so-called freestyle process this is a bit the older way how we run jobs inside jenkins there's also a declarative way with using a click on ok the next thing that will appear here is this configuration wizard tool how you want to call it and essentially here we're going to specify what this build should do in this case i don't have a git repository but if you have a git repository you can also clone your git repository for example you could store your collection inside your git repository click on git you also have to provide some credentials but this is not what we're gonna do today we're just gonna look into how we can run human so let's go here to the build step and this is where we're gonna add commands to our build so i'm gonna add here a build step this step is very important and you need to make sure you select the right one i am on a unix system so i'm going to use the shell if you're on a windows system you're not going to have a shell probably that's not going to work so i have to execute windows patch command make sure you select the right one for your operating system where jenkins is installed i'm going to execute shell and the first thing that we want to check is that we have all the dependencies that we need so we want to make sure we have node installed we want to make sure that we have newman installed and the easiest way to see if this is working i'm going to write something like node version and i'm gonna also write here human version sometimes windows will not be able to execute both commands so if you don't see them in the logs in the next step you can try them one by one just to make sure that they are working properly so i'm going to click here on save and this job will not start automatically i'm going to click here on build now so this only took one second to execute i can click here on the build number and from the left hand side menu i can click on console output and whenever you're working with this it is very important that you spend some time and read the logs to understand what these scripts are doing you will see here i have executed node version so i got them back this version and i've executed newman dash version and gotten this back as well if this is not working properly check the video description because i've added some troubleshooting information there it's quite common for this not to work on your machine so if you're running jenkins locally on your machine this may not work if jenkins is running somewhere else you need to make sure that these dependencies are installed so this is if this is a different machine from the one you have previously installed newman you have to go again install node install newman and make sure that you're getting this displayed here now this step is working for us without any issues so we can go back to the job itself i'm going to click here again on configure and we can keep these commands here but if you're on windows you may want to remove them because the third command may not get executed so make sure that you're seeing all the commands that you have written here inside the build step so i'm going back to my terminal and i'm simply going through the previous commands that i have executed i'm just going to copy this entire command here i'm going to copy it i'm going inside here essentially all the parameters everything that i had is still available so i have newman run this is the path to the collection i see that some question marks have been added here so i'm not sure why that happened we're going to check in a second if this is still working properly so i'm going to click here on save let's run it and if you just want to check what this job is doing if it's running you can simply click on it on this progress bar but this one only took a few seconds and what you will see here is more or less the same thing you have seen on your own terminal so it's saying here that's running this command so you see the previous commands now it's using newman run to execute the collection you see for each individual endpoint what has happened and finally it's gonna say here finish success so everything worked properly now as you remember you have generated an html report and that is available here inside the workspace so simply click on workspace you will see here newman and this is the report.html that has been generated and you can take a look at the size and you can see that this is not an empty file and you can also click on the report itself and view it if you are using localhost you may get some errors but if you are running jenkins under specific domain that may look a bit better going back here to the build job let's take a look at some other things that we could improve or change now first of all what i don't like is that i have secrets inside my job execution and i don't want to expose any secrets because anything i write here will be put inside the logs and anyone can just open this and read this information so generally it is a very bad idea to put secrets inside here so i'm just gonna remove it all together and go back to the dashboard i'm gonna click here on manage jenkins and under security you should see manage credentials i already have here a credential store which is called jenkins i'm going to click on it these are global credentials i'm going to click on this as well i already have some credentials here but probably for you this is going to be empty from the left hand side i'm going to click on add credentials and going to be asked for like what kind of a credential is this username with password or something else in this case this is simply a secret text the scope will be global this is the secret so this is the value that i previously copied this is the token to the postman api and i can give it an identifier so essentially i'm telling like how i'm gonna identify this so i'm gonna simply call it postman api key right i also i'm gonna copy this name you can also give a description to specify like if there's anything else i should know about this if i may later forget why i created this click here on ok and the next step would be to go back to the job and configure it again and here from the build tab i'm going to simply jump here to the commands that we had and what we can do here is at one point we should be able to define i think this is in the build environment you should be able to inject these credentials it says here use secret text or files i'm gonna check that and what i'm gonna do here is i'm gonna add a secret text and from the list here i need to specify which credential i want to use so this is postman api key and i can simply name here the variable api underscore key so this will be available as an environment variable inside my script so i can simply go ahead here and where it says api key i'm going to remove this and what i'm going to write here is dollar sign i'm going to paste your api key so what jenkins will do at this point when the job starts it will automatically read this credential will create this environment variable this environment variable will be used here so let's go ahead and save it right again and see if it's working properly no errors and what's also important now to realize is if you're looking inside the job execution everything is still working properly so obviously this credential has been injected if i'm looking here you will see api key and this credential is not available in the logs well this is a very important step to ensure that this information doesn't get disclosed so anyone reading these logs will be able to see the api key otherwise in this case this has been masked so no one has access to it at least no one who shouldn't have access to it now there are two more things in terms of integrating this with newman there are two types of reports one of them is html so we have already generated that but there's also a junit report that gu need to report is something that jenkins can read so let me show you how we can configure that i'm gonna go here to the newman documentation i'm gonna search for j unit and you will see this is a reporter this is already available so we can simply add junit as a reporter but we also want to specify a path so what we're going to need here is reporter junit export and we're going to specify a path because we don't want to save it in the default path let's configure the job again so add reporters i'm going to add junit here in the list of paths the space reporter junit export and specify the folder which is human import report dot xml this is going to be in an xml format so dash dash reporter dash junit dash export space newman report.xml let's save it run it again if everything works properly we're going to take a look directly into the workspace so inside the workspace there's the newman folder and what we're gonna get here is the report.xml and we can also view it i'm gonna see it has a size and this is essentially a report with all the tests that we have executed now why have we generated these reports well we could look into the workspace and see them directly from there but at least this xml report this is not very useful for us because if we're gonna read this xml it's not gonna really tell us a lot so this is not something we have generated for us this is something we have generated for jenkins so there is a way to publish this report and this is what it's called in jenkins a post build action so essentially a post build action is something that happens after the build so the build is essentially running the collection and then as a post build action we're doing something this is called report publishing in this case i'm going to select from here the possibility of publishing a junit test report so i'm going to click on it this additional configuration will appear here so what we need to do here is to specify the path to this report so we know that this report is inside newman slash report dot xml now it is very important that you write it exactly as you see it here if you write report somewhere else for example with a capital r this is not the same file that newman has saved so this will not be found and the entire text execution will fail so double check the name it has to be exactly as you have specified in the report exactly as you see it in your workspace i'm going to specify it here and simply go ahead and click on save click on build now and what will happen here is that after this job is executed this report will be published and just after i think two or three executions we should see something here on the project page let me refresh this page what you will see here is essentially that all the tests are passing so i'm going to go inside postman and i'm going to simulate that something is not working properly let me close this we don't need this anymore so i'm going to simulate here i'm going to simply write an endpoint that doesn't exist right so this is something that doesn't exist i'm going to save it this is automatically synchronized and available in jenkins and here in jenkins i'm going to run it again build now and you will see this is failing and you will see here in red we have test that field so we have one test that failed uh eight of them have passed so we can go back let's fix this save it run jenkins again and you will see another one appearing here there's also the possibility of publishing the html report for that we're gonna need a plugin and if you go here inside manage jenkins at one point you should see manage plugins and if you go here to available and write html you'll find here the html publisher you can select it install without restart if we go to the job configure it again and look at the post build actions the one post build action has been published in the junit test report we have another action here which is publishing html reports and the way this is working is pretty similar to to units so i'm not going to go too much into this but the device is still the same make sure that you specify the right path for this one so this is how you can execute your postman collection inside jenkins by defining a job and using human for automation purposes i hope that this tutorial was helpful and if this was the case give this video a thumbs up leave a comment in the section below definitely check the video description because it is filled with very useful resources and essentially everything i'm talking about in this tutorial is also available in the resources hope to see you next time bye

Channel: Valentin Despa

Views: 20,071

Keywords: postman, run postman in jenkins, run newman in jenkins, run newman on windows, run jenkins locally windows, run jenkins job, run postman collection using newman, run postman collection from jenkins, postman collection runner csv file, postman collection runner

Id: iS7HPNswv-8

Length: 35min 34sec (2134 seconds)

Published: Thu Dec 02 2021