PowerShell remoting - Installing and troubleshooting in a... by Anthony Nocentino & Richard Siddaway

you this session if any of you watched soccer we hear the phrase especially English soccer game of two halves this is a session of two halves you can start off with Anthony talking about SSH or emoting and troubleshooting and installing probably not in that order and then I'm gonna take over and look at win RM remoting and troubleshooting any questions keep ask them as we go we'll have questions at the end you can catch us later we'll be around and I'm gonna ship mmm you're on cool thank you sir we got an audio okay so this is gonna be power saw promoting with OpenSSH installation and troubleshooting we're going to do that today so we're gonna start from basically a zero configuration system and get all the way up to 80 integrated authentication together and then we're gonna look at some things I can go wrong in doing that along the way so I'm Anthony no steno consultant trainer and founder of Santino systems where I do system architecture and performance I'd like to make build systems and make them go fast and all that stuff there's my contact info please if you have any questions email me hit me up on Twitter whatever you want it'll be at the end of the deck - during the Q&A phase I blogged pretty frequently and I'm implore aside out there so a lot of the concepts today I have in deeper dive material on Pluralsight so I have some free cards up here for you guys to get access to that content for free and so let's talk about we're gonna talk about how many of you folks were in the session yesterday that I did on OpenSSH okay that's not there give me a little bit of overlap not too much but I there's some things that I had to have today to kind of get some continuity in the conversation that you guys might have seen yesterday so but the context will be different we're gonna start off with installing OpenSSH on windows and linux which is pretty cool right that we have this capability now I'm very excited that that is the case then of course after installation we need to get those users authenticated want to have them on our system configured and want to control who can have access to what I will do that together on Windows and Linux and then we'll layer on top of that PowerShell remoting right because now the underlying substrate we have the option of the underlying substrate to move data with power sorta Modine on OpenSSH which is pretty darn cool we'll talk about troubleshooting some common errors that you'll see throughout and also some common errors that you'll see we'll talk about it throughout the conversation and also specifically at the end and so let's start off with talking about installing open SH on windows and linux but before we do that I do want to talk about the key features of OpenSSH and why it's critical to have that or not critical why it's a good solution to be the underlying substrate for PowerShell remoting because you get a lot of functionality out of the box with next denoted configuration from your OpenSSH server so the main thing is we have the ability to transmit data between two systems securely right well you can encrypt it we can guarantee that it's going to get there and do things like that we have the ability to do remote command execution I can blast a command at a Linux system or a Windows system it'll execute that command and via standard out I'll be able to receive the output of that command on my local system you guys are probably familiar with that concept with what's the PowerShell remoting command the blasted command at invoke command right yeah so we have that functionality as well you guys have a little bit more of a sophisticated output with the whole object oriented thing we have to deal with that as byte stream data I can move data around I can copy files to I can retrieve files from with no additional configuration out of the box functionality of SSH right which is pretty cool and not having to open up shares and firewall ports and all that stuff that you have to do in Windows just kidding and also the ability to tunnel arbitrary TPC TCP services so if I want to have access to a remote system that is running a service that's not exposed to the Internet I can tunnel my traffic get access to that thing as if it was running on my local machine I just send a traffic over and SHS on off to that remote device another big thing with SSH again this is out of the box functionality is that ensure who the remote system is who it says it is right if you guys are using Windows based or Bodeen you're used to the concepts with Kerberos and things like that that guarantee the authenticity of the remote system right well in a cross-platform environment and in a world where we might not trust the remote system of the domain lowercase T not like Active Directory domain but the domain of that other system we have the ability to authenticate that remote system with host keys right and that's gonna be able to authoritative lycée that remote system is who it says it is message integrity is huge right again out-of-the-box functionality with OpenSSH is did I send the remote system then the road system received the command or the piece of data that I sent to it right without having somebody come along and change that piece of data right in between is if I executed command and someone intercepts that and it replays that command that could be bad news right and of course this is going to be the underlying substrate for PowerShell remoting in the future at least as an option right so let's go through kind of the taxonomy of how we would install open SSH on Windows this is a white hot topic right and very rapidly changing max and I were joking about this before the session right because this thing changes very very frequently so I emailed Joey Aiello who's the p.m. for PowerShell core and I believe OpenSSH is under his control as well at the gate kind of the state of the art so this is literally as of last week right and what this current state is if you haven't if you don't know the PowerShell team is managing a fork of the actual OpenSSH code and it's available to you on github right and it's really cool that these guys are cultivating this and at some time in the future that might actually be one single codebase hopefully that's the case if you want to you can go to github read the code I strongly encourage you guys action is having a conversation with Jeff Hicks last night about like the value of going through and reading the code for something as simple as open SSH or even things like telnet because then you can understand how things communicate between servers how commands get through remote servers and then get executor rights a process execution secure transmission it's really cool stuff when you talk about remote access and being able do remote command execution so I strongly encourage you guys to read the code it's in C it's challenging but you know it's cool like suppressing bad C jokes in my head so here's the thing about PowerShell or not PowerShell core OpenSSH on Windows you're going to get your hands dirty right over the years the installation has changed when it first came out you had to like literally use PS exact to run the processes certain user and use SC to register the service and it was ugly now you basically you extract it and you run a PowerShell script right and I'm pretty sure that guy right there Darwin is the guy that's responsible for the simplicity of that sort of say thank you right and that's the value of open-source right does this guy right here can go and contribute this very valuable thing into a literally not let Microsoft product per se but into a product that's consumed by you guys right and going to become a core part of the thing that we use that's power shop so I think that's really cool on the other side of the house it's going to become a Windows feature right it's in beta on Windows 10 and in the fall and fall creators update and the 1709 build of Windows Server right so right now okay oh okay so this yeah so the full story is there right for that was in late December of the release right and you get this functionality be able to bring it so it is just a client is that the case yeah that's just a client all right so it's just a client on for the as a windows feature it will become a feature in Windows Server in 2019 I cheat I've cheated for a long time and I used Windows services for Linux to facilitate me getting access to the client because that's actual I'd always I shouldn't say that because I would undermine the work that the open SST the windows open SS th team did but uh I would use that the window service for services for Linux install bash shell and then run SSH on there as a client but certainly you can use the github project impact code well but the sir is that right okay yeah so these are like the things that need we need to be concerned about when we were talking last night about like dropping that binary in the particular location particular paths and potentially breaking things so that it's very white-hot right from a development standpoint so this is kind of the label I think is the current state of affairs and then also in 2019 it's gonna be a full-fledged feature right guys so I think that's pretty cool right being able to install OpenSSH very easily so now let's talk about the other side of the fence Linux right how many of you guys are running in a cross-platform world Windows Linux in your data center cool right well to install it on Linux it's pretty easy it's already there right so but I do want to call out something actually serious it seemed funny is it depends on distribution and the configuration of that SSH daemon right so a Red Hat has certain conditions in which its configured by default Ubuntu configures it in a certain way I'm a big red hat sent the west person and so out of the box if you do the minimal install on rel or sent the less you get SSH and you get it open to the world with the firewall and you get root log in remotely which eventually is bad news depending on your security profile just know that that's what you're gonna get out of the box functionality on a Linux side of the house I believe Ubuntu disables remote login for route and I believe routes actually has no password by default but we can talk about you offline if you guys seen that meme with the guys from like West Coast Choppers alright the thing was like the five different boxes and they're arguing about open you're arguing about FreeBSD versus and bun to and the last thing the last thing is you know the guy throwing the chair and it's like why don't you get yourself a real OS right so I got a lot of respect for the FreeBSD world what I operate mostly in ascent to West world so it's time to install OpenSSH I have a test lab here I'm it's a remoting session so we're gonna be jumping around a lot between various systems so if you get confused or I do a poor job telling you what system I'm on just raise your hand and be like what are you doing no since you know I have a domain controller haven't worked a Linux management workstation which is where we'll drive most of our demos from I have a Linux server just sitting there and also a Windows box that we're gonna work with so we'll jump over to here and get going hold on me get these updates done do you guys mind I'm kidding I'm kidding I'm kidding sir okay so I just pulled together some quick Power Cell scripts to make this easy and predictable from an installation standpoint normally you just grab it from the URL at this location I use the version that came out about a new version came out about two weeks ago that changed some functionality but I kind of wanted to keep my demos stable and I didn't take that new code and so there is a newer version than that available so you can download it but I'm cheating and I'm keeping it local because I don't want to rely on the Wi-Fi I'm using f8 it's for the demo and oops if anyone knows how to shorten a PowerShell prompt please tell me how so we're gonna take that or I'm gonna install it in this directory by default we're gonna stick it in Program Files when SSH daemon extracts and lungs just for the first time it's actually going to put its information it's keys it's logs and things like that and its configuration files in percent program data percent SSH and we'll look at that a little bit later today so let's go ahead an extract that archive run this beautiful PowerShell script that's gonna do the heavy lifting for us and we'll have OpenSSH installed just like that pretty straightforward this is gonna be on this windows targeted 2016 machine we now have SSH installed we have the SSH daemon and the SSH agent which is gonna allow us to hold things like keys and things like that in a runtime space I'm gonna add a firewall rule there's no new net firewall rule as far as I know in PowerShell core so I am cheating on my demo and using 5.1 so don't tell anybody see don't tell anyone so that's gotten to run that code to open up that port gay firewall ports so when you do this installation it's not running by default so we have to turn it on straightforward stuff let's go ahead and set the services automatic was that the agents automatic will start the service and since I want to know if it's actually running and up and going we use netstat - be ano it'll tell us the protocol The Listening address is 20 - that it's actually in a state listening in the process ID and then obviously the ipv6 address because everybody loves colons so that's it man it's as simple as I have to get OpenSSH on a Windows Server pretty exciting functionality in this cross-platform world that we live in so let's talk about getting users on our server right so now it's installed now we need to talk about authentication right I want to get people on these boxes securely such that I can control who has access to stuff Fran and we're going to talk about authentication method so this can be a common tripping point for folks in configuring and managing SSH systems and so what kind of go through the lineage of what you have available to us so gssapi generic security services API Kerberos but this isn't the Kerberos that you're thinking of this isn't like ad authenticating the users from on the system this is Kerberos you know authenticated by the SSH and game in itself we'll talk about how ad works in a second host based authentication so I could say host a and host B trust each other and without off well with host based authentication they're able to exchange information without usernames and passwords with their keys or things like that public and private key pairs but I can generate a public and private key pair I can have the private key locally the public key locally put the public key on the remote system I can log into that system using that as an authentication method when it comes to public and private key authentication most people don't put a password on their private key which means you get this thing called password this authentication which is great you don't have to have a password but if you put a password on that key you now have two-factor authentication right something you have in something you know and you didn't have to pay a software vendor a million dollars to do that right so with the other thing would probably come private key carrots I made this joke yesterday when I was a PhD student my advisor came from the University of New Hampshire to the University of Mississippi where we did a lot of grid computing work hundreds of servers and all these things like that he had an 11-year old private key with no password right not good news and he used that key on all of the systems right these are hundreds of boxes with this one simple key right so if this thing gets compromised that's bad news so people often will generate one key and distribute that that's not a good idea so take the time generate keys for maybe I don't say for each individual system but in a way that you're kind of controlling the surface area that you expose if your key gets compromised right challenge/response so actual two-factor authentication key fobs and things like that are available to you and then passwords everybody those passwords right one of the things about SSH is it's called password authentication but it's actually a poor name for the functionality that it provides when OpenSSH is going to do pass for authentication what it's actually doing is handing the authentication request off to the underlying OS for it to authenticate the user not it's not actually doing that itself so this is where ad off comes in right we configure the underlying operating environment to participate in ad authentication we configure password authentication the user request comes in over SSH hands it off to the underlying operating environment the operating environment decides is this a local request or is it an ad request and they'll send it to the right place and we're gonna configure this today together in a demo so you guys see how that works the other thing to know is that this list is processed in this order by default right your default configuration in SSH will define this order of authentication so you'll literally see like an authentication failure request for Kerberos fail you'll see the key requested fail you'll go down the path of authentication if you're not using a public key and so that could be interesting you like you might want to think that I'm gonna remove password authentication because all my users using keys well if that's the case and your key breaks then you can't get in right because you don't have another method to get into the system if that is not functional anymore so let's talk about authenticating users and we dabbled with this just a second ago we talked about users being authenticated by the underlying operating environment so we have local user databases right on Windows and Linux yeah Etsy password and the shadow password file that authenticates users locally on Linux systems and you guys have that similar concept in Windows right that local account dated something Sam or something like that we have ad which can be configured on both Windows and Linux for Windows to configure 80 off over SSH you do nothing you join it you install SSH you join a domain and you get 80 off that's pretty cool out-of-the-box but remember the functionality of AD is pretty cool in that we get both user look up via LDAP and we get Kerberos authentication via ad or the functionality that provides for both the user and host authentication right guarantee and again guarantee in it that host is who it is so back in the day before let me tell the story in a second yeah hold on to that one for a second so yeah so we talked about Windows and the Linux out of the house we have to do a little bit of work and it's gotten significantly easier than a half was back in the day when it comes to a tee off for users we have this thing called s SSD or the system security services daemon you install this you join a domain it's one line it's one command you have to execute and you'll have a tee off and what happens is under the hood SSD will determine is the user or local there's user promote a hand to request that hand that authentication request off to the correct authenticating body whether it's the local account database or ad and you do nothing back when I had to do this I had to like walk both ways you know to school up hill in the snow configuring Kerberos configuring held at adding POSIX attributes to the aid to a B such that we have user IDs and group IDs and now it's just it just works and it's awesome so we have this now is a way to I guess you could say identify that this isn't a authentication request that has to get passed off to a domain so you have this kind of funny syntax where it's the user name at the domain name at the hosting right so a en at Labatt's you know systems comm which is my active directory and we have at the host name which is the machine that one authenticate against so you can shorten this and get rid of that with some configuration but out-of-the-box simple stuff I think this is still pretty cool that we have this function out if that bothers you a lot you can also use aliases on your SSH clients to shorten this up such that you can just literally type a single character and log in to your things so let's do some stuff with authenticate we're gonna do some user keys because that's always a common tripping point for folks on a walk through how to generate a key distributed key and things like that we're gonna configure 80 off together and I'm gonna combine them together and see how that works if you guys have any questions feel free to ask like ask why am i using this Windows machine alright because max only have 16 gigs are freaking ramen up melodies right although that's kind of we're gonna spoil it that I'm complaining it I can only get 16 gigs of ram in a certain a machine right okay so we installed PowerShell hey we did that we're gonna generate some keys so I'm gonna generate a key for me key gen and it's gonna generate in my home directory an SSH directory that's gonna put a file the private key ID underscore are saved by default I can call that whatever potato Pub anything I'm not going to give it a password and we'll see that it generates those two files on my assistant idea mask RSA and an ID underscore our state pub pub is the one that goes to the remote system right right now I'm still on my local system sent the last W on the workstation I'm gonna take that key I'm gonna distribute it to the server so that's the sage copy ID is how we're going to do that sir this workstation is not the server will be in a few minutes when we do that together who so SSH copy ID this is part of the standard OpenSSH distribution right one of the most challenging time I say most challenged things but I think there's challenging for people is distributing keys to remote hosts this does the heavy lifting for you from Linux to Linux systems what happens into the hood is this is a bash script that will copy that file through the remote systems set the permissions correctly on a directory that has to land in and put it in the right place and get your authentication configured for you this does not exist on Windows yet feel free I I want to do it go to the github project and maybe make the cross-platform version of this and submit it as a pull request I imagine it would be quite appreciative of that I did demo yesterday and I'll put this on the internet for you guys is effectively what needs to occur I'm not going to do it today but if I need to copy a key to a Windows system this is what needs to occur from a permission standpoint to make everything work on the remote system we're going to talk about the theory about why that's the case but this is the code to make it work on Windows but you want to go through that together I'll be glad to show you guys but I'm not planning to demo that right now obvious I did it yesterday and I will do show you other stuff so let's execute this copy that key to remove host I'm going to accept that host key for the foreign server because I trust that server because I built it I wouldn't trust that server if you guys if I built it but that is going to be what I will use for subsequent connections to authenticate that remote host right every time I log into that system it's gonna give me that key I'm gonna look in the note the list of known hosts on my system if they match yeah I know who that system is alright so right now we have to use password off because we have the auth to that system but the key in the right spot or create the folder put in the right spots at the permissions that's what's happening under the hood so now we can go and all right there's this thing there's a caching daemon on Linux it's a complete pain in the butt and I'll tell you right now I turn it off and kill it that's his age age of minus K if I hit enter right now and I get challenged for a password I'm gonna be so frustrated because this is being works all the time except this morning when I practiced this demo all right cool I got it so what the copy ID did is it created that directory just a few minutes ago on Central Time so you can see it's at 11:21 it creates it with the right permissions right so AENA en that file is actually getting dropped down by us SH right if you're familiar with how permissions work on Linux that would actually be written by root but then it changes it to a yen and sets the permission such that it's only read write and execute what we really want to have happen here is have it be that it's not world readable or group readable we need to secure this thing so that people can't compromise the key potentially that's a facility called strict mode so then which is a default configuration in the SSH daemon you can turn off strict mode but I don't advise that but that's the permission set that's needed that's what that other PowerShell code does on the Windows environment the eye calculus stuff at the bottom of this demo if you guys want a toy with that on your own time so we logged in local user you can see I'm logged in just a en so let's get out so on window is right we set up Windows together just a few minutes ago we installed SSH and now I'm gonna eighty off into that domain joined Windows system right so windows - s1 and literally all you need to do is this it's less dramatic when I can't copy and paste fast right and vs color so yes it's the first time logging in that windows box I'm gonna accept that key it's gonna put it in my local known host file I am now logged into the system with the domain account literally installed SSH joining domain that's it no additional configuration needed so that's gonna be a just a command shell we're gonna do remoting in a minute if yet so time for configuring 80 we're gonna do that right now so these are all the things that you need to configure ad right so we're gonna do sudo yum install realm D Kerberos workstation v odd-job you know the guy had from the Bond movies odd job make home der which are gonna see what the functionality of that is in about two seconds sshd earth s sh t s SSD which is the security services daemon and some Samet and some samba libraries that we need it's cutting copy this paste it down it's a typing very fast not in the shell script and a PowerShell script you can yeah see I'm running shell and not PowerShell yeah I know I I was like thinking about that too when I was practicing the demo so I totally did that against the wrong system hold on so there we go sir so that's all you got sent the west - s1 on the server side we're doing this on yes max I would I don't know if the package names are exact that exact match but one like just go and make sure what the package names if they match up so I cheated I actually installed this earlier because I didn't wanna rely on crop inch conference Wi-Fi so it just says everything was already installed right so when you log into the right server and you install the right software and all you need to do is this pseudo realm join the domain you want to join - capital u the join user so there's to be a user in a domain that has the ability create a computer account that's what's gonna happen under the hood here and - V coz I like output and text in verbose verbose stuff Kerberos is that like an alternative to Kerberos right so it's gonna join the domain and it's gonna do a lot of stuff right like literally I had to walk both hills up away to school and this is done I'm in the domain I have a tee off now that's cool right and what happens under the hood is it does a lot of stuff it creates the Kerberos comm file and the key tabs and does all that stuff for you it configures and as I'm gonna not gonna go through each of the individual config files but if you guys want to go spelunking through these things you can it configures and a switch which is the thing that's going to say am I gonna authenticate you against a local databases or the remote database ad right and then SSD which is the thing that actually does the routing and I have the ability to do this I'm gonna use the command ID I miss I da yen at lab dot Santino systems comm and I'm gonna ask the domain for the POSIX user attributes for this particular user right so you can see I get back my user ID and my username my default group ID domain users and any of the groups that I'm a member of right so now like yesterday in the demo is I can figure out sage to restrict authentication into the SSH server to that particular group right so combining that together we're disabled in route logging you've already increased your security profile of your SSH significantly because you're controlling it who can get in and disabled in root access right with an adgroup that's simply an AV group nothing special about so let's go ahead and try it logging in best one so I'm on Sun to SW 180 off into the remote system fingers crossed and I'm logged in it and so we can see that odd jobs job is to create the home directory for this particular user because that didn't exist in the local system when we because we didn't actually have I configured the user at all right he just works so it creates that directory and we're logged into the domain pretty cool stuff pretty simple to do with a couple of commands so let's go ahead and back out of here and combine key based off with AD off right now I want you guys to think about what's happening with the user being authentic it or the user being validated and the request actually being authenticated so we'll go through that together so copy ID how about I just copy that so that's taking the key that we generated for this user on my local system and sticking it in the ad off user that we can just we just set up so if I do SSH - V if you're having any trouble with authentication on SSH SSH - feet and you're gonna get this show you right now mental note make shorter domain names for demos so - V gets you verbose output all right and so you can see I'm already logged in no password because I didn't put a password on the key but I do want to call out is what's happening under the hood when I do SSH - feet right I connect to the server connection established scroll down skip over that we authenticate the system right doing the host key authentication against what's key is found in a nose host file it presents to me it's key I compare it with the keys in my local host my local known host file at that location number one there's only one key in there and it's the first of all just two keys now because of the windows box but that's gonna be the you could say the in numeric index into which key that is so if you do have an error and it's like keys 37 is wrong you can literally go down to line 37 delete that key and get a new one next time you log in but question why did the key change don't just accept that automatically right and so so after the the host is authenticated we move into the user authentication page so like I said you're gonna see Kerberos right justice API go first and it's gonna fail and we're gonna move into passing on our public key right and it's gonna compare that to the local copy of our private key and it's gonna hopefully allow me in so we can see we're authenticated via public key authentication so two different facilities who asked the question so I've actually never configured justice API in a production system I've only done it with a B in Kerberos even years ago I was using a B in Kerberos because setting it up out of the box is a complete pain in a button with default configuration sir it is now right yes because of Kerberos as effectively as the real reason right because it's gonna it's because with Kerberos not only does it authenticate the user but authenticates the two systems participating in the authentication right so if we go and we look at so this guy is gonna participate in the conversation as a computer right and that's gonna be authentic keeping the system in domain all right so I wanted to show you copy the key and I totally just got discombobulated it had authorized keys combined key off oh yeah so here we go oh yeah so I got ahead of myself in the demo that's why I'm confused so let me go ahead and remove the sage authorized keys and back out I just I wanted to show you guys what happens with AD right the authentication you know it passes Kerberos for the system specific Kerberos passes the public key and it becomes it is a password off because like I said it's handing it off the underlying operating environment to authenticate the user so that's the point they're back out of that give me that back so we need that for subsequent demos okay how am i doing on time Richard good thank you sir okay so setting up power so remoting on windows and linux I can do what so I did this video with Jefferson over and Jason how like about a year and a half ago almost two years and the way that this goes down is in like June or July of 2016 Don John sends me an email it's like hey I need a Linux guy I'm like cool that's me and he's like I need you to do something I'm like what he's like I can't tell I can't tell you I'm like oh ok and so sounds great I'll get involved in that because Don said so right and everybody knows what that means when Don says so so he's like you're gonna work at this guy named Jason Helmick I'm like who's Jason Helmick Google Google oh cool all right so I get on a conference call with Jason and Jason's like you're gonna do this thing with me and his guy Jefferson over I'm like who's Jefferson over cool oh ok that guy and and so like that's pretty cool you know and so he's like but we need you to like have this conversation with us and shoot this six-hour training video and talk up be having a meaningful conversation about PowerShell and before that video I was like the copy-and-paste PowerShell guy right I go to the Google and I'll be like oh there I need that copy paste edit edit run right and so I had to crawl in a hole and learn Power Cell in six weeks that I can sit at the table and shoot a six hour training video with those two guys right which was a little intimidating I had to explain to my wife and I get I'm like in Cancun I'm like PowerShell yeah on vacation but long story short is during that demo or during that shoot in between two scenes Jeffrey and I are having a conversation about this whole concept like fan-out remoting being able to execute a command on many systems with one single line of code right like invoke command dash computer name and a big bucket of computer names and it goes and does that and he's explaining to me the plumbing behind how that works with asynchronous job posting and all that stuff and I'm like dude that's freaking cool and I paused and I high-fived I'm not like that's awesome right and so since then I've taken PowerShell really seriously and it's become a big part of how I manage systems because simply because of that particularly with fan-out remoting and before this conversation the night before I've never even heard of DSC right and the night before Jason and I are at the bar at the Marriott down the street and we're going to these showing me how DST works and the concepts and things like that on my oh that's cool so if you watch this video you'll actually see me on the side taking notes while they're talking about DSC right because there's stuff that I want to apply to situations that I have to deal with even on Linux systems so which is pretty cool to to get that so a lot of value for me as the Linux got to learn like what's going on and the things that you guys have been using for years right so a very exciting stuff so let's talk about how we're gonna bring PowerShell remoting together with the underlying stuff that we just all want to write we just built up to the point where we have the ability to run SSH on Windows systems and of course on Linux systems right so we're going to focus on server side configuration we have the ability to do client-side configuration things like aliases and stuff like that I covered yesterday but we're gonna focus on service at configuration and we have to be able to bring together PowerShell the program and sshd on reliance substrate or transport layer and on linux systems that configuration is going to live in Etsy SSH SSH Dee underscore configure there is an SSH underscore config file that is for client settings we're gonna focus predominantly on SSH D which is the server saturdayman settings right on Windows systems that file lives in % program data % right or talked about little bit ago when we start up the SSH daemon for the first time it's going to put its logs and its information inside that directory sir this is where we are right now like today if I went and did this I don't know the case for if I use the github project this is what's gonna happen I know Joey talked about the potential changes and things like that so yeah it's entirely possible this moves to some other location at some time in the future but if you do this tonight you're gonna get that so that lives there in sshd underscore config on Windows systems and yes search directory logs get written to there now which changed which I'll talk about in a second and your host Keys live here as well sir you mean the drop-down config files yeah so this is like the big difference between obviously a big difference but this is a fundamental difference in Windows admins and Linux admins right not that undermining you know what you say but that's no big deal right for me to copy a file to a target system that's like totally standard practice would be to drop a line into that remotely with like cetera that's totally standard food for us right but um so I'm happy to see that when you guys like why isn't it in the registry right that kind of stuff so that's just not how we roll yeah so um logging logging logging I forget I've been getting a little bit confusing yesterday's session and today's session in the latest version of power shell or the OpenSSH project that just came out last week the one that I'd not upgrade to you upgrade to conventionally your logs will land in a text file in this directory right they just changed it so that the stuff defaults into etw which lands in your event log which is convenient for you guys right but I'd rather have in a text file no offense so PowerShell core in subsystems this is how we're gonna bring this stuff together right we're gonna go and configure powershell core such that it can talk to OpenSSH actually really the other way around but what else in that sshd configuration file you have the key word subsystem you name the subsystem something in this case it's PowerShell and then you point it to a binary right so in this case it's gonna be CD program files PowerShell yadda yadda yadda - all the way to PW Ice Age XE those parameters on the right are actually PowerShell parameters and that then is now a configured subsystem such that when I come along and you do something like enter PS session what's gonna happen in the SSH request that's built underneath is I'm gonna drop in the fact that I want to switch the channel to a subsystem right which is then gonna spawn a powershell process and a remote system gonna connect the standard streams of the two processes together over ssh and that's literally how data go is to-and-fro right have you nervous that like you're raising your hand again alright so pretty easy stuff like standard streams is a normal paradigm for a way therefore unix's systems to do it to process communication this just happens that we're plugging the two together over and staged connection right which is really cool from the Power Cells standpoint because now I don't have to deal with the plumbing anymore right that's all built-in stuff in the SSH right and that your security people won't have a cow when you're like let's turn on win RM let's not let's you just a sage right and they're probably gonna be cool with that so there this is the windows world if I needed to go and configure PowerShell core + ssh to get remoting over ssh but it you have to deal with this per platform right and so who notices the big difference between those two lines text right one's got slashes to the left the other has slashes to the right for whatever reason I should have the developers right here yesterday I should ask them is in the config file slashes to the right even though it's a Windows path right and you guys aren't used to this kind of this content with slashes to the left so if you copy and paste this and you put it in that file you're gonna be sad because it's not gonna work right do it make sure you put the slashes in what I would call with the correct it direction if you're on a Mac PW SH lives there if you're on a Linux system Kingdom SH lives there so just simply substitute that out the parameters are still the same because those are PowerShell parameters that get passed into the PowerShell X cube all the us our bin is generally a symlink to us our bin don't be confused it's literally the same path on your system it's just that conventionally over the years there's been a divergence between slash bin and us our bin so let's talk about let's do that build subsystems and stuff we are going to go to the windows box log in jump over to here third demo and do this see I took advantage f8 there hey we're gonna add a subsystem copy and paste right because I don't want type throwing down on it's gonna be brutal one thing that I've noticed if you've noticed this is built in here right that can be troublesome over time if we change out versions of PowerShell or a feature right because then you can control exactly which version you're using depending on how you view that glass is half-empty glass is half-full right so if we go down a little bit oh sorry for the font size business I didn't think about that font if you see this this is a default configuration right it specifies that but it's commented out but that's the default value so that you can go and you can see what the default configuration parameters are for SSH if it's not commented out then it's gonna be explicitly defined so if we scroll down a little bit here there's another subsystem that lives there and we're just gonna there's no real reason to put that there you can literally stick this anywhere other than the fact that it's just nice to have it in the same place right so SFTP is another subsystem we're gonna define that subsystem we're gonna save this out we're gonna bump our SSH daemon to read that config and we are gonna jump over to back to the workstation and test out our remoting capabilities I am in PowerShell cornel 6.1 and I'm going to try to connect to that system anyway in there so you're gonna see this the first time you do it probably it's gonna try to use when RM right out of the box because the positional parameter for entropy s session is - computer name right if we don't put anything in there I think so we have to tell powers our enthalpy a session that we want to use SSH right under the hood so let's go and do that and then we get logged in so now this connection purely running over SSH which i think is pretty darn cool there is another parameter in the universe of entropy s ession l - hostname which by default will just work because now it knows to use SSH as the underlying transport for that now let's do 80th right again no configuration to make this happen other than what we've done so far joining a domain and putting SSH on there so I'm back on the workstation and I'm about to connect to that Windows - s1 by simply defining the username and this is all facilitated by OpenSSH are under the hood and PowerShell remoting on top and I get exactly what I want on the remote system now before someone asked what shell that I get now I get PowerShell alright this is a PS and we are going to do this now we're gonna connect to enter PS session we're gonna connect to the hostname sent to West - that's 1 and give it our ad auth easier an eightieth user because we just configured ad authentication together and I get an error why is that the SSH client has ended the error message substances on the request failed on channel 0 ready because I haven't gone into the sshd configuration on the remote box and said let's use a subsystem Frank because what happens is SS agency tries to do it they call invoke the power shell subsystem and it cannot so let's make that config change together oops I wanted that and when login regular SSH edit this config file oops sorry sudo yeah see good see I didn't just log in as root you know you want to maintain like good security practices this bums me out ctrl F like it doesn't work the way that would expect on because ctrl F is page down on non-windows keyboards but I have a page down key for the first time in 20 years it's kind of nice alright gonna drop this line in right here we're just adding a subsystem to bin PWS hm power saw core is already installed all that heavy lifting has occurred I got I got right that our systems yeah or too much coffee let's be serious restart sshd there was sudo on the front bang out we'll go to control D logs out I don't ever use that bash shortcut that's awesome back out up arrow up arrow enter and I get it right so now we run PowerShell remoting I can do all this cool stuff that what is it PS you know all that good stuff just to prove to you guys and so that's that this is the powershell process is gonna be dangling off of ssh that's going to interpret all of our commands on for us right it's the job of the subsystems that kick this off on the remote box troubleshooting OpenSSH let's get into that or why doesn't this really work all right there's one way that you're gonna troubleshoot this right did you're gonna make sure that asset stage works right just take remoting out of the picture make sure that you can log into that system get a shell and then bring remote it into the picture and troubleshoot whatever your issue is we've already used the - V parameter use that learn that authentication model right host is first with all the various authentication models that you're using for user off and things like that you have the ability to do service ID bug this is actually extremely helpful when things get dicey but you have to have control with the remote system do you do this right so we get to go into our sshd config we can modify the debug level we can ramp it up to debug 3 which is going to be a vomitorium of text and you're gonna get in sshd it's gonna up until that previous version would spit it out into the lin to program data % program data % in the log file in there and SH directory or in Farah log messages no far log SSH on Linux systems and it will literally tell you what's wrong for example when I was prepping the demos I had strict mode is enabled I'm like why isn't like key based authentication working for that windows box right and all you get on the client side even with like 17 V's after on a client side when you're launch that staged client it's just going to tell you that your key fail to authentication it's gonna ask you for a password I cranked up debug 3 restart at the SSH daemon on the windows box it's like oh this security identifier has read access to this particular directory in your thing because of strict modes your auth failed it's that explicit so go in there and look at that and it'll tell you what's wrong if things are getting dicey there's some good docs there on the methods on how to do that so you'll have to rely me just blabbing about it you can actually go and read that stuff to get that going live oh here's that line that I was trying fine currently lives in the log files but moved to etw in the latest Rev and then they talk about that there so there's good commentary on that so common problems that you'll have outside of like how we would go troubleshoot you this is host key mismatches a big one right something changed and I get it presented with a new key and your funkier your authentication will just fail because that's gonna happen first that's the very first thing that occurs is that host key exchange so if that change to figure out why before you go and accept that new key and then figure out if you want you accept that new key this is always a big problem it was my problem on the windows side of the house because I just didn't quite understand what I needed that happened when it came to setting that up and I'll use that technique to figure it out subsystem we did that right it's pretty explicit I think that's a newer thing because when I first started dabbling with this you didn't get such a nice error that was like you didn't configure the subsystem before it would just I think just give you nothing which is the bad news yeah and the powers our client will yell at you and you try to do things without a subsystem so I'm not gonna be in 403 I'm gonna sit in the corner over there until Richard's done but we can hang out have some questions after I'm gonna hand it off to him in a second there's my contact info again and I have Pluralsight courses free access up here so if you want to have like go into deeper detail about the different features and functionality of OpenSSH and how to do that that's all available to you guys there so cool any questions what's that I can we put that on the website we can - you can put the PowerPoint there I have to get I'll zip the code up and put I have intended to but put the code up as well all right we'll go from left to right here go so the there's a keyword in sshd config that's called allow groups right and all I literally it's just easy allow groups then it's gonna be the group at the domain name so we'll just say like my user is a member of SSH users when I did the ID command simply put that there because remember you're asking the underlying operating system to do that work for you so all it's really gonna happen under the hood is UNIX and Linux systems don't have a content like security identifier with like the big long alphanumeric level bloody you guys use we just simply turn them into integers so I'm gonna go and then resolve that group ID and it's just going to be a number and if you're in that group then you get in and so that's all happening by the underlying operating environment allow rupes it should I gotta tell you I haven't I haven't tried it but I can't see why it wouldn't for that same concept of just a value like resolving it to a you up and if you are in that group not we can do that afterwards we'll try it out sir yes by default out of the box of this yeah that's one of the things I struggled with configuring that demo with remembered I didn't demo it today but I have the code in there the iCal cool stuff that was a total pain yeah so yes I'm not quite understanding exact time okay yeah okay so you're like in a local in a system where I have a local right and they're gonna yes if they're sick theoretically yes if the group if the number matches you'll get access to the resource cuz all UNIX cares about is if the two numbers match ray it doesn't matter what they could so I'm not a ad ninja but it's my understanding that when in newer versions that ad schema post exaggerated boots are included so that user ID and group ID yeah it's going to be a value that's a for that particular user in your ad so I would if you as long as the you if you have the ability to change that user I would probably change it in AD before I changes it in cool one more because I'm totally in the red zone in time sir yes right um yes so you know of course let's do this and you can't read that it's in there there's like it's literally like key transport or something like that trust me because I'm up here there is uh you fall that so it's all you sir okay my name is Richard siddaway director of partial dog I'm responsible for the parts of the organization of the summit if there's anything that you liked about to submit please come and tell me if there's anything you don't like please going to tell dawn because that was his fault been partial MVP for ten year ten years I've been working PowerShell since be one how many people were working with v1 PowerShell yeah all we had for remoting was get to bring my object yeah that's fun and then Along Came remoting and I'm gonna cover in this session those two girls one is to think a little bit about troubleshooting techniques and in specifically talking about troubleshooting PowerShell remoting it's enabled by default so a lot of the install problems go away before that it was just enabled PS remoting and servers have got it already network connections can be a pain and you've got to enable it on the client OSS and the old service and it usually just works if you're in the domain but how many have you been troubleshooting remoting ever you love the error messages aren't they wonderful in terms of travel troubleshooting methodologies over the years I found it tend to fall into two groups the guy this'll work you will be pressured into do this because your manager will be coming what's wrong what's wrong fix it fix it fix it usually it's quicker if you've got a methodical set of tests that you can work through and you fix the problem so when I was thinking about this talk because originally it was just gonna be troubleshooting remoting but then I thought would put a bit of methodology around it so how can I get a methodology around troubleshooting if you've dug into the commandlets that come there is some troubleshooting commandlets but they came in Windows 7 I think and they'd not changed then and they're not really extensible so I'd a few beers and I thought about it and I had a few more beers and I thought about it and I had a few more beers and then eventually I remembered that we got pestered so I've set up some tests and pestered to use as a troubleshooting think so I'm gonna show you how that works and put it forward as a troubleshooting technique that you may or may not want to adopt few obvious tests when you test in remoting is it your machine can you connect to another machine can you actually get across the network is double yes man there so what I'm going to do is show pesto based methodology it doesn't have every possible testing because we'd be here till Christmas I'm just showing some options and some methods just to get you thinking about it and we've been looking at things like the impact of taking down the winner I'm service what happens if you break the listeners and ports endpoints firewalls so I'm basically going to break remoting and then show you how to troubleshoot it the one thing I'm not really going to cover apart from this slide is PowerShell direct how many of you use that how many of you know what it is okay partial direct is for Windows 10 and Server 2016 hyper-v hosts and clients and your remoting over the vm bus rather than w s-- man or ssh so in the session commandlets and invoke commands you've got a v name which uses the virtual machine name or you've got a vm ID parameter which is used is this gooood which one you want to type I'll leave up to you but I use the name and if you get an error with PowerShell direct it's usually one of two things either you put your credentials in wrong or your path has been screwed up and partial F partial X C is not on the path on your VM and you can't find it I think that's it the slides yeah thank you do you want to see some code oh you keep making me work okay usual nags at the top of my demo to remember to start the things that I want to use okay when we were when we remoting if you had problems with remoting the error messages and not the most helpful that's about the Politis way I can put it and what's worse is if and I noticed this when I was going through the setting this up the error message is that you get don't always match what's in the documentation so there's an about troubleshooting remoting or about remoting troubleshooting or whatever it is that's out of date which confused me it doesn't take much but that did confuse me and said we at this point we're assuming that the network connectivity exists and has been checked i'm not going to cover the double hop problem does everybody know what that is anybody not know what it is good sorry you don't know what the double hot problem is okay double hop problem is a classic remoting issue I'm on my box here I want to remote to this box here that all works and then one on the remote session I try to remote to another box over here and it fails and it's because of the Kerberos the way Kerberos works it won't allow you to by default to delegate your credentials to that second box so that you can connect to the third box the way that most people get around it is to use credits this is P there's a few issues with that ashle McGlone did a session last year which I think we managed to get it recorded but it's covered in his blog posts he's got some very very neat ways of getting around it if you want to know more about it come and see me afterwards and I'll go through the issues with you okay so remoting should work by default and we look know we don't and remove the session and it just works and that's what we expect now let's start playing with everybody's expectations what I'm doing or what I'm gonna do is I've got a bunch of scenarios I'm gonna work through we'll see how the time goes to tell me with we do but basically I'm just going to break remoting and the fun is that you don't know what I've done to break it and I can't remember either so what I do want you to do is take note of the error messages that come up and what we do to fix those because you'll see that we get the same error message in a number of places where there are different causes but you get the same error and that's one of the things that I find confusing when I'm trying to remoting because it could be this or it could be that or could be the other so hopefully this will help you well this sort of concept will help you get around that it takes a little while to fail come on faster faster oh come on it doesn't take you this long wake me up within this right thank you okay so that's a lovely message with our M cannot complete the operation thank you okay so when we get to the end I'll go through what all the tests are doing I don't want to spoil fun all right so we're now should be running let's fix that so start the winner em service what else can we do making Nestor that error message turn this anyway yeah yeah we're now M complete the operation you know listener you all know what the listeners are anybody not know what the WS man listener is that's what it's connecting to and it's actually a country because I've got rid of it let me put it back just ignore that no seriously just ignore it it it it's a it's a false error message okay the listener is this thing and it basically says what transports am I going to accept you know SS man remoting which server HTTP HTTP and what addresses am I going to get so that's all right let's break this again we've done that one it's time oh that's good we've got different message the Bissman service cannot process the request well that helps us so we've got the service running the listener's available it's enabled we're using HTTP dresses okay but the endpoint doesn't exist okay what we should have is a Microsoft PowerShell end point that's the default end point that you're going to be using so somebody's gone in and played around and got rid of that because they thought there's two of them and you don't really need both that is why it failed so my fixed oh yeah that's the same fix enable PS remoting that gets you at a lot of trouble if in doubt try that but sometimes the speed if you it's always worth just trying that rather than actually going through the methodology any questions so far what is that M get AB it will be available after the summit sometime towards the end of next week I've got a choice I can either do that for you I'm going to see my granddaughter's you guys lose okay so we got we got this another identical mer error message back what have we done this time what I'm trying to show you with this is that you can get the same error messages for an awful lot of reasons and this this one's the firewall blocking it and I don't know whether you're in the internal firewall on your machines or not but if it's not configured properly some bright spark from Security's come around and said oh you don't need that sure that there gives you remoting now it's just a matter of putting the firewall rule back on you know when you get that helpful junior admin that goes and tries to tidy it stuff for you yeah these are the sorts of scenarios I've seen people do this one's probably little a little bit obvious why it's not going to work access is denied that's this this is this is good one you might think that's something to do with permissions yeah you disable remoting and you get an access denied which is which is bizarre pardon yeah yeah so it's it's it's giving you an error message that saying your access is denied which is true but normally you think of access denied as a permissions issue not the fact that yeah yeah if anybody wants anything to do rewriting the troubleshooting remoting to actually get this stuff across would be a good idea pardon this this is just the series of pester tests running in the background but it just struck me that we use pesto for testing code and infrastructure why not use of the testing troubleshooting yeah so we just forced that to be enabled and with we're back in business and what are we going to do next yeah you said that that access denied area sometimes you'll get a connection to the remote host was refused that didn't that gives you a little bit more of a clue as to what it is but I have yet to see that one actually come up more than once 99.999999 percent of the time you'll get the access denied and remember that if you've got actually that that's wrong that should be public I'm just at that late at night changing demos on the fly all right if you've got a public network and you try to ennoble PS remoting it will throw a hissy fit at you and tell you it's not going to play you either reset the network connection profile or use skipping it skip the check anybody want to guess what this one might be what else can we break w us man can't process the request I think we've seen that one before I'll make all of these breaks and fixes and tests available so that you can you can play around with it yourselves and see what it's it's got this is back to the endpoint it's it actually exists this time but it's disabled which show you how to fix that again just enable the session configuration the one one thing to be careful about is there's two endpoints there's the Microsoft up PowerShell one which is the default if you're running on a 64-bit machine which all the servers are you'll also have a Microsoft up PowerShell 32 which is for anybody wanting to make a 32-bit connection if you've got people playing around on the servers that don't actually understand that you do need both of those you may find that one of them disappears so what port does remoting use anybody fault five and six what I've actually done is change it to 880 and then that works and you can see when I run the test it picks up that the port is different this is when it starts to get subtle when people are changing ports on you because somebody's had a bright idea Oh it'll be more secure if we change the port you've obviously get the same security guys that I have does any sorry I keep poking at security guys guys but it's just I've had some interesting very interesting conversations with them over the years and I suppose we should fix that because it'll confuse me if I don't I did that in when I was practicing this I forgot to run the fix and then try to do the next one and I fixed the second trouble but I didn't fix the first one and good yeah I was confused I deliberately dropped the firewall as part of this test because I couldn't be bothered putting setting up another rule on the firewall and then getting rid of it he would actually more likely if you changed the port you would fail at the firewall rather than at the listener oh I'll skip that basically if you have got for any reason machines on different ports it doesn't matter once you've created a session to it you can mix and match the sessions because you're running over the sessions and it it just works now so this one you might this is one that you might see so poor old bill is going to try and connect this we get access denied again and what do people need to be to do remoting like pardon yeah so looking at the securities on the end points the administrators are allowed the remote management users are allowed local administrators contain the domain admins and the local administrator there's nobody in remote management and if we look at parole bill all he's in is the domain users so he's got no chance and the other alternative is do you know this one you can actually pop the pop thing up and have a look at the security on the end point so how do you get around giving somebody access that isn't in domain admins or isn't an administrator or a member of that group you could add them to the group good pardon who said year you're the ones for you I'm not going to throw it because if you miss the catch it'll take your head also yeah this is where Gaea comes in if you've got the people that you don't want to give full administration access to but you need them to do some sorts of things like this G is great because you can lock them down and you can do that and if you try to do anything else no no chance anybody tried to do this connecting by an IP address rather than a computer name this is a client-side issue it's nothing to do with you remote machine you either use trusted hosts anybody use trusted hosts yeah yeah don't put star in it naughty stand up and apologize to everybody guess what project Honolulu does puts a star in trusted host why not on your computer possibly if you're talking about your work station then it probably is okay but I've seen a lot of people do that on servers jump off servers that's where I would not want to see it is that since yeah so SSH yeah it's a similar concept but it's no it's actually it's alright no feel free the trusted hosts is and the way it's handled in documentation it it it's bugged me for a lot of time a long long time the way oh just use trusted hosts no do properly use certificates yeah and Windows has a certificate server if you want a certificate to do remoting do you know how much that would cost you using the Windows server nothing why mess around with trusted hosts put a certificate on there because that also handles you non domain remoting which instance incidentally the SSH remoting is for me that's one of the big keys it makes it a lot easier to do non domain windows remoting both directions so trusted hosts it's just a strength you can put a set of machine a set of values in there it's just a simple set out and that's how you get to get around that notice yes I did use trusted host rod and the certificate because I'm coming on to that next okay I jumped ahead slightly by mentioning non domain remoting do you like my naming convention Indy okay we can't process it error occurred because Kerberos authentication said yeah and that's because it's not in the domain and we know that oh we hope it's there yes I did remember to start that one up you use just it host those certificates now that should be listening on five 986 there we go and we can create a connection and it's just a standard connection there we go the only thing that you need is that you will occasionally need your certificate server just stood up so that the server the certificate can be tested what other thing that you will need is a HTTP listener do you remember the listener I showed you it's just HTTP so standard listener looks like that with just the HTTP he usually is called that for whatever bizarre reason but if you've been playing around with remoting like I do that can't change so don't rely on the listener name and on the remote machine we've got an HTTP and a HTTP and the other thing that you will see is in the listener you've got the transport and the address everybody know how to create an edge to another listener all right I'm not actually going to run through it but the code is here so get the thumbprint of you or we'll get yourself a certificate SSL certificate first get it installed on the box you all know how to do that yet great get the thumbprint of it and then you WS man instance winner I'm config listener address equal star because you you don't actually know where your connectivity is going to come from transports HTTP hostname the certificate thumbprint and you're good apart from make sure there's a rule and the firewall that let's HTTP through and that's all it takes easy okay some other things that you might see if you get these sorts of error messages cannot complete the operation within the time specified that's timeout setting issue you can modify the timeouts most of the time the default settings will work for you period you'll see certain people who insist that you have to change this and you have to change that and you have to change the other no the vast majority of the time remoting just works leave it alone let it get on with its job the two places where you might very very occasionally hit issues are timeouts and the amount of data that you're transferring if you fall over on the amount of data you're transferring the question then is why are you transferring massive amounts of data think about the process rather than changing remoting and in the yes is this one that's what I wanted to show you so you see the you've got timeouts you've got envelope size and there's something down in here somewhere as well yeah you've got another timeout down here so you can modify those if need be but think about what you're actually doing as to whether it's you actually do need to change them and the way to change them is you've changed you can actually override them on the PS session option if you've not seen that before there's a whole bunch of stuff down in here for the data sizes and there's some stuff for timeouts as well yeah down here right so I've shown you a few ways that remoting can be broken and how you can go about fixing them the so how was I doing the testing and basically all I was doing was running so it's just that one I'm just running the series of Pesta tests so is the service running is the listener they're using test path just the listener enabled is the transport set for HTTP is it listening on the right addresses is the port correct does the endpoint exist is it enabled these are a firewall rule blah blah blah blah is remoting enabled that was an interesting one to test and the only way I could think of it was actually testing the the permissions but if anybody thinks of a better way let me know and then so that's the series of pass the test what I wanted to do though was I wanted it to set to stop when it hit a problem rather than going through and doing all the other tests so what I did was wrap the and just run each test individually and as soon as one breaks we stop and output the results and that's why it always said this is failed at the bottom and that's it that to me that is a testing and troubleshooting methodology that you could put against anything and it's extensible it's understandable you can give that somebody else to run without any problems and I think it's another good use for pester and I think that's it yep we got to that so if there's questions that's great if there isn't that's great thank you very much from myself and Anthony you've been great audience and I was trying to ignore you but yes sir [Applause]

Channel: PowerShell.org

Views: 1,583

Rating: 5 out of 5

Keywords: powershell, windows powershell, techsession, powershell summit

Id: cBQqewAZkFc

Length: 93min 48sec (5628 seconds)

Published: Wed May 02 2018