SF21VEU - 18 Trace Files Case Files (Jasper Bongertz)

yeah um welcome to my trace file case files session um i've done those a couple of times in the past at various shark fests and it's usually something where i just present a couple of interesting cases that i had during my normal daily routines of working on whatever problems the customers have and since yeah for quite some time now i'm focusing more on security so you will notice that there's a slight um well depend uh way of going into security topics in this one i also did not have much time to prepare because i got a bit sick and had to recover from it so no it wasn't coveted but well we'll see how we go all right so let's see first things first oh i just said i'm just for longer i'm here because i'm loving i love analyzing packets um and you can find me at package a on twitter if you want to if you don't want to that's fine too i have a couple of cases for you three and before we go into that well maybe i should tell you now that my current day job is working for a more or less small company in germany um mostly doing insulin response which means that some when customers get attacked by hackers or have any kind of security incident they usually need somebody to help them because that is not what their daily operation is meant to handle and very often what we get at the company i work for which is g data advanced analytics are ransomware cases and since i'm one of the per people or guys persons women whatever that go out to the customer side to help them um i see a lot of those in my time um i had a couple of big cases in the past and uh yeah one or two of them we will take a look at here and see what wireshark can do to help us and the first thing is something that is about two years ago i think i call this case a beachhead case it is a ransomware victim in this situation and um now the problem is they called us and told us everything is encrypted um completely there's no way to stop any encryption activity from going on which sometimes happens sometimes companies notice that the encryption started and can do something to prevent it for example shutting down servers or what most of them do is actually disconnect their internet connection to take away the control that the attackers have over the network because if you're not connected to the internet usually the attackers from the outside worlds cannot do anything anymore in your network but you still of course have sometimes stuff still running in your network which means shutting down the servers and cutting the internet connection is usually a pretty good idea in a ransomware case there are other cases of cyberattacks where it's not such a good idea to cut the internet connection because sometimes you don't want the attackers to know that you spotted them or that you suspect that they are there in which cases you should not disconnect your internet connection but this is something that needs to be decided by case by case and in this case they called us and they told us everything is encrypted the encryption's completely gone through and we haven't disconnected the internet connection should we disconnect the internet connection well in this case we said it's probably a good idea not to connect disconnect the internet because then we can take a look and see if we can find anything that they're doing because if they can't do much more damage to the data because it has had been completely damaged already then maybe we can take a look if they have some kind of beat chat or well any kind of callback or other things they're communicating still with and um this minds might sound a bit puzzling because i just said they encrypted everything what the attacker groups usually do is they leave everything in tech that you need to communicate with them and let them have bitcoins which means usually they don't take down your active directory infrastructure which by the way doesn't mean that you need to completely do it from scratch because when an attacker has taken over your active directory and they usually do you need to recover it and recovering it means basically building it from scratch so in this case the active director was still online and i think a couple of not so important servers were also still running but everything else was encrypted and that was really tough for the company because they didn't have backups anymore because the backups were deleted first [Music] the attackers did that by finding the storage system that was used to backup stuff and they initialized it to factory defaults wiping everything off it which couldn't be recovered anymore so no backups nothing anymore so what we were trying to find out in the beginning of this before starting to do everything that is required to get the company back online is see if there's an attacker still in the network and how do we do that well we install a sniffer usually at the perimeter firewall and if you take a look at this very simple diagram here and to the right of the diagram with the internal networks which of course there's a lot more than just one switch it's the company like with about 1 000 employees i think so there's much more tech technology on the inside of the network i just left that out because it's not relevant that much but if you're into security a little bit and even if not one thing should be puzzling you or frightening you here in this picture because there's only one firewall um they didn't have a second firewall between them and the internet which you usually do if you expose anything to the internet like having an outer firewall and an inner firewall with a dmz demilitarized zone in between they didn't have that so what i did is i put in a full duplex tab one gigabit copper and hooked up a sniffer and captured everything that was going on it was a gigabit line so yeah there was not much going on on that link usually they have like i think about 600 megabits of traffic on that link but since everything was shut down on the inside of the network or almost everything there was not much going on and capturing when not much is going on is actually quite nice because then it's easier to find stuff that you want to look at so this way i captured the data and then i pulled it off the sniffer notice here that i'm not doing a span part i'm also not capturing on any system involved um especially not on the firewall because in a situation like this at the beginning you never know what you can trust so it could be possible that the firewall is compromised that the switches are compromised and everything is compromised and that the attackers have found a way to hide their communication from me because theoretically and i say that in well quotation marks because it doesn't happen that often really really they could actually try to hide the communication from you which normally doesn't happen but it's a good idea to use a tab because the tab is your device it's completely passive from a network point of view it's not completely passive from a lecture point of view but forget about that but if you put a tab in like this a copper tab is quite good a fiber tab would be better the attackers have no way of hiding their packets they have to go through that device and if they go through that device they get forwarded to my sniffer and there i can capture them they have no way of hiding their communication if i put in a tab like this the only drawback here is you need to be sure that this is the only internet connection that the company has if they have more than one well sometimes in different branches or something like this and they're all interconnected then you need to think about where to put what but in this case they had only one and i put the tab in there so let's see if i have the trace flowing here or here is it i need to find my wire shark yeah i have too many wire sharks everywhere each head anonymized there it is so it looks like this and i know that the font is too small i will make it bigger i hope this is okay like this and of course it's anonymized because well you wouldn't want me to show your compromise traffic if you got hit by ransomware attack right so anyway um this is some of the traffic that i captured i captured a lot more i think over the whole case i capture about eight terabyte of data in total because we also often capture during the recovery and re-establishment phase or the build up of the network to check if everything is there and nothing is there that shouldn't be always keeping an eye on things and if you look through this there's a lot of packets and scrolling through packets sometimes is helpful sometimes it's not but in this case looking for things like re-transmissions and other let's say performance errors is not my problem what i want to know is is there an attacker in here somewhere and what you can also see in some cases here is that there are destination unreachables um like a lot of them um and they will they would also be interesting in a performance troubleshooting but not so much in this case because we know they shut down all their servers so a lot of communication is trying to reach the service because the internet is still online and can't get there so you get destination and reachables and you also get destination algebras in the in the other direction for some reason that's how it is so how can we find if there's an attacker and what attackers usually do and that is the kind of knowledge that you probably need to have is um if they're in your network um they often create something like a callback mechanism that means that they're pinging one server of their own in the internet to um get new commands and and receive anything that the attacker wants to do in the network and that is like um commander control traffic so to speak and command and control traffic is often if the attacker is not very very careful hiding it very easy to find and we can take a look at for example the statistics here look at conversations and we see there's a lot of conversations here and a lot of them are going from the same address to the same other address but is that enough to be a command on control well in this case sharks may be quite okay to do this and take a look at it but what i did in this case is i actually used my own tool which is trace winger i don't know if you've seen trace regna it's used by almost all the presenters now i think to sanitize their traces but i also use it a lot to work on great amounts of data because it's often complicated to load more than a couple of gigabytes into wireshark and even loading gigabytes into wireshark is not fun at all so if i add this file to my own tool it will scan it completely and extract a lot of details that i can take a look at and now i have a tool here that is called endpoints and conversations and if i open that i go to the ip address range this looks familiar right it's like the conversation summary in the background from wireshark and what i have here is a column and this is really small so let's see if i can zoom in that is actually telling me how many flows there are between two ip addresses and i can sort it and you can see here that there's 1430 flows between these two ip addresses and that is quite a lot if you consider that this is a i think 20 minute capture or something let me check i think in the statistics we can see that the total duration is just 20 minutes you can see it here 20 minutes and in those 20 minutes there were 14 000 connections from one ip to one of our own and that is quite a lot no i needed to look at the conversations i was in the wrong tab there was addresses i wanted the conversations but still that is the same thing i can also see here the flows it's only 9000 but still in 20 minutes and there it is 20 minutes duration 9 314 flows yeah that is quite a lot right so um when we look at this conversation and what i can do here is if i select this i can copy a filter in wireshark syntax for this thing and insert it here it takes a while because we have about 500 000 packets then you can see that there's a lot of conversations going on and it's always like if i colorize it not very long it's since the neck hack which is a handshake then there's some push push push push egg and so on fin egg fin egg so this looks like well there's something going on but we don't know exactly what but i had the advantage of of course i wasn't working with a sanitized trace right so i was able to look at the payload and you can see from the port 443 that this is supposedly https so in the beginning of an encrypted http communication over https um you have this whole certificate handshake kind of thing tlsn check client hello server hello a lot of packets in the beginning are just for the handshake and i had to remove them because the transfer so far is not capable of sanitizing that in a way that is useful so all this information is missing here but let me tell you that in there you had a crypto handshake so to speak but not a normal one from https like you would see from a website it was more like a self-written tls protocol thing it was using a correct crypto but it was not doing anything like a web browser would and it did that over and over and over again you can see here in the delta times between this conversation the the colored one and the next one there's only 77 milliseconds until it tries again now let's colorize that one and when that's over it waits for almost 100 milliseconds and then it waits for there's a little bit of interleaving here but it's really like firing quickly and it's always doing the same thing it is connecting with the tcp handshake it's doing the crypto exchange and then nothing happens and it closes the connection again then it starts the connection again does the crypto exchange nothing happens it closes the connection again and it does this in 20 minutes over 9000 times so it's quite obvious that this is like the callback that the attackers are using and um well we didn't see anything in there where they were still actively trying to do anything like we didn't see any kind of larger payloads going back and forth so um as far as we can tell they just had this lifeline still going while they were waiting for the ransom money and unfortunately since um the company had no backups anymore and had no tapes and no other offline backups they had to pay the ransom um which you normally would should try to avoid but they had to and they got the decryption tool luckily and it worked which is also very lucky because sometimes they don't um and at that same point in time where the payment was made this communication stopped and you couldn't see it anymore so the attackers kept it going until the exact point in time until the ransom was paid and then they left them alone by the way they send them an an email telling them where their security issues were and what they needed to fix so that it wouldn't happen again like a sort of service to the victim that already had found out that it was not as well protected it should have been so this is a case where we could find one of these callbacks it only showed us that they're still in there we kept monitoring them we saw that they were not doing anything else um but of course it stopped when the ransom was paid and of course everything was built from scratch and this company has spent one and a half years by now rebuilding everything in a much much better way spending much more focus on security and much more network segregation um a lot of my my customers who have this kind of problem they tell me hey we have vlans and everything is segmented and then you ask them like okay you have them but is there a firewall in between them and they usually tell you no because everything must be able to connect to everything because they don't want any troubles with something not connecting to the other thing and that means that if an attacker gains access to your network it doesn't matter which part of the network it is it can go anywhere and usually after you pay a ransom you find out that it would be a good idea to have a internal firewall that separates all these nice vlans that you have so that only the communication that you really need goes through and this is one of the two key points that you probably can take away for your own company if you don't have it yet have offline backups because the online ones will be found and they will be destroyed we have seen attackers reverse engineering self-written backup software until they understood it and could destroy it so have offline backups tapes and i don't care disks that you put into a safe somewhere and segment your networks have firewalls in place and actually use them all right so that was that one this can go away i don't need tracer anymore and let me check if there's a question there are questions okay jim young is asking are you worried about session drops or state teardown by inserting the tab in normal operations it is a minor concern because i'm pretty quick inserting a tab i can usually do it with a length state down of about three seconds if i prepare myself it's like a really quick show of the cable [Music] but uh in normal operations of course you would get like a small maintenance window for this kind of thing um but in a ransomware case where you are you have all the servers shut down anyway it doesn't really matter you can take half an hour to put a tap in if you want to because nobody cares it's just you you have the time okay how do you manage capturing terabytes of data do you use packet broker what equipment or capturing do you use commercial synthetic dc what i use is of course i use commercial tabs because building your own tab is not possible anymore if you're running gigabits or higher because um the self-built ones that everybody had only work for 100 megabit and below because of the way gibbet works gigabit has only four cables that it communicates with um or no it has eight cables it has four times 250 megabits per second which is a gigabit but it needs to do that full duplex so that two signals are on top of each other and um for that means that only the end points know when the signal is sent or not sent and something in between that is completely passive can't at least not for copper so um you need to buy taps actually and the sniffer that i use is a self-built sniffer but it has a commercial fpga based capture card um so the the this pc is a very small shoebox kind of thing um it's very portable it's like this big um and the card in there is probably five times as expensive as the rest of the device because these fpga capture cards are they're just expensive but they don't drop packets and if you need to capture everything and in this case i needed to this is the good way to go okay how did the certificates look like um yeah i think they were self-signed i don't remember because it's uh almost two years ago um we just looked at it and we saw like this is not normal um this is something somebody builds like on their own certificate authority or something i think it was something bogus i'm not sure i can't remember okay you should secure your service line machines yes you should um inserting the tab sounds like a competition for the next person in person shark list who can insert the tab the smoothest way without losing the most connections actually we could we could make something out of that yes that would be a fun thing to have okay if there's no more questions i continue to the next one and guess what my mouse pointer is somewhere else there it is it is a ransomware victim yes another um i get a lot of those and um in this case um they were able to recover from backups which is good because they didn't have to pay the problem here is um well not the problem but in this case they tore down the internet connection we checked all the servers i have colleagues who can check servers like in record time we have reverse engineers that can pull apart malware and see what it does so in this case all of that was handled but what they had or what we didn't have was the initial point of entry so we didn't know how did the attackers actually get into the network and to this day i think we don't know how they did that which is a bad feeling because in a in an attack like this um you usually want to know how did that happen so that you can fix it for the next time um what we suspected or what i suspected or what at least i suggested is they have one application that is used um to order their their goods um like an order order entry system it was self-written of course it was a web application and self-written web applications in a small to medium-sized company i think they had also around 800 employees so to speak self-written applications in that size usually mean that nobody cares about security in those applications and it's a pen tester's dream to work on an application like this because they usually find all kinds of problems with it like sql injection and cross-site scripting and whatnot so they had this application and they needed it to run as soon as possible because um if it doesn't run nobody can order their goods and that means if they don't do that or if that application is running for let's say a week or two they are probably in a financial situation that threatens the survival of the company so it needed to go online again we couldn't fix or we couldn't have an audit that quickly on that application of course because it's not that easy to do that just on the spot so what they did was they put a um web application firewall in front of it and put in some a good set of rules to allow only calls that make sense to them so they they defined a policy that said only these kind of calls with these parameters are okay and the parameters are not really complicated so it's not like rocket science to do that they should have done that before they didn't but now they do so the question was is that still good enough and so the task was to verify are there any calls getting through that firewall that shouldn't and that means that we need to capture again and check the traffic for all these communications and see if only the allowed parameters are or calls are seen here's their setup um at least they had two firewalls in this case i had the parameter firewall and the replication firewall well they had that after the effect so it's basically like the first case um and the zipper was placed between the web location firewall and the web application itself um actually this was kind of a bit of a yeah issue because both the web application firewall and the web application were virtual machines so in this case i had to capture on a virtual machine inside their virtual infrastructure using a distributed switch and using a span port on this virtual switch and getting all the packets this one is um that was the advantage they had the highest vmware license i think where you can have the distributed switches with the span port feature and so we could just spend the traffic to a virtual machine if you do that like capturing to a virtual machine you need to make sure that this virtual machine is on storage that is fast enough and it should be on dedicated storage because otherwise it could drastically reduce the performance of the other machines on that storage so um that setup um i of course i had to do that because in that company nobody knew how to do a setup like this which is quite typical um but let's take a look at the trace so um [Music] there are a couple of files um i captured for a long time so um usually i do a capture that writes like a couple of 100 megabytes or something per file um and now all my taskbars every once in a while to check if all the parameters all the all the calls are correct um or if there's anything that shouldn't have gotten through this replication firewall so as an example i can show you packet number 65 i think it was that's how it looked like and there's this http call in here you can see i sanitized this one with stress wrangler it doesn't sanitize http yet so i had to do string replacements which was quite time consuming but in the end this soap action is the thing that i needed to look for that is the interesting thing this is the one where i needed to make sure that only the calls that are allowed get through in this case you can see the call here is the test call it's just called tests for i don't know what it does but that was one of the valid ones so there's also an authorization in here i replace that with shockflash chalkfest because sometimes people in those presentations grab this string and they know it's based in 64 encoded and reverted so i had to replace it too and now the question is how can we get all these soap action calls and verify them so my first idea was hey this is not that complicated we can just go and make a column for this one so as you probably know you can create custom columns with any kind of info from any field in the decode you want for example if you want to have a let's say content length column you could right click and say apply as column and then you need to re-align your column sizes and then you can see when there's a packet that has this value it's in this column so i don't need this one what i need is i need a soap action column so what i did is i clicked apply as column or also what is possible is you just drag and drop it here let's say i want it here and now it ended up here the problem is you can see my soap action is in there but that's not the only thing that's in there and the problem here is that this soap action is if you can see the filter for it it's http.request.line which is a generic field so it's just a text field um in the http header where wireshark doesn't know what it is so the soap action is something that it doesn't know so i can't create a column that only contains it or hold it which is a problem because i only want to see this one and i don't want to see the uj user agent because you can see here it has the same request line it also says http request line if i click on that right the header has its own but the user agent is is hgb no i got get myself into trouble here again um well anyway http request line is not good enough so what i need to do is i need to tell wireshark hey this soap action is a field that i need and if you have seen uli's talk on the first day which is i think was yesterday then you know or probably know or maybe know that you can do that so we can go into the preferences protocols look for http and you have something here called custom http header fields so what i do here is i add a custom field called what's it called the name was soap action it must be written the same exact same way soap action soap action like this and i give it a name so action okay okay and then wireshark has to reprocess all the packets because now i told it to give me a specific field for the soap action and now i have a http header sub action field so now i can do the same thing again move it up here maybe this works the way i want it to yeah there it is and now only my call is in there the only thing that i need really so the annoying part here now is that um i don't want to basically load all of the files one after one to check all these soap action calls and see if there's anything in there for example especially since i would now need to look at all of them and compare them to my list and say this is a valid one this is not a valid one so in this case what i do and most people usually do as soon as they know that this is an option is we go to the command line and do that on the command line instead because doing that in the gui is too much work and we're basically [Music] well lazy analysts need to be lazy because otherwise it takes way too long to process everything so let's take a look in the defense one and you can see here i have a couple of files not just one so what i do know is i use t-sharp and i use t-shark to read let's say the first one and i want to have a field extraction and my field is called http dot header dot soap action let's see if this works ah this is not valid why isn't it valid the problem here is that t-sharp uses the default profile and for this talk i'm using my demo profile so i need to tell t-shark hey please use my demo profile because only in the demo profile i have this field defined in the default one it's not defined so it doesn't know what http header soap action is supposed to be so i need to give it the profile name of the profile where i put in this extra field that i just configured in wireshark which is why you need to go into wireshark sometimes to do this before you can run something on the command line so this should work now yep good so this is um nice but um well i need to do that for all the files and to do that what i can do is for percent a which is a variable in star dot p cap and g do so that means read all the p cap and g files and now of course i give it percent a so that it reads the file of the current name and output the stuff so there's still a lot of empty lines because not all packets have this field obviously so what i need to do here is i could tell it to say i want a filter http method so that i at least only get not sort the name of this damn thing i just want the post fields so i need request method okay sorry a day's too long request method http dot request dot method yeah whatever okay nice this looks better so um you can see that a lot of calls are repeating it's themselves so what i usually do here is now pipe that into sort which is a unix tool which i download from sourceforge usually and unique so that it gets sorted and then it gets unique and i can add a counter so that it counts for me how often it found the communication or the parameters and now we wait until it's done and i only used six files here so it should be relatively fast and there it is why oh i pressed enter again so you can see here i have this number of calls and now i can easily just check hey this is a valid one this is a valid one this is a valid one basically what the application people can tell me what the valid calls are and well i can just compare them and check okay is there anything in there that i don't know then i have a problem but it turned out that every single time i ran this and i did that for a couple of days just running this script all over again all the time on the new files each time i could just get the list that it gave me and compare that to the list of commands that were allowed and after like two weeks or so we were quite sure that the replication firewall did what it should be um what we could have done is actually capture in front of it to see if anybody is trying to do funny things with it we didn't do that but we could have done that to compare if the inner side and the other side actually are different but this is how the http header fields the custom field is really useful because it's yeah it saves a lot of time uh being able to run it this way all right so let's see what do we have from questions um there's one from miroslav what kind what are you using to store terabytes of data in customers location some kind of nas um yes sometimes we use a nas if it's really getting big so if we have like two digit numbers of terabytes um up to i think i have the largest disks that i have right now a single disk are 12 or 14 terabytes and i usually copy the data from the device in the night when there's not much traffic so you can also do it with just usb connected hard drives it should be usb3 actually because otherwise it takes too long um and it would be nice to have ssds for that but um they are too expensive so what i usually do is i have like western digital black or something that are really fast and copy that data just to the external disks if it's really a lot of traffic we can also do that to a nas that we have okay um okay that is not a question applies a column is a very neat feature yes it is um do you use windows a lot for analysis because that's what customers have actually no i use windows because i'm not really good at unix and linux and a lot of my tools that i know how to use are on windows and i'm just a windows person it's i don't i mean i'm only i think we have a handful i mean i have about 35 coworkers i think and all of them use linux and only three or four of us use windows but everybody uses what he ever wants um it if it works if you if it gets the job done it's fine um so i'm used to windows i know how to use it i can troubleshoot it i can write my own programs on it tracing is on windows i would have to run it in a vm all the time which would totally annoy me and probably force me to learn c and write it again um so sake is now probably asking me to change to linux but yeah it's just i'm just using windows okay anything else no okay so we have 14 minutes left ah jim has a little info that you can sort minus nr to get the top list yeah that is helpful but i needed the complete list so in this case it's just a sword unique all right this can go this can go too and i still have one question coming from miroslav what is your best practice to do analysis on multiple monitors like what is on which monitor um i have right now i think one two three four five monitors there's a sixth one for my office laptop what i usually do is i have the wire sharks on my center monitor and the right one and i have um like rfc's and other information to the left but it's totally up to you um the one the fourth one is up there so it's up there in the corner um that is only for status and something right now if you could see it it is running a giant clock telling me i have 13 minutes and 35 seconds left so it's just for the smaller things so just set it up the way you like to um i like wireshark in the center because i need to look at it and it should be a good monitor because you're staring a lot of on a lot of small characters all the time okay last one case number three um this is called seriously um and this is not a ransomware case fortunately because i have too many of those this is something where i was asked to help with an analysis because nobody had an idea what was going on it's this concerned a virtualization environment on two data centers those data centers were connected with a 10g fiber link or a 10ge link whatever there was a service provider doing that for them and the distance between them was about 600 kilometers so it was quite far away and what they wanted to do is they wanted to move the virtual machines from one data center to the other data center and they got to i think a speed of something like 25 megabytes per second or something really slow it wasn't it was like agonizingly slow and um they wanted to know what was going on so what they did is they called somebody to to check the link of the service provider ntn so they connected a device on their one hand and their other end and basically a traffic generator generating traffic and they found out the link worked just fine they got the full 10 gigabit so it must be something else and that was when i was brought in to help with the analysis and before you can do an analysis the first thing you need to do is capture and capturing this was i think the biggest environment i've seen so far from a throughput point of view because this is how it looked like um it was more complex than this i removed a couple of things and um well they had like in the new data center center which is the upper one they had up to 100 gigabit um they had i think 40 gigabit in the lower one and we needed to capture simultaneously to see where the problem might be so what we finally decided on and this capture point one here is annoying but it's not capture point one this is capture point four we decided to capture on this infrastructure with a real big um well yeah a big load of hardware that we carried in there like packet brokers and really fast capture machines with the equipment was like insanely expensive um and we put one on each side right just before the provider connections so um capture point three here is no capture point three is um right before the provider device and capture point two was moved here for some reason i can't remember but all those switches were not that relevant anymore um or didn't add so much um to the whole thing of this so we had two capture points down here and two crypto points up here and then we could capture a well machine transfer from one to the other and you can imagine this capture file gets quite big so this will be very slow when i show it to you and i only chose to do this because this is probably something that i can only do on this machine here and not on a laptop when i travel to shark fest and i hope i will be able to travel to shockfest again and see all of you people instead of talking to my monitor which is quite annoying [Music] so here i can load it in my wireshark and show it to you without having to wait for half an hour or something so what i want to do is um get to remove that power point i have preloaded the trace so that it gets a little bit faster and resize the columns and you can see in the statistics right this is i think a throughput of 25 megabytes and that is not good enough for the customer actually so they wanted more and i can imagine they wanted more so um what we did is capture on all four points at the same time and then i got these captures and the first thing if you get a multi-point capture like this is and you need to find out if there are differences between them and if you remember how this case is called it's called seriously i looked at the first trace file like this one and i wanted to see if there's any kind of packet loss retransmission out of order all these things that you would expect from a thing like this because i mean this is a 600 kilometer distance there's a lot of traffic going on and this trace file from capture.1 and i can show you the powerpoint again capture point one is actually not here it's down here this capture point had zero packet loss zero retransmissions and zero out of order packets not a single one and i have never seen that before usually you always have some sort of packet race going wrong and the first arrives before the second and some packets get lost and whatever not a single packet lost on capture point one so i checked the other capture points and it was exactly the same there was not a single packet lost anywhere not a single packet out of order anywhere and not a re-transmission anywhere and i'm not sure how that works but it just did so that basically means that when you're looking at the trace you are not looking for packet loss and you're not looking for any kind of funny out of order whatever kind of problem um but what you need to do is you need to look at what are the machines actually doing from a timing perspective and for that i have already started to open the sequence numbers graph the tcp trace one from statistics tcp stream graphs and time sequence tcp trace because it loads forever still and you need to zoom in really quite a bit and i will show you the zoomed out graph afterwards because it's very interesting too but if you zoom in to this graph and you can maybe already see it the green thin line is the window size the available window size and this is the the ceiling of how much data the sender can send so if if it's basically standing at full speed it should try to get as much data across as possible and what you can see here is that for some reason the sender keeps waiting and waiting and never reaching the ceiling completely and maybe that's also because it's not act fast enough but there's a lot of room and it's not hitting like a window um full or something um there's always space to go on but it doesn't and it's probably because um it has some problems with getting the data to be sent or the acknowledgements only trickle in in a certain pattern and i have to admit it's so long ago that i can't remember what exactly it was i think if i remember from the report that i wrote is that in the end the sending machine was stopping for some reason and not continuing and when we opened the case with vmware which was the vendor of this thing they told us that they are had already found this issue and fixed it in a later release of the hypervisor and that was how the customer basically got his full speed back i don't know if it was a full 10 gig but it was nice and fast so basically that was a problem in the hypervisor where it didn't actually speed up and use the full potential of the conversation of the of the link by the way this is how the graph looks like i want to zoom in again why does it let me zoom in come on i probably move the mouse wheel too much if you can see it this is like a typical line when you have sequence number wrapping because it was a 10 gigabit file and of course it wraps a couple of times so it goes up to the ceiling and then it comes down and starts from the bottom again so this is quite a funny thing as well all right so um i wanted to show you this one mostly because of i have never seen a trace where you have no especially at those speeds and those distances you have no out of orders and no re-transmissions and no packet loss that was just i was sitting there for like five minutes thinking what did we do wrong this is too perfect but it was basically just not the network it often is not the network in 80 of the cases but uh this was yeah totally clear it's not the network all right so let's see do we have some questions here sorry for this new question where could i found this session for replay that is not a new question we always put the sessions or the recordings of the session on the retrospective page so you can find it on youtube and on the retrospective page could sequence number wrapping be an issue if you don't use tcp timestamps um yes it could be but most modern stacks are now able to deal with that kind of thing i have not seen an issue with that recently but i have also have to admit that i haven't looked at many performance things in the couple of months because i'm more or less helping people get rid of ransomware now um so this is like a totally different thing you're not doing performance troubleshooting that much if you're helping people to recover their data and beat the bad guys okay did you capture post vmware fixed transfer no because um um i think it took a while for them to implement it and they weren't interested in that kind of thing because it worked apparently and then they were like yeah it works it's fine okay how often is the if column needed um the if column is this one here and i can align center this is the interface number right so it tells me interface id 0 and usually if you have a capture where there's only one network interface that was used to capture you it's always zero um but i because it's very small i i have it it's intentional to call it if so it doesn't take much screen space because green space is very valuable if you have wireshark and the number itself it's just the number with one one and three usually um i use it sometimes to see if um the capture was taking more than one interface because if it is there's a whole lot of problems that often occur because people are not always using good fpga cards they're using more than one card and combining the traces afterwards and that gets that into trouble if you're analyzing so it's nice to have because if there's always a zero that just tells me it's just my interface and i don't need to worry about out of order time out of order packets and stuff like this okay any more questions because my timer tells me my session is basically done i hope you had some fun i hope you saw a couple of things that were maybe new to you and useful and if you take anything away from this please don't get hit by ransomware because if you see me coming into your company that usually means that something went really wrong and that is not a good day to have okay matthias kaiser asked after having the vm fixed were there any retransmissions then i don't know because i didn't capture after uh analyzing this one so i'm not sure probably as soon as you um start really sending more data and and maxing out the connections i would expect to see packet drops yes all right so any more questions if you don't want to ask now if you want to ask later or if you want to talk to me directly just do that on discord i will be there um other than that thank you for watching thank you for listening thank you for staying with me this long and i hope you had some fun all right thanks and bye bye

Channel: SharkFest Wireshark Developer and User Conference

Views: 465

Keywords:

Id: 9ZVla0uIlY0

Length: 60min 0sec (3600 seconds)

Published: Tue Sep 07 2021