Solving VoIP/UC Drops & Clipping Problems

this is toys good afternoon good morning or good evening wherever you might be this is cheaper and we're doing another of our at quiet afternoon specials so this is segment to a we're solving VoIP or you know Unified Communications drops and clipping problems this time and again this is Tim Titus with CTO pass solutions and we're gonna take each segment in turn and dive into the most common problems you're gonna start seeing if you're dealing with a VoIP infrastructure so Tim let's drive in what does the drop or clipping problem actually sound like well a clock a draw for clipping problem typically sound like it missing phrases in it so I think everyone's familiar with that sort of sound that's what it sounds like people get frustrated because you you're only getting half of the words missing words words that are clipped off and so that's something that I think everyone has encountered because VoIP and you see drops and clipping this is the number one problem that folks have in the world of VoIP and so okay what causes this kind of thing most simply packet loss so networks have developed to be very able to handle a lot of packet loss you can copy a file and say gee this file copy should take 10 seconds if you drop 10 percent of the packets it's gonna take 11 seconds nobody cares it'll retransmitted be just fine you're dropping 10% packets on a voice phone call it's gonna sound horrible because you're missing 1 out of every 10 words or every 10th word gets clipped so people get very frustrated about packet loss in a real-time protocol environment like VoIP or video where as data people don't care and so that don't care historic has passed through to the point where now people do have to care you have to groom your network and solve the sources and causes of packet loss otherwise you're just gonna end up with call quality issues so okay that's cool how do you find it are there tools within most VoIP systems are there third-party tools well so most VoIP phone systems they typically they walk away from the network and say hey it's your problem you got to find the problem in your network to solve it so finding out where the packet loss is coming from finding out why it's happening and also the third thing that makes it really difficult is most of these packet loss problems are transient that means that the problem will happen for a short period of time and then disappear and probably by the time you end up going to look by logging into switches logging into routers looking around the network the problems long gone and you'll never find it until it happens again which if it happens again every hour great you might have a chance of finding it if it happens once or twice a day yeah good luck in finding it you're gonna have to stumble across it randomly to actually say hey we caught it and we found it it's on this interface and it's happening because of a bad cable yeah so what I don't know about you but it's always my boss calls me late at night saying it Brian my call yesterday at 7 p.m. really stink do something about it oh how so let's go through the different causes as to where packet loss can come from typically you have over subscribed links with no QoS imagine if you had a 10 Meg link going from Hawaii to San Francisco and you were copying just massive virtual machines from one data center to another across that link you're gonna flood that link pretty quickly with these data packets you try and get a voice call in there you're gonna be dropping a bunch of packets from the voice call you're gonna be dropping a bunch of packets from the data transfers but again the data transfers don't care they're gonna be retransmitted the voice phone call you can tree transmit the RTP packets you're just gonna have poor call quality so if you have a link that is over utilized and you have no QoS you're kind of at the mercy of what the network gives it's bits what's called a best-effort and you don't want voice ever using best-effort the second thing that can cause packet loss and second most likely place is misconfigurations now what type of misconfigurations can you have on a network well the most simple one is a half duplex link or a duplex mismatch sadly these still occur in a lot of networks as just a couple weeks ago I was looking at a company's network and we found that they had a voice gateway that was running a hundred Meg half duplex that's guaranteed to end up giving them call quality problems and the good news is they were able to get it fixed pretty quickly other miss configurations you can end up having a voice device associated with the wrong VLAN if it has the wrong VLAN you're not gonna get the dscp marking assigned to it it's not going to be routing the same way as the other voice calls so making sure that your voice traffic is all segregated onto its own VLAN is important to make sure it's just part of that voice protection that you've set up another place you can have packet loss is hardware failures number one is cabling faults I think we've all heard about the the person that said gee you know we didn't realize we had cabling problems and still we started looking at cable jackets and they discovered that about half of their cabling was all cat 3 patch cords that somebody found in a box somewhere and decided to reuse honestly if you see a cable that has cat three on the jacket it's old enough it ought to be tossed and by the way good practice whenever you toss a cable cut both ends of the cable off I've worked for companies before where you just throw a cable in the in the trash can somebody looks and says oh a perfectly good cable they pick it up out of the trash can they put it back into production and your problems have just followed you back into the network if you cut the ends off then you can't have that happen yeah my favorite is I don't know why telephone companies keep sending of silver satin flat cables with their phones I can't imagine anything worse than that yeah actually that's a good point using the right type of cabling is important those silver set they should have all been retired long time ago so if anyone uses them it's like you're kind of asking for trouble as far as it's not following the ethernet standard other hardware failures you can have NIC failures I think we've all encountered the the the switch that has a bad port on it we've all encountered devices that have a bad ethernet adapter on it so there will be failures occasionally happen and those failures can be the type where the entire network adapter fails that's fine you can typically deal with that and just say let me use a new port if the adapter half fails and what I mean by half fails is you can have an Ethernet line driver on an Ethernet adapter end up not being able to have enough power to transmit the distances that you have set up if you have a hundred meters to your switch you're really operating at the the length the limitation of of how far Ethernet can go if a line driver fails and you're not transmitting at full power it can't even do that distance other problems you can have that cause packet loss you can have device resource limitations let's say you had a switch or a router didn't have enough memory or you have improperly assigned packet buffers imagine a switch or router that has gobs of memory but there's only five buffered packets per interface that means anything over those five packets it has to discard so depending upon how much buffering you've defined on interfaces that can end up changing based upon the amount of resources inside of a device but again that number one reason for packet losses those oversubscribed links with no QoS and the miss configurations that you encounter so good good tips what I think everybody wants to know is what kinds of things do I need to look for so what am what am i tools available to me to try and find and solve these bugs so there's two real cases there's being able to find it and being able to resolve it so on my slide eight I have the ways that you use to find the problem so typically finding the problem your what you're doing is you're looking to confirm that this problem does exist now a call simulator or a PM application performance monitoring you can use this to say gee we do see packet loss between a and B you're able to confirm that there's packet loss you can use a packet analyzer like Wireshark and confirm and say yes we are losing packets between a and B you might have in your phone system the ability to analyze CDR records called detail records and say yes this call had packet loss between a and B so this gives you the ability to confirm and if a user says hey I had a poor-quality call you can say yeah my call simulator showed that you probably had a poor-quality call to my packet analyzer that was sniffing the call said yeah you had a poor-quality call the CDR record says yes you had a poor-quality call confirming the problem is really only 10% of the problem and so that's really where a lot of folks get stuck saying how do I actually solve the problem because it could be anywhere between a and B so let's roll into how do you resolve the problem so with resolution in mind what you need to resolve the problem is you need to pull out your network map and you need to identify every link every switch every router used between those two void endpoints then you need to log in to those switches and routers and check all of the interfaces for all of the error counters used along the path you need to investigate the QoS configuration so if we look at slide 4 what we can see here very quickly so this is the path through the network that this call took looking at slide 6 what you need to do is investigate every one of these interfaces every one of these devices for all of the error counters all of the CPU memory health on every one of these involved links switches and devices now some people might think well wait doesn't my monitoring software already track these error counters sadly these a lot of spreadsheets man well yeah it's well the problem is monitoring software will ping your switches and routers and tell you if they're up and down they will tell you about link status they'll give you some pretty utilization graphs but they won't tell you about packet loss in fact the way to think about it is if you had a single half duplex interface along this path could your monitoring software find it and I would argue that most monitoring software packages don't have the ability to find a half duplex interface in a network they're just not designed to look at that they can't find a bad cable they're not looking at the error counters so as far as useful tools and being able to diagnose these problems let's look at slide 10 and what slide 10 is going to show is if you have the right monitoring and the right capability you'd be able to look into the interfaces the switches the routers all of the error counters to be able to analyze what happened along this path so that you could say the bad call at noon that happened between these earlier in the day is we were dropping eight percent of the packets out of the high priority queue on this one router we had some carrier sense errors on this other router and we had some collisions on this other router you solve all three of those problems your call quality problems disappear because you're now solving the root causes of the problems so this is kind of the difference of why is it hard to solve the root cause of the problem it's that monitoring software doesn't go deep enough and you can look for this manually you can build out your map log into these switches look at the interfaces look at all of the error counters and find the problem or you could look at software you know unabashedly admit this is what our software is designed to do is help make this easy to give you the root cause answer of these problems right and you know previously I I've done an awful lot of spreadsheets where I have to pull down the errors if I if I even know where to look and then try to line them up on time a lot of work it is but it's it's also that you have to know the error counters to pull and you have to also know how to get the QoS stats off of the boxes because you might have a Juniper router a Cisco router you're gonna have a number of different devices and being able to normalize all that data takes a bit of time but I think if we go back to that slide 10 that's the difficulty is that you've got so many places that you can have problems and so many reasons at each of these places you can have problems this is why it's difficult and by the way notice that this path is just one path if you hang up the phone call and say gee that was terrible and you make the phone call and it takes a different path through the network that's healthy it might be a perfectly good phone call at that point so this is what makes it so that people have a lot of challenges saying well sometimes it's bad sometimes it's good but it's that there's so many moving parts it gets very challenging and monitoring software doesn't go deep enough to answer what's really going on on these links switches or routers cool you know i I've been using all fess up I've been using total view for quite a while and it has certainly solved an awful lot of my problems well we've made it let's solve problems for engineers because it's difficult yeah exactly what kinds of other tools you know not everybody's gonna mean by total view right now but what kind of tools can we start using to help justify moving towards more advanced tools I guess is the you know the pathway we want to go how do we justify to our boss so in one sense that there's I'm gonna say you're always going to be good with Wireshark as a confirmation tool it's gonna be able to confirm that you're dropping packets the problem then is finding out where those packets are being dropped and why if you have a call simulator that can help narrow it down sometimes to be able to simulate a phone call halfway across the network and say let me test just this portion of the network and say is it stable for half the network if it's stable great you now can rule that out and say it's not part of the problem and so having good troubleshooting methodology of let's divide and conquer find out is the problem in our part of the network or is it in the carriers part of the network and if you say gee our part of the network is perfectly healthy we've been able to validate we have no problems in our switches no problems in our routers just by logging into them and looking at those interfaces looking at those areas you can then say ok we've got confirmation that we're running clean then we want to end up doing some testing across the MPLS or testing across the carrier's Network and validate and say yes the problem is in their network and so that that troubleshooting methodology should include a rule in of ruling in it's the carriers problem and rule out of saying we can also confirm it's not our problem because sometimes you might be chasing two problems at the same time and not realize it ok so let's get down to brass what kinds of best practices should our viewers be doing you know what kinds of documentation can we do help the problem you know get away from the let let's ignore the cables for now but you know what kind of things should we be documenting well so best start point is have a good network diagram know your network I mean that know your network of what devices are plugged into what how is your network connected I can tell you the first thing is an IT director when I came into a company the first thing I would do is say give me the network diagram I need to understand this network from start to finish so that I know how it's connected once I knew how it's connected I then logged into each device each switch each router made sure that the logins were working that I could get into the box made sure that we had a good inventory of these devices so that we know what comprises our network and if I found some old 3.com hub I'd say ok that needs to be the first thing that gets off of this network because I don't want to deal with the collisions the next thing beyond that is as far as understanding your network is make sure that you have any half-duplex links seriously half-duplex should be gone from the network should have been gone ten years ago but it's sad as to how many networks still have a bunch of half duplex links or duplex mismatches that still cause problems for networks that people just don't know about and if you knew where they were you'd immediately go solve them and proudly say I run a collision-free Network you know I I can't tell you how many times we've had is it might be perfectly fine gear but sometimes different brands and different models of switches don't exactly play nice together so I've had several instances where Brand X and brand Y didn't negotiate correctly and sometimes we've actually had to go and hard set the duplex just to get around those and that was one of the first things we did when we set up total view in my lab is like I have that many half duplex devices yeah yeah I hear Neil Allen in my ear saying right now if you hard set one side to be a full duplex make sure absolutely sure you set the other side up to be hard-coded full duplex if you set one side to be full duplex and the other side to be auto you are guaranteed to end up having a duplex in this match so when you go setting things hard it has to be hard on one side hard on the other side get rid of auto the reason to get rid of auto is auto works in probably 95% of the time but it's that 5% where it doesn't work it creates problems and that 5% is where you're gonna have to hard set both ends of the conversation to be full duplex on both sides yeah my other favorite one is oh there's a problem with a patch cap we just patched it in randomly don't you just love that yeah in other words they plugged it into the wrong VLAN there's no queue less on that interface I all sorts of problems say they plugged it into a sniffer port I mean you can't just randomly choose a port and assume unless you're the one who built the network and you know all those ports are on the right VLAN but you know it'd be nice if somebody somewhere made a switch that actually showed the VLAN numbers that were associated with each switch on the front of the port but I guess that would take too much space on the front yeah well I'll tell you what what we probably ought to do is go and summarize this segment you know what kinds of things you know what we we know what it sounds like because it sounds like vowels or syllables are missing typically we've got packet loss maybe some out of order packets why don't we go and do a quick summary and maybe throw out some good resources for people that they might want to use to help with their network okay so again clipping and drops it's packet loss it can also be out of order packets but typically you're trying to find out where are these packets going missing if you have QoS enable it there's many organizations they don't have QoS enabled really if you have any links that have any sort of bandwidth contention even you know you're hitting 50% utilization on your when link put QoS in place it's going to help make voice a lot better people think that oh I have gobs of bandwidth I don't need QoS not necessarily so QoS will be helpful in all those circumstances and then just know your network know where you're dropping the packets know why you're dropping the packets and then if you're interested in learning more about solving voice quality problems you can go ahead and visit our website we have some great blog entries ww path solutions comm and our blog talks all about how to solve call quality problems without buying our product but it's really to help educate folks so that you can solve these sort of problems on your own and if you need help then you can reach out to us and we can offer some tools that will help make things easier you know I think I'm gonna throw out one more I think it's document you know document your network because it's going to be that one switch that's off under someone's desk or something that's going to be your problem child and if you don't know about it it's going to bite you in the butt yeah yeah I forgot about that but you're right it's the switch hidden under someone's desk is it's always the one causing the spanning-tree loop or causing collisions in the network or just oh I just thought it'd be innocent to plug it in and it's like no a bad idea yeah well you know what this was the segment to a solving VoIP and unified communications drops and clipping problems next time well the next one we're gonna go and start looking at how to solve echo problems in segments to be but this is been Qi Bert bride Qi I am adv n ET la be on Twitter and love to hear from you and this has been a quiet afternoon special on solving voice over IP problems [Music]

Channel: TWiT Tech Podcast Network

Views: 1,740

Keywords: voip, tim titus, pathsolutions

Id: -_ubJ8-8Z1E

Length: 23min 11sec (1391 seconds)

Published: Fri May 24 2019