r/TalesFromTechSupport - IT ALL GOT SHUT DOWN

what's up guys welcome to daily dose of ready I'm your host Zach and today's subreddit is our slash tales from tech support don't forget to like subscribe and hit that bell to never miss an episode this story was called the day of the cactus for my nightmares in the mid-2000s I give you the day of the cactus it's a nice quiet morning in the call center in El to support at a major u.s. telecom it's a Sunday it's around 11 a.m. I clocked in almost three hours ago and I've taken a whopping 2 calls only one of those needed troubleshooting the other was simple it's so slow our team is gathered around on the ground yakking and playing games rather than at our respective stations the room is pretty small only about 25 people in the room fully staffed about half that are in the room up on the wall is a gigantic map of the United States made up of LEDs most of the map is green this indicates everything is normal a few parts are yellow this means trouble has been reported but the system is still responding and a few red spots and dots here and there indicating outages I've got my shoes off with my feet up on my desk reading a book all in all it's a wonderfully relaxing day then it all went wrong terribly wrong first was the beeping alarms which none of us knew existed what the heck is that god-awful racket coming out of every terminal then I see the wall and my heart has stopped for a moment California from the middle to Los Angeles has gone red millions of people are out of service in the room we were staring at the map the computers and each other the headset starts beeping as people start receiving calls I get a call within a minute there are hundreds of calls in my queue the general queue is in the high thousands and climbing rapidly my system has populated an account from the Los Angeles area customer is angry has no service whatsoever I try and cannot get response from his phone actually nothing will respond it's like a meteor has destroyed Southern California except I'm on a landline with someone right now the entire call center is slammed I have to tell my customer I'll give a call back then put my into not ready to start investigating what the heck is going on nobody in the room knows anything yet the problem is in the GSM network everything else is working I can reach all the equipment but all GSM traffic falls into a black hole of unresponsiveness then someone shouts there it is there is a station not responding along Interstate five it's completely unresponsive like I can't even talk to the Internet equipment and that is not very good it's the link between the halves of the state the rest of the network does not appreciate this link being down either problems are widespread as the system tries but cannot handle the load other systems are now being affected emergency dispatch technicians are sent to the area to determine what the hell is going on my Nortel has the call queue in the tens of thousands the previously laid-back atmosphere in the room is now grim as death agonizing useless incoming escalation after escalation where I can't tell them anything useful grinds on people are very unwilling to hang up the darn phone without any kind of answers it's grueling and repetitive finally word and pictures come back from the dispatch technicians there is no relay station it's gone the way of alderaan a truck driver with a fully loaded trailer fell asleep at the wheel and drove off the highway he made a way off the road and plowed straight through our gigantic fake cactus and associated hardware of a relay station obliterated nothing to fix and the site isn't even usable sweet Jesus the whole time I'm fielding calls that I obviously can't do anything about and people are angry I actually did get one call from someone not in the affected area who waited through that queue only one out of hundreds but it was a welcome relief so this is how we got to learn to build and deploy a mobile relay station it took a crazy effort between teams to build the equipment then figure out how to deploy it by the end of my nightmare shift the link was functioning and was literally a van Network parked around the brutalized remains of a giant fake cactus and out building things were finally coming under control around the time I left and the queue was emptying by the time I walked out the doors about a hundred two minute long calls later I felt like I had run a marathon physically and mentally exhausted I don't even remember the drive home my head hit the pillow before the Sun had set the day of the cactus was finally over oh boy okay so as someone who's worked in customer service effets old I'd love to take this opportunity to tell you guys that customer service is absolutely horrendous I hated it like people people are horrible okay complaints all day get to you and trust me when someone's calling you it's not to tell you hey you guys are fantastic no it's to tell you that you suck and you're ugly and you smell bad and you should go die in a hole basically pretty much 98% of the calls now alrighty this next story is called the day I shut down an entire government agency so a little backstory this was in 96 to 97 and I had been working with this IT services company for only a few months we had two customers with large HP 9000 UNIX mainframes I remember at least one of them was about 5 feet tall in the back of those enclosures you could swing open the door and mount rack equipment alongside expansion modules for the mainframe itself one of those customers use that space to store network switches and hubs in the rack section and it was common for us to go into the back of those servers and make changes to the network topology service networking equipment etc while the server was still busy humming away we had other customers that had Ti 1500 UNIX servers the way these servers were described to me at the time was quite literally the little brother to the bigger HP 9000 the enclosures of the tea eyes looked nearly identical to the HP's along with the back door that swung open and gave access to the expansion modules only on a much smaller scale it's also unable to store rack equipment like its Big Brother now I was not a UNIX guy by this point I had dab dab it with Linux but was hired primarily for my knowledge of Windows desktop in NT Novell and various land and wind technologies we also had UNIX specialists on staff and nearly every one of the companies we had contracts with had their own staff as well to take care of these machines suffice it to say I knew little to nothing on these machines themselves as I was in the world of Pentium Pentium Pro C ryx and AMD now on to the day in question one of our clients was a county government agency this was a fairly large County so there was a lot of IT infrastructure involved with their operation at this time they were an oval Network and on the PC side were just introducing Windows and T machines into their server infrastructure but they also utilized a single Ti 1500 that served their main application via green screen terminals either terminals themselves or via terminal emulation on PC we had just done some major changes to their network by deploying some new windows and T servers and some new switches and the owner of my company wanted us to go through the server room in equipment closets and confirm and update our maps of the connections as an added bonus he wanted us to tag the ends of cables with labels that said where the cable was plugged into at the other end so I get there early one morning with a coworker and said about doing just that we started with the equipment closet farthest away from the main server room in the game working our way towards the center of their IT universe we get nearly all the way through the main server room and are left with several cables going into the back of the TI 1500 enclosure so we do what we've always done pop open the back can close your door and start tracing then our cables leading into the back of it we're about halfway done when the IT manager rushed into the room saying that all of the agency locations have crashed he walks up to the main terminal and it's dead passed on ceased to be this is an ex terminal he asked what we had been doing before crashed so we told him turns out the TI 1500 has a kill switch that shuts off power when you open the enclosure so as we opened the back to map out those connections we also managed to kill the power to the server and shut down the entire organization in the middle of the workday when he asked why we did that even after reading the warning I said something to the effect of warning there on the backdoor of the enclosure was a large a neon green sticker which said in large print and I'm paraphrasing for memory do not open this enclosure while the server is powered on a few minutes later everything was back up in no damage to equipment or data so we were finally able to sit back and have a laugh about it both myself and my coworker are about to celebrate 23 years with the company and we still have this government agency as a client although about 10 years or so ago I moved to software development alright so that story was cool and it's kind of funny how he accidentally shut down an entire government agency however my feeble mind could not understand more than 28.4% of what I just read so there's that all right my anti Karen task force this next story is called just two small stories for the price of one yeah that's a smiley face quite a few years back I did work practice as an automation technician for a fairly large Factory in my hometown and from there to somewhat funny stories arose when I went in on my very first morning I was paired up with one of the automation technicians to go help in one of the machine halls because one of the machines wasn't working so he grabbed his tool bag and off we went we arrived the Machine Hall and got to the machine in question with the operator standing next to it well the lights were on in this machine but nothing works according to set operator so the automation technician pops open the control panel to reveal all the wires and relays and whatnots on the insides takes his multimeter and begins testing the various connections it should be noted that there's a complete set of schematics hanging on the inside of the door panel anyway everything seems to be in order so he asked me to hit the start up button just to see what's going on as I'm about to push the button I noticed there's a rather large dial with the words speed written above it so I of course asked the automation technician what speed I should put it on and out of the corner of my eye I noticed the operator facepalming whilst muttering I don't think there's any problems with the machine the automation technician just checkles yeah I think we're done here then and we went back nothing dramatic or anything just fun intermission time ok that story was insanely disappointing but it was kind of cute with the little smiley faces I kind of dig it but yeah it just goes to show you know make sure you check everything off your list before yes for help I guess alright continuing the other one though was a bit more dramatic the other story happened when I had been there for about two weeks we were called over to the research and development department because all their power had gone so I head out with another automation technician to find and fix the problems we get there and yep no power anywhere so we find the power box and see that the HP F I relay a safety master switch kind of thing had popped so we flick it back on and pop it goes again we then pop all of the fuses and turn the HP fi back on no problems there and begin plugging the fuses in again until the HP fi went out and thus we had found the culprit a quick glance over the schematics revealed that the wire set in question was of course the one that controlled the ceiling ventilators which were located on the top of the building no reason for this to be easy I inform all of the engineers in general R&D personnel that we'll be doing stuff with the power so don't turn on your machines or anything until we give the OK for it Automation technician handed me a radio and headed up and began checking the wires problem was fairly simple some debris in the ventilator shaft had damaged some of the wires which had enabled the ground wire to touch the face which in turn pops the HP fi automation technician calls me on the radio and asked me to turn on the power so I do he then does some work up there now that it's been confirmed that this was the issue it was mostly just clearing away the debris calling the boss to inform him what the problem was and whether we should just give it a patchwork or replace the wires entirely after maybe three to four minutes he calls me to turn the power back off so I do that maybe 10 seconds later one of the Rd folks comes running out screeching about why the power go out again because I turned it off but why'd you do that my machine was in the middle of its boot up because I was asked to rnd chump snaps the radio for my hand I don't know who's on the other line but get down here now Automation technician then hurries his butt down to where I'm at and looks at the R&D guy there you are turn the power off you're the one who told me that my machine don't take well to get in shuttle it shut down like that wait you turned your machine on why because the power was back on I thought you guys were done then ensued five minutes of absolute mayhem automation technician began shouting at the R&D guy who began shouting back it wasn't until the head of research and development called the boss from the automation technician department that they both calmed down but in the end research and development dude was reprimanded for having turned his equipment on when we told him not to suppose that something the writing was a little strange so I had to give the R&D guy a bit of a hick accent because that's the vibe I was getting from the dialogue again another good story but I just don't understand half the stuff they're talking about goodness all right guys this last story was called our last consultant disappeared can you take over I'm between jobs and picking up the occasional one or two days security project to keep busy one day a recruiter asks me to help them out with a problem that requires me to be on site a few states away nothing about this gig was straightforward but the rate was too good to ignore we placed a person there they did a bunch of evaluation work but then they went radio silent before explaining everything to decline and you want me to do what look over his work and present it to their CEO I'm not so sure flying out and back is some wear and tear they'll fly you I figured they'd take care of it no they have a plane they all fly you now I'm intrigued I agree I get a dump of internal documents in the draft report with a bunch of notes this is a client niche software as a service provider their customers are large entities with a penchant for vintage or unsupported software that runs everything the clients platform lets them still use them and work with less obsolete partners clients got a steady income stream but doesn't want to spend any money they don't have to their infrastructure is mostly sane but there's some things I want to call out I prep a few slides to present the findings a few days before the meeting I'm told that John the guy recruiter previously placed has surfaced from hiding and will be coming out with me where going to be presenting to clients new security engineer who is coming in to meet us I fly out with recruiter and John on a small jet to Klein city we'd be s for a while but John is a little wary of me we land grab a rental car and make our way to a hotel that's a rung up from the usual consultant storage unit an hour later I'm in the bar seeking sustenance reading a book sketching out the high points I want to make for an artificial environment such as a hotel bar it's busier than I'd expect for a weekday pretty much everyone at this bar is a business traveler smiles in corporate speak are abundant some flirting happens over open laptops it's fun to watch for some people this is a moment away from spouses and children perhaps what happens at the Marriott stays at the Marriott anyhow I'm trying to list the things that bother me in the clients environment just in case Jon doesn't convince the client or doesn't bring them up I'm scribbling in my notes when someone sits next to me it's a woman in her mid to late 20s I'm about to go back into my notes when I see what she's looking at two men who look like band members from a forgotten 80s hair metal band are in the bar looking around one is wearing a leather jacket in a few bright scarves the other looks like an extra from mclemore thrift shop video wearing leather pants with odd buckles a striped wolf shirt and some kind of stupid fake fur vest makeup his ensemble he's also wearing a top hat with goggles I try not to stare but somehow they seem familiar I smile to the woman sitting next to me subtly point to the Britney Fox tribute band and shrug my shoulders then go back to my notes I'm trying to order my issues with clients without coming across as negative they're hosting in a colocation facility in claiming all coalos audits as their own that'll be a conversation with their salespeople but they're not patching stuff or scanning for vulnerabilities I'll talk with a security engineer and see if they need suggestions maybe they need older systems to be compatible maybe we can work out some compromises over my drink I noticed that the two oddly dressed people have split up and are slowly working the room they're approaching women talking to them for a few minutes then move on when it seems they didn't get what they wanted maybe they're selling timeshares I see the recruiter and John make their way into the bar they make their way to a table and don't seem to notice me I assume recruiter wants to chastise John about becoming unavailable and discuss whose end my pay is coming out of I hear a voice behind me and notice the woman next to me looking at its owner my buddy and I have a bet going do you believe in magic spells excuse me I'm curious to see what it is my turn around and take a look and my jaw drops Ian to be continued alright guys sorry to leave you on a cliffhanger however that's all that's posted but I promise you guys this as soon as an update is posted I will be on that crap like baloney breath on a Karen don't forget to Like subscribe and hit that bell to never miss an episode

Channel: Daily Dose Of Reddit

Views: 71,744

Rating: 4.6950393 out of 5

Keywords: reddit, r/, subreddit, best of reddit, r/ top posts, top posts, top posts of all time, reddit top posts, 100% True stories, reddit true stories, top all time, reddit posts, reddit funny, daily dose of reddit, daily dose of internet

Id: 3LPa3y58n0k

Length: 18min 31sec (1111 seconds)

Published: Tue Sep 24 2019