The Plight Of Cobol Mainframe Programmers

all I want to do is read this article okay first off W mom by the way for those that don't know I believe Warren Buffer's mom also is like a Chad level Pearl programmer literally if if if Warren buffering mom were to quit we would have like an incredible issue happen in our society she literally holds up our world with pearl okay that's not a joke maybe one day I can get her on the stream Mama buffering to talk about just like what dude what she's done dude she is she is crazy look at this holding kids giving them bottles and also programming in Cobalt damn the day my mom has been working for one of the largest banks in EU since before I was born and I've always been fascinated by her line of work uh especially in these last years since I've become a programmer myself I've been uh asked to interview her plenty of times and finally I decided to do so some notes the banking programming world is complet completely different world than what most of us are used to and for the next couple hours after posting this I'll accept whatever questions you may have asked uh ask for her but keep in mind that some questions may not be answered due to security concerns I'll be posting this to Hacker News Reddit and we'll answer these questions over there as well we don't write this as a Q&A but more like I'm telling you her story all right so 1991 most of us playing battle toads at this point okay most of us this this is not even this is not even on a radar to program okay the the year she uh started internal training at nordia uh which back then was known as Nord banken the North Bank uh but changed uh name to nordia in 2001 during the training she had to take various tests most notably an IQ test to see if she had the intelligence to work within this field yo dude that's crazy that's crazy back then they're just like you can't be a programmer unless if you exceed some sort of like level of of of IQ damn test include a psychological checkup to make sure she had the psyche to handle that line of work and multitasking test where she failed horribly where she forgot where she got a score of 22 out of 100 she did very good on the other test and among the 16 available positions she managed to get one dude EU you crazy you crazy you crazy oh my goodness the position in question was an IBM Mainframe cobal programmer which to this day 25 years later still work uh as for the same bank the position is the most important one in the bank at least from a technical standpoint if let's say my Mo mother and everyone on her team would quit their job the bank would go under within a matter of weeks if they're lucky they have a rotation of people on her team being available 24/7 I remember when I was younger and she had to take a taxi to work in the middle of the night on a Sunday to fix a deadlock problem the mom is literally Chad Chad level here uh Cobalt is not a fancy programming language really is that what it stands for I didn't realize that that's what it stands for you know I did a little bit of cobalt myself just a little bit though like your functional hasal or concurrent goang it's an imperative procedural language in since 2002 object-oriented so it looks like every every language if you live long enough you truly do become a villain there's nothing wrong with the language itself really uh the problem is that it barely uh that barely anyone knows it so I I feel like that I feel like I disagree just in general with that statement in the sense that I do think there is a problem with the language just saying just saying uh but it's incredible that I mean it it is just an old language so it has a lot of the older paradigms it's it's very it's very interesting right at least not in the context of Mainframe programming okay that makes sense my mother is the next youngest person on her team and she's born in 1964 and the youngest person being 2 years younger sends almost all the largest banks in the world runs on IBM Mainframe with Cobalt as the primary programming language this is a global issue the smaller Banks however are better off which usually run something like Java without mainframes yeah they get to move a little bit faster they get they get to be updated they haven't been around since Legacy this is such an amazing Testament to there's this like there's this odd duality of programming which is that whatever you write to day will likely be thrown away in 6 months and not used at all or it will stick around for decades until someone rewrites the entire system it's like such a gamble what one are you doing do you even know that you are writing code that could be around for a long time and so that's why a lot of people you know that's why I'm so frustrated with a lot of modern uh web like ideas in the sense that it constantly assumes that you that we work in this in this field that is available to change right that oh we're just out here we're just throwing stuff out here and if if if something new comes along it's better and we should all use the better new thing right but what you really get is you just get this weird amalgamation of every person's hot soup dour idea throughout from 2013 to 2024 and so like if you look at some of the code bases at Netflix it's literally class components of function components to this to that to ramda just ra raming my code base with their functional curring programming back swinging the other direction it's just like you could just see what was Vogue in the year that it was written inside the code and it's just so wild to see that because it is it is genuinely a it genuinely can be very painful and so maybe there is something in some sense nice about all this cobal programming is that at least it's like you know what you're writing and you know how long it lives I don't know it's just it's just an it's just an oddity it's just an oddity oh no problem hey I appreciate that oh nice hey we hey we raided this guy yesterday and he got put over the affiliate threshold well let's go nice job nice job all right my mother used to ask me if I wanted to learn but since working with more fancier stuff like post redis boo h reddis f rdus F garganta or whatever the name of the company is in the background Noe cryistal damn the man's porn out Cristal PHP imagine imagine listing out things fancier than the programming language your mom uses and PHP makes the list okay PHP is not known as the wine and dine language okay like Cristal I mean now you're you're way out there okay Christal is like you're programming life on the edge postgres your standard good database redis a horrible company that has taken advantage of Open Source and rug pulled people node I mean some would call this Cutting Edge than PHP okay hey I'm not saying PHP is bad it's just typically you wouldn't call this thing a nicer like this is not this is not where you say the great things happen Okay uh but to be completely fair this is also this article is written in 2016 which I would argue that 2016 PHP is actually the the worst PHP of all time because it's it's not the modern amazing LVL experience and you're mostly doing classes from 5.4 I know there was newer versions of PHP at that point but you were pretty much stuck in class hell at that point it was pretty bad yeah it was it was pretty bad now PHP is really good you know PHP is had a quite the Renaissance I I must say I'm I'm impressed with the language staying relevant that long I've always answered never ever I am still very interested in what she uh what she does but these types of systems gives off the probably worst enterpris feeling feeling that you can imagine which I'd like to avoid I can only imagine the fat paycheck a 20-year-old Mainframe programmer would get though because your age in this case would be invaluable such also a crazy notion that somehow like this is kind of like the unionization of programming is that kind of what we're seeing here is this what you guys do over in EU that age makes you more valuable I mean I would say age makes you more valuable because you make better decisions but it doesn't it's not a guarantee there's no guarantees here actual gold mine I I would assume the reason why it's a gold mine has nothing to do with your age it's the fact that you're willing to learn cobal get good at cobal and then do it and work on Legacy code that has the single greatest requirement for you to be good at it age would be important because the the youngest cobal Dev is 63 who said that the age would be valuable because the youngest Cobalt Dev is 63 years old all right databases their primary database is called IMS literally never heard of it um International missile system which is a higher archical database built by IBM for the Apollo program okay so it might actually be involving missiles here internally they call it dl1 which is short for database language one I've never programmed in the first of a language what's this language well it's database language one we made it we we made a place to store data and then we thought you know it' be cool is to throw some language at it to get the data back out it's written in Latin but don't worry it's not like 2,000 year old Latin okay we brought in some of the modern niceties of Latin from like uh 100 you know ad so it's it's pretty nice uh they're trying to migrate uh to migrate to db2 clearly better which is a relational database that speaks normal squeal but considering the sheer volume of data that nordia is storing this task uh that is going to take years it's not as simple as just moving the data from IMS to db2 they ALS have to update their modules to load and save data from db2 instead of IMS and they have thousands of modules I don't even see like in this case I don't even know what a module means right in my head a module is like a file a file that has an API for you to be able to get set and create data from a module in this case I don't know what this means okay many of which were developed by programmers that have either passed away or have retired these were the these were the Pioneer Cobalt programmers back in their day okay bring out the a force and it's just like a bunch of 80-year-old people I'm coming back here to show you little swell fellas out again to db2 you know um this is Legacy code this is so Legacy that the person who wrote it died of old age okay each transaction is stored in db2 they're avoiding writing to IMs as much as possible only reading data from it until their newly purchased system is fully integrated and they can start storing data there IMS is extremely old and very slow for some tasks okay uh searching for data can take hours ha and here we are arguing that my squeal delivers 2 Ms better query speed than post that's a bit ironic I don't know if that's the proper use of the term ironic but I would I will accept it um they uh they also use a flat file structure for a lot of tasks which themselves have different flavors that IMS supports one such example is the GM operator which has forced my mother to take a taxi into work many times due to the modules working on the same GM file at the same time producing deadlocks crazy I don't even know what that means what the hell is a GM file then why is your mom taking taxis to go fix GMS on the internet for banks at 2 in the morning that's dude again sometimes it's easy to forget how hard things are I'm happy you know like that's why I I've never I don't believe I'm someone who looks into the past and thinks that those were better programming days and looks at modern stuff and think it's somehow worse it's just mostly different you know like the nice part is you don't have to go fix Deadlocks at 2: in the morning cuz you had a lock file try to access a try to write to a lock file at the same time as another one and now you're effed but instead you just have to know like 19 different technologies that you Plum together it's like different it's different let's talk about the size of their database for a moment db2 is only storing data about transactions and each transaction is different in the size depending on what type of account the transaction goes to SL from private accounts like my personal bank account is orders of magnitude more simple than a bank account used by businesses why uh each transaction is somewhere between 2kb to 500 bytes uh so let's say the average is 1 kilobyte that this confuses me I mean at the end of the day isn't every account just a pointer to something uh at the moment their db2 database holds 11 billion transactions and the law requires them to save each transaction for 10 years having uh settled for 11 years at the moment transactions are only 7 years old and the amount of transaction is estimated to grow roughly 5 to 8% each year until they hit the 11-year Mark where they can nuke transactions that are older than 11 years so at the moment the db2 database hold about 10 terab of data and that data is only transactions in four years it'll be somewhere between 13 to 14 terabyt of data okay okay okay that's not a lot you got to remember that they're using they're using flat file storage with with gasm locks okay it sounds it sounds like a lot of data when it comes to that okay you got to remember that this isn't this this is not some nice distributed postgres database okay you're working in the past the Deep past okay so whenever someone's just like oh that's easy you like you got to remember the time in which this was written okay the time in which they're trying to like maintain here uh there is Al let's see they also store some data on tapes where their program tells a machine to load and save data from a specific tape which is pretty cool cool Cool's a word cool cool is definitely a word to describe this C is also another word IDE everyone has some form of ID or text editors right well yes they do their ID is called ispf which is like an entire OS ispf can be extended and the part we would call an IDE is an extension of ispf which is called Endeavor are they describing emac right now is this emac ispf is directly connected to the Mainframe there is no such thing as local development environment here o proprietary emac damn this takes terminal velopment to a whole new level batch roughly 80% of their systems are batch jobs these jobs these are jobs that run uh that runs at certain time or interval doing some processing on their data and sends data to other Banks agencies etc for example when I buy a can of Coke the money is withdrawn from my account balance however the money is not actually transferred anywhere until one of their batch jobs does so these jobs are usually executed during the night which is usually why it takes a day before transaction between banks are completed transactions to s to the same bank are usually instant because it executes immediately ah okay this makes sense I've seen this okay batch jobs are working with data uh data sets on a gigabyte sometimes terabyte level and can in some cases take hours to complete I'd love to see how IBM's main frames all of the sudden starts working at full capacity the second the clock turns 00 in their database centers that'd be pretty cool it would be pretty cool the problems databases are facing uh are the problems that banks are facing banks that run on Main frames have quite a lot of issues that they need to take care of but unfortunately time is scarce programmers are getting older it's like the only place you can really say that I would say that in the last 10 years our industry has gotten significantly younger on average but here this is the only one where like time is scares people people we don't got a lot of time left in y'all yall are about to retire or die we need you programming fast people not many wants to learn and the time uh before a new employee can stand on their own two feet is 2 to three years and even then there is a lot of gray areas for them oh my oh my there are programmers that are decades old that nobody even knows what they do and the person who wrote it is long gone many banks are trying to purchase shiny new systems that replace what they currently have while almost all of their programmers are saying this is not going to work you cannot replace this monster that we have uh have here you know when I read these things it just makes me real realize how much I appreciate anything I do today anything right boom or take I you know like real talk a fool looks at a problem that other people have and they're solving in a specific way and say oh I could do it better an experienced programmer knows that almost every time you've ever solved a problem better by redoing it you've started off by saying look at how smart I am and then halfway through you realize oh I can kind of see why they did this one shortcut because you know yeah there's this one edge condition I was just unaware of and then by the end of it you've literally Rewritten line forline what they wrote and you go dang it it turns out I am equally as bad I've done that I've literally done that and so when someone says you can't just replace this program here with something new there is probably so much that you have no clue about how cobal Works what they're doing how they're lying on stuff how things are stored how they can be retrieved all of that that it's just like what you're saying is that when you say boom or take what you're saying is I know better than them you probably don't just rewrite I can tell by your last couple comments that you're probably a pretty young fella here right you you hit him with an L take on on the Boomers and then now you're saying just rewrite it it's just not how it works it's just not how a lot of these things work you know it's a you can't just rewrite this Devon will do it easily you should convince a bank you should convince a bank to do that I think you can make millions you're kind of a young fell in my eyes uh when are you let's see when you're my age the world makes more sense aren't you one year older than me lithium aren't you 38 I turned 38 this year you're like 6 months older than me bro maybe I am old man yelling at the clouds but not all old man yelling at the clouds is bad not going to lie I'm probably less than a year older than you the reason their systems have become so huge is partially because they've made the mistake early on to tailor their own systems toward other systems a great example here is how we build restful Services where programs that want to get data from us have to tailor their systems to match ours that is not the case with some banks instead they're sending data to other systems such as tax agencies which may at any point update their system which in turn breaks the current system the bank the bank are using to send data I've heard about this this right here is crazy we've heard about this stuff where they just like they just decide they just decide that it's it's time hold on there there was a there was a Banning that just went on oh yeah okay yeah yeah yeah yeah don't worry don't worry about it thank you flip take that out all right banking systems are extremely Advanced a personal bank account differs uh a lot from business bank account and there are at least 50 different type uh different types of bank accounts for each of them in nordia case they also have the Swedish government accounts which are different from both personal and business accounts I think they have a Finnish government accounts and maybe a portion of denmarks as well which differs too man imagine you used some external system that just broke you with no telling you you have just constant government regulation for every last way you're supposed to do things and people are sitting there going dude why did you use cobal why did why haven't you just Rewritten it like dude this is this is such a hard problem this is just truly a hard problem it's going to be very interesting to hear what nordia and other banks will do in the upcoming years and what new tech they'll be implementing on I may have forgot to ask something so if you have any questions go ahead and ask I'll be forwarding them and writing down uh her response in the next couple hours just absolutely awesome this was so cool so we got some q&as going on here let's see what let's see what we got here Q&A why did you choose to work at UH with IBM Mainframe Cobalt programming language I've always wanted to work with computers but I wasn't aiming for a specific job I saw an ad in a paper different time people this is the different time she didn't even use LinkedIn okay imagine not posting inspirational LinkedIn memes to get hired for your job like if you're not out there talking about empathy on LinkedIn like are you even employed uh and applied and got the job I had a bit of computer background before I applied though she she what is the worst thing you've uh seen on a day my cooworker forgot to add a DOT to the end of a statement by the way a DOT at the end of the statement in Cobalt is like a semicolon in in C at the end of a statement for a module in the most critical part of our system which we'll called the cash register it is the part that handles all the money the result was that the entire bank was down for 16 hours straight due to a module continuing executing statements when it actually should have stopped after that statement it basically overloaded our systems a self dos of sorts it's just so incredible it's so incredible what do you think will happen in the future for banks that run on the same infrastructure as nordia most banks have acknowledged that uh we need to replace the older main frames with something more modern nordia has bought a new system which is promised to replace the current one within four years but nordia takes care of multiple countries in a more reasonable number is four years per country so 16 years in total Banks and finance uh that haven't acknowledged this is going to have a very rough time ahead of them I still think we'll live on IBM infrastructure though like I will be retired probably before IBM main frames are out what were the challenges you faced as a female programmer starting in the '90s no problems at all I've got a couple female co-workers most of them are men it doesn't bother me that much Chad level she needs to work in the Silicon Valley to experience it she doesn't understand uh let's say you've been working for the same entity and possibly the same codebase for more than 20 years dude based mom this mom is [ __ ] based none what have you faced none I've been working on cobal you got to realize the horrors that she faces every single day which is cobal and a missing period doses yourself okay imagine the difficulty that has to go on at this job crazy all right you've been working on the same entity and possibly the same code base for more than 20 years yeah I know Diamond thick skin here does it ever get old yeah it definitely gets tiring after a while like most other lines of work becomes yeah but during my time I've uh built many completely new systems for various things in finance and that has always been fun unfortunately from now on we will never build anything new only maintain what we currently have and wait for newer systems we've purchased to replace the current ones yeah maintenance never it's never been that great how scary is it the right code for a bank it is very scary especially like imagine like by the way I don't even know if she's going to say this here let's read this and then we'll we'll talk about it uh especially when we push changes to production which happens on Sundays whenever we push a new change to production huge parts of the system has to be taken offline one of those parts is our entire IMS During the period anything can happen but we have very robust test environment so it usually goes through smoothly also remember that during the beginning when this was all being written think about how little push at all there was for automated testing right like nothing nothing right and so like they had to like come up with testing they didn't even have tddd they didn't even just have t okay there's there's not even it's not even tdd it's just there's no te okay they got none of it Panic driven development PDD man have you uh have you caused any huge mistakes for the bank definitely have a quite a large mistake I did back in 97 when my youngest son uh that's me the writer just started preschool and my parental leave was over we have a system for saving money for retirement those type of bank accounts wasn't locked back then and this law says you cannot withdraw money from that account before your 5 years old since the account wasn't locked it was possible to withdraw money if you had the bank account number so the solution was simple not give the customer their bank account number let's go that's the way we do it I managed to screw up by modifying a module that added their bank account number to a mail that was sent out to our customers so what happened was customers started withdrawing money which was not taxed yet from the retirement savings account before they are allowed to let's go this triggered a huge inspection the Swedish government stepped in in the financial in inspection the media were all over it that was me do we have any sweds in the chat right now that remember this we got any sweds do we got any sweds here does anybody actually remember that because that is so hilarious way to go B born I know do that guy sweed don't recall here okay yep right here really damn I'm swed uh too young for that yeah any sweet Boomers that's a high stakes programming dude that is high stakes ass programming that's crazy Sweden president let's go throwback Thursday to 1997 yeah yes but I don't remember it okay so most people don't remember it but we could be doing what's your working environment like we've recently moved to a more hip location we used to have a personal desk but now we have this pick whatever spot is available open area I just like it a lot beautiful I cannot believe this thing only has one like and one comment this is probably from my Discord too or from my Discord from my Reddit consume probably who who's consume who did it who dropped this one who dropped this one all right here we got a great article holy cow this was so good that was so good I worked at a newspaper and we rewrote a pearl subscription management system that had been in production for like seven years if I started over I would have just left it in Pearl typically that's how most do not come soon that's how most rewrites should be done most rewrites should be done like I mean okay I'm going to give a super hot take here are you guys ready for my super ultra super duper just just just flaming hot take um I think modern UI has caused Progressive refactoring to be impossible like if you're on react and you want to move to say spelt or if you're on react and you want to move to view or angular that's hard okay you're going to be sending down many you're going to be sending down many a client Library just to get something going it's like a very unique vendor lock in like when I look at uh when I look at like when I was you know when I was at Netflix X Netflix engineer by the way which technically I'm not xnfx engineer till tomorrow at 5:00 P p.m but X Netflix engineer here if you look at the C++ code base you can watch it go from c99 to like c17 like you can watch the transition they progressively refactor some places stay old some places get newer some places get upgraded right you can kind of progressively refactor and make things better and better over time we had curl at one point we don't have curl anymore like right there's been like this whole thing that has happened all the way from the beginning which is kind of amazing to see because these refactors to make the system better and more resilient over time was able to keep on keeping on you know what I mean where it's extremely hard to do that elsewhere you know yeah yeah yeah yeah yeah I have a video on and everything oh no you don't you don't do this it took me six months to transfer our react webpack app to react Vite app that's just a build tool I mean at the end of the day it's literally just a build tool it's meant to take an interpreted language and transfer it into an interpreted language that can be executed in the browser I just want you to think about that for a second that's what that's what a JavaScript build system does is it takes a bunch of little interpreted languages and morphs them into a bunch of little interpreted languages that are then assembled in the correct order and executed I honestly I don't think Netflix could get off of webpack in any reasonable amount of time I honestly think it would take them six plus months real talk it would take them an enormous amount of time to do that anyways okay hey the name is that dude this this was amazing by the way this was a great article Che to that Mom way to crush ass and just destroy and do this and do this for years and literally program some of the most highest Stakes thing ever holding kids being a ba Tom Tom your mom's your mom's a saint your mom is a saint

Channel: ThePrimeTime

Views: 197,960

Keywords: programming, software engineer, software engineering, developer, web design, web development, programmer humor

Id: qNytWcW38us

Length: 29min 15sec (1755 seconds)

Published: Tue Apr 23 2024