What is SciPy? | Travis Oliphant and Lex Fridman

you created sci-pi over two decades ago now yes right yes i'd love to talk about sci-fi sci-fi was really my baby what is it uh what was its goal what is its goal how does it work yeah fantastic so scipy was effectively here i'm using python to do stuff that i previously used matlab to use and i was using numeric which is an array library that made a lot of it possible but there's things that were missing like i didn't have an ordinary differential equation solver i could just call right i didn't have integration hey i wanted to integrate this function okay well i don't have just a function i can call to do that these are things i remember being critical things that i was missing optimization i just want to pass a function to an optimizer and have it tell me what the optimum value is uh those are things like well why don't we just write a library that adds these tools and i started a post on the main list and they're previously been you know people have discussed i remember conrad henson saying wouldn't it be great if we had this optimizer library or david ash would say this stuff and and i'm you know i'm a ambitious uh and this is the wrong word and eager and uh probably more time than sense i was you know poor graduate student uh my wife thinks i'm working on my phd and i am but part of a phd that i loved was the fact that it's exploratory you're not just you know taking orders fulfilling a list of things to do you're trying to figure out what to do and so i thought well you know i'm writing tools for my own use and a phd so i'll just start this project and so in 99 98 was when i first started to write libraries for python especially when i fell in love with python 98 i thought well there's just a few things missing like oh i need a reader to read dicom files that was in medical imaging.com was a format that i want to be able to load that into python okay how do i write a reader for that so i wrote something called it was an i o package right and that was my very first extension module which is c so i wrote c code to extend python so that the pos in python i could write things more easily that that combination kind of hooked me it was the idea that i could here's this powerful tool i can use as a scripting language in a high level language to think about but that i can extend easily easily in this in c that easily for me because i knew enough c and then guido had written a length i mean the only the hard part of extending python was something called the way memory management networks and you have to reference counting and so there's there's a tracking of reference count you have to do manually and if you don't you have you have memory leaks and uh so that's hard plus then c you know it's just much more you have to put more effort into it it's not just i have to now think about pointers and have to think about stuff that is different i have to kind of you're like putting a new cartridge in your brain like you're okay i'm thinking about mri now i'm thinking about programming and there are distinct modules you end up having to think about so it's harder when i was just in python i could just think about mri and high level writing um but i could do that and that kind of i liked it i found that to be enjoyable and fun and so i ended up oh well let me just add a bunch of stuff to python to do integration well and the cool thing is is that you know the power of the internet i just looking around and i found oh there's this net live which has hundreds of 4chan routines that people are written in the 60s and the 70s and the 80s in fortran 77 fortunately it wasn't for trends 16. so they've been imported to fortren 77 and 1477 is actually a really great language fortunate 90 probably is my favorite 4chan because it's also it's got complex numbers got a raise and it's pretty high level now the problem with it is you'd never want to write a program in fortune 90 or fortune 77 but it's totally fine to write a subroutine in right and so and then 4chan kind of got a little off course when they tried to compete with c plus plus but at the time i just want libraries to do something like oh here's an order different equation here's integration here's run cut integration already done i don't have to think about that algorithm and you could but it's nice to have somebody who's already done one and tested it and so i sort of started this journey in 98 really looked back at the mailing list there's sort of this this productive era of me writing an extension module to connect wrench cut integration to python and making an ordinary digital equation solver and then releasing that as a package so we could call od pack i think i called it then quad pack and i just made these packages eventually that became multi-pack because they're originally modular you can install them separately but a massive problem in python was actually just getting your stuff installed at the time releasing software for me like today it's people think what does that mean well then it meant some poorly written web page i had some bad web page up and it put a tarball just a gzip tar ball of source code that was the release but okay can we just stand that because that the community aspect of creating the package and sharing that yes that's rare that to have to both have at that time so like it was pretty early yeah so well not not rare maybe maybe you can uh correct me on this but it seems like in the scientific community so many people you were basically solving the problems you needed to solve to process the particular application uh the data that you need and to also have the mind that i'm going to make this usable for others that's um i would say i was inspired i'd been inspired by linux i've been inspired by you know linus linus and him making his code available and i was starting to use linux the time and i went this is cool so i'd kind of been previously primed that way and generally i was i was into science because i liked the sharing notion i like the idea of hey let's if collectively we build knowledge and share it we can all be better off okay so you weren't energized by that so it's energizability already yeah right and i can't deny that i was i'm sort of had this very i liked that part of science that part of sharing and then all of a sudden oh wait here's something and here's something i could do and then i slowly over years learned how to share better so that you could actually engage more people faster one of the key things was actually giving people a binary they could install right so that wasn't just your source code good luck compile this and then it's compiled ready to install you just you know so in fact a lot of the journey from 98 even through 2012 when i started anaconda was about that like it's why uh you know it's really the key as to why a scientist with dreams of doing mri research ended up starting a software company that installs the software i work with a few folks now that don't program like on the creative side the video side the audio side and because my whole life is running on scripts i have to try to get them i'm have now the task of teaching them how to do python enough yeah to run the scripts and so i've been actually facing this whether it's on the condor some with the task of how do i minimally explain basically to my mom how to write a python script and it's an interesting challenge it's a to-do item for me to figure out like what is the minimal amount of information i have to teach what are the tools you use that one you enjoy it to your effect of it and they're related to related questions and and then the debugging like the the iterative process of running the script to figure out what the error is maybe even for some people to do the fix yourself yeah so do you compile it do you destroy like how do you distribute that code to them and it's interesting because i think it's exactly what you're talking about if you increase the circle of empathy that the circle of people that are able to use your programs you increase it its like effectiveness and its power and so yeah you have to think you know can i write scripts can i write programs that can be used by biomedical engineers by all kinds of people that don't know programming and actually maybe plan to see have them catch the bug of programming so that they start on their journey that's a huge responsibility and ultimately has to do with the amazon one-click buy like how how frictionless can you make the early steps frictionless is actually really key to growing any community is every any friction point you're just going to lose you're going to lose some people yeah right now sometimes you may want to intentionally do that if you're early enough on you need a lot of help you need people who have the skills you might actually it's helpful you don't necessarily have too much too many users as opposed to contributors if the co if you're early on anyway there's uh uh sci-fi started in 98 but it really emerged as this collection of modules that i was just putting on the net people were downloading and they you know no i don't know i think i got 100 users right by the end of that year but there but the fact that i got 100 users and more than that people started to email me with fixes like and that was actually intoxicating right that was the that was the you know here i'm writing papers and i'm giving conferences and i get people would say hello but yeah good job but mostly it was you're reviewed with it it's competitive yeah right you published a paper and people were like oh it wasn't my paper you know i was starting to see that sense of academic life where it was so much i thought there was a cooperative effort but it sounds like we're here just to one-up each other right and you know it's not it's not true across the board but a lot of that's there but here in this world i was getting responses from people all over the world uh you know i remember pierre peterson in estonia right was one of the first people and he sent me back this make file because the first thing is yeah your build thing stinks and here's a better make file now it is a complex make i don't think i never understood that makefile actually but it worked and it did a lot more and so then thanks this is cool and that was my first kind of engagement with community development but you know the process was he sent me a patch file i had to upload a new tar ball and i just found i really loved that and the style back then was here's a main list it was very it wasn't as it certainly were the tools that are available today it was very early on but i really started that's the whole year i i think i did about seven packages that year right and then by the end of the year i collected them into a thing called multi-pack so 99 there was this thing called multi-pack and that's when a high school student knows a high school student at the time guy named robert kern took that package and made a windows installer right and then of course a massive increase of usage so by the way most of this development was under linux yes yes it was on linux i was a linux developer doing it on munix box i mean at the time i was actually getting into i had a new hard drive did some kernel programming to to make the hard drive work i mean not programming but modification to the kernel so i could actually hard drive working i i love that aspect of it i was also in you know at school i was building a cluster i took mac computers like uh and you put yellow dog linux on them they were at the mayo clinic they were just all these macs that were older they were just getting rid of and so i kind of got permission to go grab them together i put about 24 of them together in a cluster and a cabinet and put yellow dog linux on them all and i wrote a c plus program to do mri simulation that was what i was doing at the same time for my day job so to speak so i was loving the whole process at the same time i was oh i need a ordinary differential equation that's why ordinary difference equations were key was because that's the heart of a block equation for simulated mri is a ode solver and so that's um but i actually did that it doesn't happen at the same time that's why it kind of what you're working on and what you're interested in they're coinciding i was definitely scratching my own itch in terms of building stuff and uh which helped in the sense that i was using it for me so at least had one user yeah i had one person who's like well i know this is better i like this interface better and i had the experience of matlab to guide some of what those apis might look like but you know you're just doing yourself you're building all this stuff but with the windows installer it was the first time i realized oh yeah the binary installer really helps people and so that led to spending more time on that side of things so around 2000 so i graduated my phd in 2000 end of year 2000. so 99 doing a lot of work there 98 doing a lot of work there 99 kind of spending more time my phd you know helping people use the tools thinking about what i want to go from here there was a company there's a guy actually eric jones and travis vott they were two friends who founded a company called nthot it's here in austin still here and they eric contacted me at the time when i was a i was a graduate student still and he said hey why don't you come down we want to build a company you know we want we're thinking of you know a scientific company and we want to take what you're doing and kind of add it to some stuff that he'd done he'd written some tools and then pierre peterson had done ftp let's come together and build pull this all together and call it sci-pi so that's the origin of the scipy brand it came from you know multi-pack and a whole bunch of modules i'd written plus a few things from some other folks and then pulled together in a single installer sci-fi was really a distribution of python mastering as a library you

Channel: Lex Clips

Views: 8,783

Keywords: ai, ai clips, ai podcast, ai podcast clips, artificial intelligence, artificial intelligence podcast, computer science, consciousness, deep learning, einstein, elon musk, engineering, friedman, joe rogan, lex ai, lex clips, lex fridman, lex fridman podcast, lex friedman, lex mit, lex podcast, machine learning, math, math podcast, mathematics, mit ai, philosophy, physics, physics podcast, science, tech, tech podcast, technology, travis oliphant, turing

Id: lff9SY5ssDY

Length: 13min 8sec (788 seconds)

Published: Fri Oct 01 2021