[NOISE] Okay. Hey, everyone. Um, morning. Welcome to CS230, Deep Learning. Um, so many of you know that, um, Deep Learning these days is the latest hardest area of computer science or AI. Uh, arguably, Deep Learning is the latest hardest area of, you know, all of human activity, uh, uh, maybe. Um, but this is called CS230 Deep Learning where we hope that we can help you understand the state of the art and become experts at building and applying Deep Learning systems. Um, unlike many Stanford classes, this class will be more interactive than, than others, uh, because in this class we offer in the flipped classroom format where we'll ask you to watch a lot of the videos at home, uh, uh, a lot of the deeplearning.AI content hosted on Coursera, thus preserving the classroom in discussion section time for much deeper discussions. Um, so to get started, let me, let me first, uh, introduce our teaching team. So, the co-instructors are Kian Katanforosh who are actually, uh, uh, one of the co-creators of the, uh, Deep Learning specialization, the deeplearning.ai content that we're using in this class. Um, and the, uh, the rest of the teaching team, uh, Swati Dubei is the class coordinator and she has been working with me and others on coordinating, uh, I guess CS230 and also CS229 and CS229A to, to make all of these classes run well and let you have a relatively smooth, you know, uh, uh, experience. Younes Mourri is the course adviser and he's also worked closely with Kian and me in creating a lot of the online contents that you use and Younes is also, um, head TA, uh, for CS229A, which some of you may also be taking. And then, we have two co-head TA's, Aarti Bagul, who's worked on machine learning research for a long time and Abhijeet who's still traveling back I think, and also a large team of TA's that, uh, I think about half of our TA's in CS230 had previously TA this course and, uh, their expertise spans everything from applying machine learning to problems in health care, applying machine learning or applying deep learning to problems in, um, uh, in robotics, to problems in computational biology to problems in, um, so, so I hope that as you work on your projects this quarter, as CS230 you'll be able to, um, get a lot of great advice, uh, uh, and help and mentorship from all of the TA's as well. Um, so the plan for today is, uh, I was going to spend maybe the, uh, little bit of time, uh, sharing with you what's happening in Deep Learning, why, you know, why Deep Learning is taking off and how this might affect your careers. Uh, and then in the second half, I have, Kian will takeover and talk a bit more about the projects you work on in this class and not just the final term project, but you know, the little machine translation system you build, the face recognition system you build, your art generation system you built and all of the, uh, many pretty cool machine learning and deep learning applications that you get to build throughout the course of this quarter, uh, and, and also share a view the, the, the detailed logistics for the plan for, for the class. Okay? Um, so I think that's, um, uh, let's see. All right. I'm gonna just use the whiteboard for this part. [NOISE] Cool. So, um, [NOISE] you know, deep learning, right, you know, it seems like the media still can't stop talking about it. Uh, and, um, it turns out that a lot of the ideas of deep learning happened around for several decades, right? The, the basic ideas of deep learning happened around for decades. So, why is deep learning suddenly taking off now? Why is it, quote, Coming out of nowhere? or what- whatever people say. Um, I think that the main reason that deep learning has been taking off, uh, and why, you know, suddenly all of you hopefully will be the, do really, um, powerful things with it much more effectively than two or three years ago is the following. Um, for a lot of, over the last couple of decades with the digitalization of society, we've just collected more and more data. Uh, so for example, all of us spend a lot more time on our computers and smartphones now and whenever you do things on a phone, uh, you know, that creates data, right? Uh, uh, and, and, um, and, um, what used to be represented through pieces of paper is now much more likely to be a digital record as well. So you're, if you go take an X-ray, uh, as at, at least in the United States, uh, less than some other kind, in developing economies, but basically the United States, there's a much higher chance now that your X-ray in the hospital is a digital image rather than physical piece of film. Or, um, if you order a new marker, right? There's a much higher chance that the fact that you ordered a marker, you know, off a website is now represented as a digital record compared to, uh, 10 years ago when the state of the global supply chain. Actually if you order, uh, if you order 10,000 markers, um, there's a much higher chance, you know, 10 years ago that the fact that you place that order was stored on a piece of paper. There's someone scribbled saying, Hey, ship 10,000 markers to Stanford. But now there's much more likely to be a digital record. And so the fact that, um, so many pieces of paper are now digital has created data. And, uh, for a lot of application areas, the amount of data has sort of, you know, exploded over the last 20 years. But what we found was that, um, if you look at more traditional learning algorithms, traditional machine learning algorithms, the performance of most of them would plateau, uh, even as you feed it more and more data. So, by traditional learning algorithms, I mean logistic regressions, support vector machines, you know, maybe decision trees depending on influential details, uh, and it was as if our older learning algorithms didn't know what to do if all the data you can now feed it. Um, but what we start to define several years ago, was if you train a small neural network, right? Its performance may look like that. If you train a medium, neural net, its performance may look like that. And if you train a very large neural net, you know, the performance kinda keeps on getting better and better, uh, up to some, usually up to some theoretical limit called base error rate, which you'll learn about later this quarter, but where performance can never exceed a 100 percent, but, but sometimes, uh, uh, sometimes there's some seeding of the performance. But onset, we've been able to measure on many, many problems, uh, with not yet, I think that across machine learning and deep learning broadly, I think we've not yet hit the limits of scale and by scale I mean, um, the amount of data you can throw at the problem that is still useful for the problem as well as, um, the size of the neural networks. And I think, um, you know, GPU computing, uh, uh, was a large part of how we were able to go from training small to medium to now training very large neural networks. And once upon a time, I think, um, you know, the first, actually I think a lot of the early work on, uh, training neural networks on GPUs, which is done here at Stanford, right? Crew there using cruder to the train neural networks. But, um, what use to be, you know, one thing, one lessons we've learned over and over in computing is that, uh, what yesterday's supercomputer is today's, uh, uh, you know, processor on your, on your smartwatch, right? And so what used to be an amount of computation that was accessible only to, you know, large research labs in Stanford. They could spend a $100,000 on GPUs. Today, you could rent that on a- on a cloud relatively inexpensively. And so the availability of relatively large neural network training capabilities has allowed really students, really almost everyone. Many people, not, not many, many people, um, to have enough access to computational power to train what, uh, large enough neural networks to drive very high acc- levels of accuracy for a lot of applications, right? Um, and it turns out that, uh, uh, if you look broadly across AI, you know, I, I think the mass media, right? Newspapers, you know, reporters use the term AI. I think, uh, uh, within, within academia or within the industry, we tend to say machine learning and deep learning. Uh, uh, but i- if you look broadly across AI, it turns out that AI, um, has many, many tools that's beyond machine learning, that's even beyond deep learning. And if any of you take, you know, CS221, right, Stanford's AI class, great class, uh, uh, you'll learn about a lot of these other tools of AI. But the reason that deep learning is so valuable today is that if you look across many of the tools of AI, let's say [NOISE], you know, there's a deep learning [NOISE] /machine learning, um, and, and, and again, some of you know, uh, neural networks and deep learning mean almost exactly the same thing, right? It's just that, um, as, you know, as we start to see deep learning rise over the last several years, we found that, uh, deep learning was just a much more, um, attractive brand. And so, you know, and so, so that's the brand that took off. Uh, uh, but, um, if you look at, if you, if you take an AI class, you look broadly across the portfolio of tools you have in AI. Um, I think that, you know, our often use deep learning and machine learning, um, are sometimes also use their probabilistic graphical model, right? We shall learn about it in CS- CS228, also a great class. Um, sometimes I use a planning algorithm, you know, when I'm working on self-driving car, right? You need a motion planning algorithm. You need various planning algorithms. Uh, sometimes I use the search algorithm. Uh, sometimes I use knowledge representation, it's very easy. This one is the technologies, uh, specifically knowledge graphs is one of the technologies that is widely used in industry, but I think often under appreciated in our academia. Um, uh, i- if you do a web search, you know, web search engine, pulls up a hotel and then there's a room prices and where there's WiFi, where there's swimming pool, that's actually a knowledge graph or a knowledge representation knowledge graph, but it's, uh, so it's actually used by many companies. These large databases, but this is, this is actually maybe under-appreciated in academia, um, or sometimes even game theory, [NOISE] right? So, if you learn about AI, there's a very large portfolio of many different tools you will see. But what has happened over the last [NOISE] several years [NOISE] is, um, [NOISE] if you go to a conference on probabilistic graphical models, right? [NOISE] If this is time and this is a performance A [NOISE]. You'll see that, you know, every year, um, probabilistic graphical models work a little bit better than the year before. In, in regard to the UAI conference, Uncertainty in AI conference. Maybe the- one of the leading conferences, maybe the leading one, I'm not sure on PGM's, you'll see there every year, you know, researchers publish papers that are better than the year before. The state- the, the, the field is steadily marching forward. Um, same for planning, if we go with triple AI or something, you'll see, you know, a few. There's advancing, search algorithms are getting better, uh, knowledge representation algorithms getting better, game theory algorithms are getting better. And so the, the, the field of AI marches forward, um, across all of these different disciplines. But the one that has taken off, you know, in- in- incredibly quickly is deep learning, machine learning. And I think a lot of this progress was, uh, initially driven, uh, by scale. Scale of data and scale of computation. The fact that we can now get tons of data throw it into a giant neural network and get good performance. But, uh, more recently has been also driven by, um, uh, the positive feedback loop of, um, seeing early traction in deep learning thus causing a lot more people to do research in deep learning algorithms. And so there's been tons of algorithmic innovation in deep learning in the last several years and you hear a lot about other algorithms that were, you know, relatively recently invented in this class as well. All right. Um, and so really, I think that the initially the twin forces of a scale data scale computation, but now the triple forces have also a lot of algorithmic innovation and massive investment, um, is continuing to make deep learning, uh, make tremendous progress. And so in CS230, uh, because of the, uh, uh, you know, I- I- I think our two main goals. Uh, the first is to, uh, um, have you, uh, become experts in the deep learning algorithms. Have you- have you learned the state of the art? Have you- have you- have you have deep technical knowledge on, um, the state of the art in deep learning. Um, and second is to give you the know how to apply these algorithms to whatever problems you want to work on. So, uh, one of the things I've learned- so I think, you know, actually some- some of you guys know my history, right? So, you know, I worked at Stanford for a long time. Then, um, I started as leading the Google Brain Team which, uh, did a lot of projects at Google and I think the Google Brain Team the, you know, built from scratch was arguably the leading force for, um, helping Google go from what was already a great Internet company into today a great AI company. Um, and then, uh, did something similar, Baidu in China or its Chine- you know, which was headquartered in China which kind of helped Baidu go from al- also what was already a great company into today. You know, many people say China's is greatest AI company. Um, and I think through work on many projects at Google, many projects at Baidu. And now leading Landing AI helping many companies on many projects and running around to different companies and see many different machine learning projects they have. I think I've been fortunate to learn a lot of lessons, um, not just about the technical aspects of machine learning, but about the practical know-how aspects of machine learning. And, um, if you, uh, uh, and- and I think that, uh, what you can learn from, um, you know, the Internet or from a purely academic sources or from reading research papers is a lot of the technical aspects of machine learning and deep learning. But, uh, there are a lot of other practical aspects of how to get these algorithms to work that, um, I actually do not know of any other academic course that- that kind of goes into great depth [NOISE] teaching it. Uh, there might be one but I'm- I'm- I'm not sure. But one of the things that, um, uh, uh, we hope to do in this class is to not just give you the tools but also give you the know-how and how to make it work, right? And I think, you know, I actually spent a lot of time thinking about, uh, so actually, uh, uh, late last night, I actually stayed up very late last night reading this new book by, um, John Osahalts on, ah, software architecture. Right. And I think that, um, there's a huge difference between, you know, a junior software engineer and a senior software engineer. Maybe everyone understands the C++ and the Python and the Java Syntax. Yeah, you can get that from- from, uh, from, you know, you just figure out, "Hey, this is how C++ works. This is how Java works. This is how Python NumPy works.". But, um, it's often the high-level judgment decisions, of how you architect the system. Uh, uh, what abstractions do you use? How do you define interfaces? That defines a difference between a really good software engineer versus, you know, a less experienced software engineer. It's not understanding C++ Syntax. Um, and I think in the same way, uh, today there are lots of ways for you to learn the technical tools of machine learning and deep learning. And you will learn that in this class. You know, you'll learn how to train the neural network. You learn the latest optimization algorithms. You, uh, understand deeply what a conv net is. What are, uh, recurrent neural network is. What LSTM is. You- you understand what intention model as you- you learn all of these things in great detail. You work on projects in computer vision, natural language processing, speech and so on. Um, but I think one other thing that is relatively unique, uh, to this class, um, uh, and- and- and to the- I guess the- the- the things you see, uh, on, uh, the deeplearning.ai Coursera websites as well as the things we do in class, is trying to give you the practical know-how so that when you're building a machine learning system, you can be very efficient in, uh, deciding things like, should we collect more data or not, right? And the answer is not always "Yes". Uh, I think- I think, um, with- I think that many of us try to convey the message that having more data is good, right? And that's actually true. More data pretty much never hurts, but I think the message of big data has also been over-hyped and sometimes it's actually not worth your while to go and collect more data. All right. Uh, but- so when you're working on machine learning project and if you are either do it by yourself or leading a team, your abilities to make a good judgment decision about, should you spend another week collecting more data, or should you spend another week searching for hyperparameters, or tuning parameters in your neural network. That's the type of decision that if you make it correctly, can easily make your team 2X or 3X or maybe 10X more efficient. And so one thing we hope to do in this class is to more systematically impart to you this- this type of knowledge. All right. And so I think, um, uh, e- even today, um, I, you know, actually- I actually visited lots of, uh, machine learning teams around Silicon Valley and around the world and I kinda see what they're doing and [NOISE] um, you know, recently I visited a company that had a team of, uh, 30 people trying to build a learning algorithm and the team of about 30 people was working on a learning algorithm for about three months. Right. And- and they had not yet managed to get it to work. So they're basically, you know, uh, you know, not succeeded after 3 months. Um, one of my colleagues [OVERLAPPING] have the datasets. [OVERLAPPING] Oh, Kian? Kian, you're broadcasting. [LAUGHTER] Don't say anything bad. [LAUGHTER] All right. [LAUGHTER] All right. Um, so, uh, one of my colleagues, um, took the dataset home and spent one weekend working on it. [LAUGHTER] Let's see what you're doing now. [NOISE] All right. Cool. Um, and- and- and one of my colleagues, uh, uh, working on this problem for one long weekend, he worked over a long weekend for three days, was able to build a machine learning system that outperformed what this group of 30 people have been able to do after about three months. So, what's that? That's like a, uh, I don't know, that's more than a 10x difference in- in speed, right? And- and a lot of the differences between the great machine learning teams versus less experienced ones is actually not just do you know how to, you know, implement, uh, um, uh, it's not just do you know how to implement an LSTM, right, in- in Tensorflow flow or Keras or whatever? You have to know that but it's actually other things as well. And I think, um, Kian and I and the teaching team are looking forward to trying to systematically impart to you a lot of this know-how so that, uh, when- hopefully someday when you're all leading a team of machine learning engineers or- or deep learning engineers that you could help direct the team's efforts more efficiently. Um, and oh, and actually if any of you are interested, uh, uh, one of the things I've been- actually, how many of you have heard of Machine Learning Yearning? Machine Learning Yearning? Wow, almost none of you. Okay. Interesting. Um, so this is, uh, if this is your first Machine Learning class, this may be too advanced for you. But if you've had a little bit of other machine learning background, um, Machine Learning Yearning is a book that I've been- I've been working on. It's still in draft form, but, um, uh, if any of you want a better Machine Learning Yearning, it's, uh, my attempts to try to turn- gather best principles for turning machine learning from a black art into systematic engineering discipline. And so, uh, if you go to this website, uh, uh, you know, this website will send you- actually I just finished the last- just finished the whole book draft last weekend. Uh, uh, and so email allows students. If you want a copy, go to the website and enter your email address and I'll make sure that, you know, when we send out the book- actually it might be later today, I'm not sure. That we'll- we'll- that you'll get a copy of the book draft as well. I tend to write books and then just post them on the Internet for free. So you could- but then here, we just email them out to people, uh, so you can- you can- you can get it if you go to the website. Um, and I think this will- and- and I think this class will talk a lot about a lot of the other principles of Machine Learning Yearning, but give you much more practice as well than- than just reading a book might. Um, so [NOISE] let's see. Okay. So, um, Kian will give a greater overview of what we'll cover in this class but, uh, uh, one of principles I've learned as well is that, you know, it- it- so I think, um, uh, actually some of you know my background, right? I- it's a, you know, co-founded Coursera, worked in education for a long time. So I spent a long time really thinking a lot about education and I think CS230 represents, you know, Kian and my- and our teaching teams, uh, uh, really best attempt to deliver a great, uh, on-campus deep learning course. Um, and so, oh, interesting. [LAUGHTER] Um- Um, and so the format of this class is, um, what's called the flipped classroom class. And what that means is that, so, you know, and I think, uh, I've taught on SCPD for a long time, right? For, you know, many, many years I guess. And I found that, uh, even for classes, uh, like CS229 or other Stanford courses, often students end up, you know, uh, watching videos at home. Uh, and, and I think with the flipped classroom what we realized was if many students are watching videos of these lectures at home anyway, um, why don't we spend a lot of effort to produce higher-quality videos, uh, that you can watch, that are more time-efficient for you to watch at home. Um, and so our, our team, uh, created videos, uh, deeplearning.ai created, you know, kind of the best videos we knew how to create it on deep learning, uh, that, uh, are now hosted on Coursera. And so, with- I, I actually think that it'll be, uh, uh, quite time-efficient for you to watch those videos, um, do the online programming exercises, do the online, uh, uh, quizzes. And what that does is it preserves the class time of both the weekly sessions that we meet right here on Wednesdays, as well as the TA discussion sections on Fridays for much deeper interactions and for much deeper discussions. And so, um, the format of the class is that we ask you to, uh, you know, do the online content, uh, created by deeplearning.ai hosted on Coursera, and then in class, uh, both the meetings with Kian and me- I think Kian and I will split the sessions roughly 50-50, uh, as well as for the deeper small group discussion sections you have with the TAs. That lets you spend much more time interacting with the TAs, uh, interacting with Kian and me, and going deeper into the material than just the- than, than the- than the, um, uh, than the- than the online content, uh, by itself. And, uh, that will also give us more opportunities to give you, um, advanced material, uh, that goes beyond what's hosted online, as well as, um, uh, uh, give you additional practice with these concepts, right? Um, and so let's see. Yeah. And so, um, I've also finished up with, uh, uh, two more thoughts, and then I'll hand it over to Kian. Um, I think, you know, uh, machine learning, deep learning, AI, whatever, it's changing a lot of industries, right? I, I think, you know, I think AI is the new electricity. Uh, much as the rise of electricity, uh, about a 100 years ago, starting with the United States, transform every industry, uh, really, you know. The rise of electricity transformed agriculture because finally we have refrigeration, right? That transformed agriculture. It transformed healthcare. I- imagine going to a hospital today that has no electricity, and how do you- how do you even do that, right? Without computers, medical devices, how you even run a healthcare system. Trans- transformed communications through telecom, through the telegraph initially, even now. So, much of communications really needs electricity, but electricity transformed every major industry. And I think machine learning and deep learning has reached a level of maturity where we see a surprisingly clear path for it to also transform pretty much every industry. And I hope that through, um, this class, uh, after these next 10 weeks that all of you will be well-qualified to go into these different industries and help transform them as well. Um, and I think, you know, after this class, I hope that you'll be well-qualified to, like, get a job in some of the big shiny tech companies that have, uh, a large AI teams. Um, I think a lot of the most exciting work to be done today is still, is to go into the less shiny industries that do not yet have, um, AI and machine learning yet and to take it to those areas. Actually, on the way in, I was chatting with a student, um, that works in cosmology who was commenting, was that you? No, sorry. Who has it? So- oh, at the back, who was commenting that cosmology needs more machine learning, right? And, and, and maybe he'll be the one to take a lot of the ideas from deep learning into cosmology because I think even outside the shiny tech areas like- and, and maybe since I hope it would play a role in their AI transformation of two large web-search companies I'm like done transforming Internet search companies. And I think that- but I think- and I think it's great that we have those great AI teams like Google Brain, Baidu AI group, other large tech companies have great AI teams and that's wonderful. I think a lot of the important work to be done, that I hope many of you will do, is to take AI to healthcare, take AI to computational biology, take AI to civil engineering, take AI to the mechanical engineering. I think all of this is worth doing. Um, just like electricity didn't have one killer app, it's useful for a lot of things. And I think, uh, uh, many of you will go out after this class and execute many exciting projects, both in tech companies and in, you know, other areas that, that- like, like, like cosmology, right? Uh, or other areas that were not traditionally considered, um, CS areas. Um, so, uh, just wrap up with, uh, uh, uh, two last thoughts. Um, I think that, uh, one of the things that excites me these days is I'm hoping, uh, you know, I, I wanna share with you one of the lessons I learned, right? Uh, watching the rise of AI in multiple companies and spent lot of time thinking about, you know, what is it that makes a great AI company. And one of the lessons I learned, um, was really, uh, hearing Jeff Bezos speak about what is it that makes for an internet company, right? And I think a lot of lessons, um, that we learned with the rise of the Internet will be useful, you know, and Internet was maybe one of the last major technological ways of disruption. And just as it has a great time to start working on the Internet maybe 20 years ago. I think today is a great time to start working on AI or deep learning. And so, is there a way to turn on the lights on this side as well? Do I- do I control that? [NOISE] Oh, thank you. Oh, thanks again. Great. So, so, I wanna show you one of the lessons I learned. Really, spend lot of time trying to understand the rise of the Internet because they will be useful to many of you as you navigate the rise of machine learning AI in your upcoming careers as well. Which is, um, one of the lessons I learned was, uh, you can take your favorite shopping mall and build a website for the shopping mall. That does not turn your shopping mall into an Internet company, right? So, you know, uh, like my wife, like Stanford Shopping Center, uh, uh, and I, I, I, and Stanford Shopping Center has a website. But even if, you know, a great shopping mall sells stuff on a website, there's a huge difference between a shopping mall with a website compared to a true Internet company like an Amazon or, and, and whatever. So, what's the difference? Um, about five, six, actually, six, six, seven years ago, I was chatting with the CEO of a very large American retailer. And, uh, at that time, he and his CIO were saying to me, they were saying, "Look, Andrew, we have a website, we sell things on the website, Amazon has a website, Amazon sells things on the website, it's the same thing." But of course it's not. And today, this particular large American retailer's, you know, future existence is actually a little bit in question partly, [NOISE] partly because of Amazon. Um, so one of the lessons I learned, um, uh, uh, really variances with Jeff Bezos is that what defines an Internet company is not just whether you have a website, instead it is, have you organized your team or your company to do the things that the Internet lets you do really well? For example, Internet teams, uh, engage in pervasive A/B testing, right? We, we, we know that we can launch two versions of the website and just see which one works better, and so we learn much faster. Whereas a traditional shopping mall, you can't launch two shopping malls in two parallel universes and see which one works better. So, you just- it's just- it's much harder to do that. Um, we tend to have short shipping times, right? You can ship a new product every day or every week, and so you learn much faster. Whereas a traditional shopping mall may, uh, redesign the shopping mall once per, once every three months, right? Uh, and we actually organize our teams differently. Um, we tend to push decision-making down to the engineers or engineers and product managers. Uh, because in the traditional shopping mall, you know, things kind of move slower, and maybe the CEO says something, and then everyone just does what the CEO says and that's fine. But in the Internet era, we learned that, um, the technology and the users are so complicated that, uh, only the engineers and the product managers, for those who don't know what that is, uh, own, are close enough to their technology, to the algorithms and users to make good decisions. And so, we tend to push decision-making power in Internet companies down to the engineers or engineers and product managers. And you have to do that in the Internet era because that's how you organize a company or organize a team to do the things the Internet lets you do really well, right? So, I think that was the rise of the Internet era. Um, I think with the rise of the AI era or AI machine learning or deep learning, whatever you wanna call it, um, we're learning that if you have, you know, a traditional company plus a few neural networks that does not by itself turn the company to an AI company, right? And I think what will define the great AI teams of the future, um, uh, will be, do you know how to organize your own work and organize your team's work to do the things that modern, you know, machine learning and deep learning and other AI things lets you do really well? Um, and I think, um, having met AI teams at Google and Baidu I'm a bit biased I think, you know, Google and Baidu are great and ahead of many other companies in thinking this through. But I think even the best companies in the world haven't completely figured out what are the principles by which to organize AI teams. But I think some of them will be that, um, we tend to, um, uh, I think that AI teams tend to be very good at strategic data acquisition. And so, you see AI companies or AI teams, even, even, uh, you know, do things that may not seem like it makes sense and why do these companies have all these free products, that don't make any money? Well, some of it is to acquire data that you can monetize through other ways, right? Uh, through advertising or through learning about users. And so, uh, there are a lot of data acquisition strategies that at the surface level may not make sense, but actually do make sense if you understand how this can be married with deep learning algorithms to create value elsewhere. Um, and I think that, uh, uh, AI companies tend to, um, organize data differently. AI teams tend to be very good at putting our data together. I think before the rise of deep learning, many companies have fragmented data warehouses, where if you have a big company, if you have 50 different databases, you know, in 50 different divisions, it's actually very difficult for an engineer to look at all this data and put it together and to train a learning algorithm to do something valuable. So, the leading AI companies, tend to have unified data warehouses. And I guess and I, I know we have a large home audience, the SCPD or other home audience here. So, if any of you work in large tech companies, you know, this is something that, that many companies are investing in today, to lay the foundation for learning algorithms. We tend to be very good at spotting pervasive automation opportunities,. Which is very good at spotting opportunities, where you could instead of having people do a task, have a deep learning algorithm do a task or have a different AI algorithm do a task. Um, and we also have a [NOISE] new job descriptions, which I don't have time to talk about. But just as with the rise of the Internet, we started creating a lot of new roles for engineers, right? I, I think, actually once upon a time, the world was simple and there was just a Software Engineering title. But as technology gotten- got more complicated, we started to specialize. So that's why, you know, with the Internet, we have front-end and back-end, mobile, right? And then we have, you know, and then we're increasingly other roles, right QA, DevOps, IT with increased specialization of knowledge. And so, with the rise of machine learning, we're starting the creation of new roles like, machine-learning engineer, resource- Machine Learning Research Scientist, er, and and our Product Managers in IT has also behave differently than Product Managers in tech companies. And so one of the, um, things we'll revisit a few times throughout this quarter is- and I don't mean to corporate, I know that many of you are- some of the SCPD audience or online audience already working in a company, many of you when you graduate from Stanford will end up maybe starting your own company or joining an existing company, but I think that's solving a lot of these questions, of how to organize your teams effectively in the AI era, will help you do more valuable work. And I think to, to, to make one more analogy, you know, I think that, one of the things I hope Kian and I will share with you throughout this quarter is, just as in the software engineering world, it took us a long time to figure out what is Agile development, right? or what are the pros and cons of, you know, Waterfall model versus Agile? Or, how do you- what is a Scrum process, right? Or is code review a good idea? It seems a good idea to me, right? But these, these practices after, after programming languages were created or invented or whatever, we still had to figure all these ways to help individuals and teams write software effectively. And so, if you worked in, you know, high-performing corporate, industrial AI teams using these software engineering practices, everything; code review, to Agile, to, to, to whatever, you know, you know that having a team work effectively to write software is more than everyone knowing C++ syntax, everyone doing Python syntax. And I think in the machine learning world, we're still in the process of inventing these types of processes. What is the strong, what is the Agile development, what's the equivalent of code review for developing machine learning algorithms and I think probably this class more than more than, this class and Machine Learning Yearning more than any other visuals I'm aware of right now. I think we'll try to systematically teach you these tools so that you don't just are able to derive a learning algorithm and, and implement a learning algorithm but that you're actually, you know, very effective in terms of how you go about building these systems. So last thing before I pass it to Kian is, the other question that I've been asked I guess several times this week now, that I'll just preemptively answer is, so, there are multiple machine learning classes going on at Stanford this quarter. So, the other frequently asked question is, which of these classes should you take? So let me just address that preemptively before someone asks me because I've been asked twice already in the other two classes this quarter. So, I think, actually what, what has happened over the last several years in Stanford is, the demand for machine learning education has, you know, been rising dramatically because the majority of CS PhD applicants to Stanford you, are applying to do work in machine learning or applying to do work in AI. And I think all of you can kinda of see that, there's such a shortage of machine learning engineers, right? And then there's a little bit of, and I think that shortage will continue for a long time. So I think many people see that. If you become expert in machine learning, there will be great opportunities for you to do meaningful work on campus, to take machine learning to comp-bio or- or cosmology or mechanical or do great research on campus as well as graduate from Stanford and do very unique work. When I wander around Silicon Valley, I feel like there are so many ideas for great machine learning projects that exactly zero people seem to be working on because there just aren't enough machine learning people in the world right now. So, by learning these skills, you could- you have many opportunities to be the first one to do something very exciting and meaningful, right. And, and you probably read in the newspapers about how much money machine learning people make, I'm actually much less more, I actually find that, I hope a lot you make a lot of money. Perhaps you you personally don't find that that, you know, as, as exciting. I think that, every time there's a major technological disruption, it gives us an opportunity to remake large parts of the world. And I hope that, some of you go improve the healthcare system, improve educational system, maybe, you know, see if we can help preserve the smooth functioning of democracy around the world. I think that it, it really your unique skills in deep learning would give you opportunities to do that. I think hopefully very meaningful work. But because of this massive, massive rise in demand for machine learning education, there are, so for longtime, CS229 machine learning was the core machine learning class at Stanford. And then CS230 is actually the newest, new creation I think. And the other class that we're involved in, that Eunice and I are involved in this quarter is CS229A. So, um, so if you are trying to decide which of these classes to take, um, I think, I think that these classes are little bit like Pokemon, right? You really should collect them all [LAUGHTER] [NOISE] But, but, but I think, we've been trying to design these classes to actually teach different things and not have too much overlap. Uh, uh, And so, there is- uh, so I have seen students take two classes at the same time and that's actually fine. There's not, the degree of overlap is fine, that you actually learn different things, if you take any two of these classes at the same time. CS229 is machine learning, is the most mathematical of these classes and we go much more, CS229 goes much more into the mathematical derivations of the algorithms. CS229A is applied machine learning, is much less mathematical but spends a bit more time on the practical aspects. Is actually the easiest on ramp to machine learning as well as the least mathematical of these classes. CS230 is somewhere in between. It's, it's a bit more, is more mathematical than CS229A, less mathematical than CS230. But where CS230 focuses on is on deep learning, which is just one small subset of machine learning but it is the hardest subset of machine learning. Whereas there are a lot of other machine learning algorithms, from your PCA, K-means recommender systems, support vector machines that are also very useful, that I use, you know in my work quite frequently, that we don't teach in CS230 but then it's taught in CS229 and CS229A. Where, so- so the unique things about CS230 is, it focuses on deep learning. So, I don't know, if you want to list deep learning on your resume, I guess maybe this is the easiest way to do it, I don't know after. Again it's not what I tend to optimize for but- but and I think CS230 goes the deepest in the practical know-how in how to apply these algorithms. And so, uh, and I, I want to set expectations accurately as well, right? So, what I don't want, is for you guys to complain in the other quarter that, you know, there wasn't enough math because that's actually not the point. What has happened in the last decade is, the amount of math you need to be a great machine learning person has actually decreased, I think. Uh, and I wanted to do less math in CS230 but spend more time teaching you the practical know-how of how to actually apply these algorithms, right? So, um, yeah. and I think 229A is probably the easiest of this class, that's the most technical, this is the most, most hands-on applied, you do a lot of projects on different, different topics, right? And I think these courses are often the foundation or some subset of these are often the foundational course as you would say. Because if you say, learn deep learning, so common sequence for our students is that, you know, learn the foundations of machine learning, uh, or uh, machine learning or deep learning. So, you have the foundation, uh, first before you go, which then often sets you up to later, go deeper into computer vision or natural language processing or robotics or deep reinforcement learning. And so common sequencing that, common tactic that Stanford students take is to use these as the foundation. You'll see a bit of everything from computer vision, natural language processing, speech recognition, you will touch a little bit on self-driving cars. But that gives you the foundation to then decide, do you want to go deeper into natural language processing or robotics or reinforcement learning or computer vision or something else. So these are common sequencing of classes that students take. Okay. So, um, look forward to spending this quarter with you. Let me just check where there are any quick questions and then I'll hand it over to Kian. Yeah, go for it. What's the third bullet in the AI era? Oh, what is the third bullet in the AI era? Decision-making by engineers and product managers. We'll be pushing decision-making, I wrote the decision making by engineers there, but really engineers and product managers. AI era I'm sorry. Oh, AI era, a pervasive auto, sorry, pervasive automation. Yeah, please So, uh, when you are talking bout AI being the next like electricity, um, I was wondering like, so- so to say that like uh, well like so far, you, like wha- what are the most, like the most meaningful successes of machine learning, that you think have happened already? So all of you are using learning algorithms probably dozens of times a day maybe even hundreds of times a day without knowing it. Right, every time you use a web search engine, there's a learning algorithm that's improving the quality of search results. There's also a learning algorithm trying to show you the most relevant ads and this helps those companies actually make a lot of money. Uh, every time, it turns out that um, uh, actually both Google and Baidu have publicly said that over 10 percent of searches on mobile are through voice search, uh, and so I think it's great that you can now talk [NOISE] to your cell phone rather than type on a tiny little keyboard if you're gonna [NOISE] do a- do a web search on mobile. Uh, if you go to, you know, websites like Amazon or Netflix or, uh, uh, uh, there are learning algorithms recommending more relevant movies or more relevant products to you. Uh, every time you use your credit cards, uh, there's a learning algorithm uh, trying to probably- for almost all companies I'm aware of, there's a learning algorithm trying to figure out if it's you using your credit card or if it's been stolen so they should- so they should, you know, disallow the- see if it's a fraudulent transaction or not. Uh, every time you open up your email uh, the only reason email is even usable is because of your spam filter which is because of learning algorithm that works much better now than, than, than before uh, I don't know, uh,uh, I,I yeah- so, so there's uh, I, I think, uh, you know, one of the amazing things about AI and machine learning is, I love it when it disappears in the background. Right, yo- yo- you use your-, you know, you use these algorithms. You boot up your map application then it finds the shortest route for you to drive from here to there, and there's a learning algorithm predicting what traffic would be like on highway 101 one hour from now. But you don't even need to think that there was a learning algorithm trying to figure out what traffic would be like one hour in the future. Seems pretty magical. Right, that, you know, tha- tha- that you could just use it. There is this, we could build all of these wonderful products and systems that helps people but abstract away a lot of details. So that's the present and I think in the future, near future, uh, most of my PhD students, my- my- most of my research group peers work on machine learning for healthcare. I think that will have significant in roads, you know, my, my, my team at Landing AI spending a lot of time with a lot of industries from manufacturing to agriculture. So all the things, uh, I'm excited about machine learning for education, uh, get people precise tutor help people with recommended precise content. There is fascinating research done here at Stanford by Chris Peach and a few others on using learning algorithms to give people feedback on coding homework assignments. Uh, I, th- so- so- sorry there are so many examples of machine learning I could talk for quite some time. Yeah. One last question I hand over Kian so. Yeah, go ahead. So will lectures be very different from what is on Coursera? Um, let's see. So the, uh, so the format of the class is that you watch, uh, videos created by deeplearning.ai and hosted on Coursera. So you'll see me a lot there. But in addition Kian and I, will be having lectures here in this classroom every Wednesday, and that will be, you know, completely new material that is not online anywhere. At least right now. Yeah- yeah. And then also the- I think that, tha- that the point- the point of that the flip classroom thing really is some of the things is really more time efficient for you to just learn online. So there's the online content. But what that does is it leaves this classroom time for us to not deliver the same lecture year after year but to get- try to get- spend time to get to know you and we have more time answering your questions and also give you more in course practice on these things. Right, so there is the Coursera DIY Coursera content. But what we do in CS230 is to augment to give you a much deeper practice, more advanced examples, some more deeper mathematical derivations, and more practice so you- so you deepen your knowledge of that. And with that, let me hand it over to Kian [NOISE]. Yeah, I'm gonna get back at him by making noise while he's talking. Yeah okay [LAUGHTER]. It's OK. Okay. Thanks Andrew. Hi everyone. Uh, I'm Kian. We're excited to have you here today. Those of you who are in class but also those of you who are SCPD students. Uh, we wanted to take a little more time to explain a little bit about the course logistics, what this course is about and also what it is to be a CS230 student in fall 2018. So, the course online is structured into five chapters or sub courses let's say. Uh, what we will teach you first is what is a neuron. You need to know that. After understanding what a neuron is you're going to be layers with these neurons. You're then going to stack these layers on top of each other to build a network that can be small or deep, uh, this is the first course. Unfortunately, it's not enough to deploy a network. Uh, just- just building, uh, neural network is not enough to get it to work. So in the second course, we're going to teach you the methods that are used to tune this network in order to improve their performances. This is the second part. Um, as Andrew mentioned, one thing we're really, uh, putting a huge emphasis on, uh, in CS230 is the industrial applications and how the industry works in AI. So the third course is going to help you understand how to strategize your project that you'll do to for the quarter, but also in general how do AI teams work? You can have an algorithm, you have to identify why does the algorithm work, why does it not work, and if it doesn't work, what are the parts that you should improve inside the algorithm? The two last courses, course fours-, ah, course four and five are focusing on two fields that are defined by two types of algorithms. First, Convolution Neural Networks that have been proven to work very well on imaging, um, or videos. And on the other hand sequence models that include also recurrent neural networks that are applied a lot in natural language processing or speech recognition. So you're going to see all that from the online perspective. Um, we use a specific notation in CS230. So when I will say C2M3, it refers to course two module three. So the third module of improving deep neural networks. Okay. And, I'd like everyone to go on the website CS230 syllabus after the class to look at all the syllabus for the quarter, check when the midterm is and when the final poster presentation is. Um, the schedule is posted there. Um, so check it out and we're going to use the Coursera platform as you know. So on Coursera, uh, you will receive an invite on your Stanford email and you should have received it already for course one, um, in order to access the platform. From the platform you will be able to watch videos, do quizzes, and do programming assignments. And every time we finish one of these courses, so C1 has four modules. When you're at C1 M4, you will receive a new invite to access C2 and so on. Okay. Inside CS230 we're going to use Piazza as a class forum for you to interact with the TAs and with the instructors. Uh, you can post privately or publicly depending on the matter. Okay. So let's see what it is to be One week in the life of a CS230 student. So we're going to do 10 times that over this- the fall quarter. So, what is one module? In a module, you will watch about, uh, 10 videos on Coursera, which will be about one hour and a half. Uh, you will do quizzes after watching the videos. This is going to take you about 20 minutes per module, and finally, you will complete programming assignments, which are on Jupyter Notebooks. You will get cells to test your code and also submit your code directly on the Coursera platform. In one week of class in Stanford, here, we will have two modules, usually. On top of these two modules, you will come to lecture for a one hour and a half in-class lecture on an advanced topic that is not taught online. And after that, uh, you will have TA sections on Fridays that are around one hour, and it's a good chance for you to meet other students for your projects, and also to interact with the TAs directly. Um, finally, we have also, uh, personalized mentorship this quarter, where, uh, every one of you will meet 15 minutes per week with the TA in order to check in on your projects, and gives you the next steps. So, we put a huge emphasis on the project in this class, and we want you-, you will see it later to build, to- to decide of your teams by this Friday in order to get started as soon as possible. Uh, next week, you will have your first mentorship meeting with the TAs, okay? It's gonna be fun. Uh, assignments and quizzes, uh, that are part of modules are due every Wednesday at 11:00 AM, so 30 minutes before class, so you can come to class with everything done and understand it. Uh, and do not follow the deadlines displayed on the Coursera platform, follow the deadlines posted on the CS230 website. The reason the deadlines are different is, because we wanna allow you to have late days, and Coursera was not built for late days, so we, we put the deadlines later on, on Coursera to allow you to submit even if you, you wanna use a late day. Does that make sense? Okay. So we are also using a, a kind of interactive, uh,- this, this is gonna start course two. We, we will use, uh, an interactive, uh, tool that is called Mentimeter, uh, to check in attendance in class and also, for you to answer some interactive questions. So, it's gonna start, uh, next, next week. Sorry, not course two. Uh, regarding the grading formula, uh, here it is. So, you have a small part on attendance, that is two percent of the final grade, eight percent uh, on quizzes, 25 percent on programming assignments, and, uh, big part on the midterm, and on the final projects. Uh, so this is posted on the website, if you want to check it. Uh, attendance is taken for in-class lectures for, uh, 15 minutes TA meetings and for the TA sections on Friday. You can have a bonus, and we've had students very active on, on Piazza, answered questions to other students, which was great, and they got a bonus, so I encourage you to, to do the same. Maybe we don't need TAs and instructors anyway. Okay. So, I- I wanted to take a little more time to go over, uh, some of the programming assignments that you're going to do this quarter uh, so that you, you know where you're going. Uh, in about three weeks from now, you're going to be able to translate these pictures here in the numbers that they corresponds to in, in sign languages, so it's sign language trans- translation from images to, uh, the output, uh, signification. Um, you're going to build a convolutional neural network, uh, and the first logistic regression and then a convolutional neural network in order to solve this problem. Uh, little later uh, you're going to be a, a Deep Learning engineer in a house that is not too far from here, called the Happy House. So, there is only one rule in this house, and the rule is that no sad person should enter the house, should avoid that. And because you're the only Deep Learning engineer that has the knowledge, you're given this task, which is don't let these sad people in, just let happy people in, and you're going to build the network, uh, that will run on a camera, that is in front of the house, and that is going to let people in or not. And unfortunately, some people will not get in, and other people will, will get in because they- they're happy, and you will save the Happy House at the end of the assignment, hopefully. Uh, this is, uh, one of the, the, the applications of, of deep learning I- that I personally prefer. It's called, uh, object detection. You, you might have heard of it, so this is running real time, and that- that's what is very impressive. You're going to work on, uh, Deep Learning architecture called YOLO v2, and YOLO v2 is an object detection algorithm that runs real-time, and is able to detect 9,000 objects, as fast as that. So, it's, it's really, really impressive. You have a few links here if you want to check the paper already, but maybe you will need, uh, some weeks to understand it well. Okay? [NOISE] Yeah, actually we have a- we can even run it directly on my computer, I think. It's going to be fun. [NOISE] We can run it. So here, you see it's running live on this computer, and so you see that if I move, it will find out that I move, so I cannot escape. Yeah, here it is. Okay. [NOISE] Okay, a few other projects, uh, one- two weeks from now, you will build an optimal goalkeeper shoot prediction. So, in soccer, you're a goalkeeper, and you want to decide where you should shoot the ball in order to make it land on one of your teammates. You're going to find, uh, what's the exact line on the field which tells the goalkeeper where to shoot, two weeks from now. About, um, i- in the- in the fourth course, uh, convolutional neural network, you're going to work on car detection, so this is a, a bigger image. Uh, this is exactly the programming assignment, so you're going to work on the autonomous driving application, that is finding cars, finding stop signs, finding lights, finding pedestrians and all the objects that are related to road features, okay? This is pretty cool, and you will generate these images yourself. So, these are pictures taken from a camera put in the front of, uh, of a, a car and was, was generated by Drive.ai. You will have a face recognition system that is going to first do face verification. Is this person, is this person the right person, but also face recognition, who is this person? Which is a little more complex. We're going to go over that together, uh, both online and in lecture. Art generation, some of you have heard of this, uh, it's an algorithm called Neural Style Transfer. And again, we usually put the, the papers at the bottom of the slides in case you want to check in yourself, for your projects. Uh, but this is a problem where you give the content image, which is the Golden Gate Bridge, and a style image which is an image that was painted usually by someone or an image from which you want to extract this style. This algorithm is going to generate a new image, is go- going to mix the contents of the first image with the style of the second image. Music generation, which is super fun, you're going to generate jazz music, um, in the fifth course, um, sequence models. You're going in the same course also generate texts by giving a huge corpus written by Shakespeare along time ago of poems. You're going to teach the algorithm to, to generate poems as if it was written by Shakespeare. So, you can even write the first sentence and is going to continue. Emojifier, you, you all have smartphones and I guess you notice that when you write a sentence on your smartphone, uh, it usually tells you what you should put next and sometimes it's an emoji. You're going to do this part. You're going to implement the algorithm that takes an input sentence and tells you what's the emoji that, that should come after it. Machine translation is a- is one of the application that has been tremendously performing well, uh, with deep learning. You're going to implement not a full machine translation from one language to another, but a similar task that is as exciting, which is, uh, changing human-readable dates to machine-readable dates. So, you know, let's say you're, you're, you're, you're filling in a form and you're typing a date. The, the entity that, that gathers this data will have a hard time convert all these dates into a specific format. You're going to implement the algorithm that is going to take all these different dates in different formats and generate the right formats, translated to human, from human-readable to machine-readable dates. Finally, trigger word detection that I also love and, and some of you have, have seen us build this algorithm, uh, a year ago I believe, which was- which Eunice and, and Andrew and I have, have worked on. Um, trigger word detection is the problem of detecting a single word. So, you know, you, you probably have, uh, objects from big, uh, companies that detect the voice and activate themselves under a trigger word. You're going to build this algorithm for the trigger word activate and many more projects that you will see. Now, these are the things that you will all build in this course. Every one of you will build it through programming assignments. But you also have to choose your own projects to work on throughout the course. And these are examples of projects that CS230 students have, have built in the past and which have wor- worked very well. One is coloring black and white pictures using a neural network into the color representation of these pictures. So, it's pretty cool because we can now watch, uh, movies that were, that were filmed in the 1930s or 1950s or I don't know when, uh, in color, which is super cool. Predicting a price of an object from a picture. So, this was a great project in the first iteration of CS230 where you give it a bike and the neural network guesses how much is the bike. So, if you wanna sell stuff you don't know how much, you just give it then you sell it at the, at the price. Uh, the student had actually implemented an algorithm to see which features of the bike are related to the price. So, it was super fun to see if it's the steering wheel or if it's the wheels or if it's the body of the bike that's makes this bike expensive according to the algorithm, and many more. So, last quarter, specifically, we had a lot of projects, uh, in physics and, uh, and astrophysics and chemical engineering and mechanics, which was great. Uh, some examples are detecting earthquake precursors signals with a sequence model. Um, predicting the atom energy-based on the atomic structure of an atom. So, you have, you have, for instance, softwares that run, that are really computationally expensive that look at the atomic structure of an atom and will output the energy of this atom. This takes a long time. These students have tried to make it a three second problem by running a neural network to find the energy of the atom. So, you have a bunch of problem across industries. So, healthcare, cancer, Parkinson, Alzheimer detection, we've had a lot of this. We've had brain tumor segmentation. Segmentation is a problem of on an image, classify every pixel. Tell me which pixel corresponds to the tumor, for example. So, we- we're really excited, uh, to see what you guys are going to build at the end of this quarter. And that's why we want you to build your teams very quickly, get started, because the project is what you should be proud of at the end of the quarter. We hope that you guys will come at the poster session proud of your poster, proud of the final project that you sent us and you can talk about it in the ten next years or 20 next years, hopefully. And I guess Andrew can, can, can confirm that CS229 students from the few past years have done projects that are amazing today and have been featured, uh, around the world in- as a researcher or, or industrial project. So, to sum up, in this course you will build a wide range of applications. Uh, it's very applied. There is some math but less than CS229 more than CS229A. Uh, and you have access to personalized mentorship thanks to the amazing TA team and the instructors. Um, and finally, we'll have, uh, to build a ten week long, uh, project. So, now we, we get to the serious thing. What is, uh, what we are up to this week. So, at the end of every lecture, you'll have one slide that's gonna remind you what you have to do for next week. Uh, next Wednesday, 11:00 AM. So, create your Coursera accounts based on the invite that you receive. If you didn't receive an invite, uh, Send it as a private posts on Piazza, we will send it again. Finish the two first modules of course one C1M1 and C1M2. It corresponds to two quizzes and two programming assignments and around 20 videos, okay, which are listed here. And for Friday, it means two days from now, uh, by the end of the day, uh, find project teammates and, uh, fill in the form to tell us who are your teammates. It's going to help us, uh, find you a mentor. Um, finally, there is a TA section also this Friday, no project mentorship. It will start next week, uh, but we, we will see you on Friday. Uh, I'm going to take a few questions if you have about. Yes? Are these slides also available to us? Yeah. These slides are going to be posted, uh, at the end of this class. So, the TA sections, we're going to have a large range of TA section on Friday. So, there's going to be, basically, every time you're going to be assigned to one of them and if you wanna move, you can send an email as a pro- um, uh, a Piazza post privately to ask to be moved to another section. So like how big is the team? How big is the team? Usually, it's from one to three students. Exceptionally, we, we would accept, uh, four students if the project is challenging enough, yeah. Yes. Can we combine the project with other classes as well? So, uh, it is possible to combine the project with other classes and it's been done in the past. Uh, what we want is you to, to give a project and a poster that, that is framed as CS230 wants it to be framed. And you discuss with us, in order for us to validate if you can merge this project with another class because it requires to have deep learning, of course. You- you're not supposed to combine this project with something that doesn't have deep learning at all. Okay. One more question. [NOISE] I think on Coursera you can retake quizzes. I think on Coursera you can retake quizzes and can we retake the quizzes in this class? So, you can, you can retake the quizzes as much as you want on Coursera. Uh, we will consider the last submitted quiz for this class. Okay. So, you can resubmit if you didn't get full way, yeah. Okay. Thanks guys and see you on Friday.
