[MUSIC PLAYING] JOSH GORDON: Hey, everyone. Thank you very much for coming. My name is Josh Gordon. I work in TensorFlow
developer relations, and I'm really excited to
share with you some thoughts on deep learning today. A whole bunch of
resources you can use to learn much more
about it and a pair of my favorite
open source models for image classification,
natural language parsing, and some neat
tricks you can do with these. And I'll also show you some cool
things and how you can actually use a pre-trained
image classification network to generate
your own artwork, which I find really interesting. So before we get
started, I just wanted to take a moment to
reflect on this idea of reproducible research. So we can do a lot
of things today with deep learning that I
think we take for granted. So in 2005, I spent
about six months trying to do basic
image classification. I was working in a
medical lab, and we had a slide containing some
solution with some cells on it. And my goal was to
use a neural network to identify cells that were
diseased and cells that were healthy. So I was literally just
tagging regions of an image. And there were many machine
learning libraries in 2005, including some great
ones that I still use today in a teaching setting,
like Weka, which I absolutely love. But we were still writing a
lot of our neural network code by hand. And it took us about
six months to get a network that worked well just
for this binary classification task. And it was really
hard to experiment with different
network architectures and different optimizers. And what's interesting-- we had
a lot of really good engineers working with us. And doing this type of
image classification today looks very different. So today, a strong
Python developer with some basic
background in TensorFlow and willingness to try
different open source models and read some literature, could
do a much, much, much better job than I could have done in
six months, like full time, in just a few days. And so there's really been
a lot of value created. And the reason that we can do
all these great things is that a lot of researchers,
academics, and developers, both from universities,
and Google, and other large
companies-- and small-- have been, I think,
very generous in sharing their tools and their
ideas and their code, so that the next
generation of developers can iterate and
refine on their ideas. So we have a lot of
value at our fingertips. And because I started my machine
learning career in medicine, I just also wanted
to take a moment to reflect on some value
that's been created by a lot of this reproducible
research and these open source models. And so here are
three applications of medical imaging from the
last eight months or so. And they all rely on an
open source TensorFlow model that I'll show you
today, called Inception. And Inception is one of the
world's most accurate image classifiers. It's completely
open source, which means you can check it out
and train it on your own data to classify things
you care about. Here we're looking at work
in diabetic retinopathy. I'm going clockwise
from the top left. Diabetic retinopathy--
and the basic idea here is there's a eye disease. And if it's caught
early, it's treatable. And if it's caught late,
it can lead to blindness. And so in this work, we're
using image recognition to identify patients
in need of care. The next image here
was a "Nature" paper from Stanford University. And this was using TensorFlow
and image classification-- actually on Android-- to detect skin cancer. And the cool thing
about this is that-- I don't mind going
to the doctor. But I'm kind of a lazy guy. But if I had a Android app
that I could just quickly scan my arm to find out if
something was malignant or not, I'm much more likely to
do advance screening. And then in the
lower left, we have work using image classification
in medical pathology. And the idea here is
you have a pathologist, and their job is to
take a look at a slide and find cells
that are cancerous. The problem is these
slides can be enormous. They can be 100,000
pixels by 100,000 pixels. So we're talking about billions
of pixels on this slide. And they might be looking
for a small region that contains cancerous cells. And so you can imagine that
scanning these slides by hand can be incredibly demanding. But, of course, if you have
an image recognition model, you can analyze
every single region on that image with
a fine tooth comb. And you can really build
tools to assist physicians in their diagnostic work. So we can also do awesome,
fun things with, like, art, and music, and sound. But there's also, like, I think,
real humanitarian value here that's becoming
increasingly accessible. And the reason I really
like this medical stuff is-- this is a field that is
not computer science. But because of a lot
of reproducible work and deep learning,
we're seeing a lot of value being created
in many disciplines, including things you
wouldn't expect-- like artwork. So I mentioned the word deep
learning a couple of times. And so here I'd like to show
you a cartoon diagram of a fully connected deep neural network. And this particular network
is classifying this image as a cat or a dog. And I wanted to show
you this to highlight one of the differences with deep
learning and some of the work that I was doing in, say, 2005. So the way I would have written
an image classifier in 2005 is I would have written
a lot of Python code to extract features
from the image. I would have thought
about what features do I want to feed
into my classifier. And these would be lines,
and shapes, colors. I might have used a
library, like openCV, to do face detection. But in deep learning, we
can give the classifier the raw pixels from the image. And we'll let the network
find good features on its own. And we can actually visualize
what it learns later. The reason this is called
a deep neural network is there is more than one layer. We have a stack of layers,
it's a deep neural network. That's where the term
deep learning comes from. And at the very
top, this network happens to be classifying
that image as a cat or a dog. There are many wonderful
deep learning libraries, but, of course, my
favorite is TensorFlow. It's from Google, and as
you guys probably know, it's used both for
research and production. This means that
TensorFlow has everything you need to build products with
hundreds of millions of users, like Google Photos, as
well as do actual research, like the work you saw
in medical imaging. And having the same code
base for both of these is very valuable. So these are the-- I'm going to have to fly
through this, by the way. I have a lot of content that
I want to share with you. So I'm going to have to
accelerate a little bit. So here are some of the papers-- well, these are
actually blog posts. But all of them point
to paper that I'd like to share with
you today and show you how you do practical and
fun things with them. And, of course, there's
much, much, much more. Google Research, in the
last couple of years, has been prolific in
publishing papers. And many of these
are state of the art. And often, in
addition to the paper, they publish everything you need
to try this out on your own. And I'm going to show
you what that means. So one question that comes up-- if you have a, say, state of
the art natural language parser, that has a lot of
business value. And one question
that's asked me a lot is, well, why-- why
open source that? Why share it? And one really
important reason is that it empowers others
to continue your ideas. So I'm not doing research now. But even as just a hacker,
I have many more projects than I will ever
finish in my lifetime because you guys are all busy. There's not enough
hours in the day. But by open sourcing
your code, you make it possible for
the next generation to continue your
line of thought. And then, another huge benefit
is that it reduces barriers to entry. So rather than having to design
your own image classifier from scratch, you
can immediately go to this state of
the art and open source and get started right
away on the problems that you care about. So, briefly, here's what makes a
good, reproducible, open source release. Of course you want
to share your code. But in machine learning, you
want to share all of the code. That means the code you need
for inference, for training, as well as for evaluation. So you can't have
any hidden pieces-- otherwise it's not reproducible. You always want to share
the data set that you used to train the model. I know sometimes there's
lots of copyright issues, especially if you're
working with images. But you should at least
make it reproducible. It's also really helpful
to include a toy data set when you share a model. And by a toy data set,
I mean a small piece that makes it really
easy for someone to test if their end to end
pipeline is working properly. A pre-trained checkpoint
is super valuable. So I believe the first
implementation of Inception-- and this was about a year ago-- took something like a week to
train on a machine with 8 GPUs. Of course, not everyone happens
to have 8 GPUs sitting around. So by sharing a
pre-trained checkpoint, that means the model works out
of the box to classify images and it makes it much
faster to explore with. And then something that I'm a
recent convert to is Docker. So oftentimes, you have
a lot of dependencies in your environment. And so by sharing
a Docker container, it makes it much faster for
people to try your ideas. So this talk is about
open source models, but of course, you can
also develop with deep learning by writing code. And briefly, I want to show
you what that looks like. So the point I want to
make is that developing with deep learning does
not have to be hard. With basic Python skills,
you can write neural networks in something like 15
lines of code now. So this is code from
a library called Keras, which is a high
level machine learning API that I really like. It's very concise. And it's fully
supported in TensorFlow. And the idea is that
in TensorFlow, you'll get the best of both worlds. And here is almost
the entire code to define a network very
similar to that cartoon diagram we saw earlier. I'm adding a stack of
layers, and most of the work here is going into making sure
that the weights and the sizes all line up. I've got to pick up
the pace of this. Here's code to
compile the model. And so, in deep learning,
if you're learning it for the first time, one
thing to be aware of is you'll run into a whole
lot of new terminology. And oftentimes, the
terminology you encounter can be quite scary. Like here, we're
looking at something that says loss equals
categorical cross entropy. And the first time I saw
that, I was like, oh my god. This is incredibly different
from a lot of stuff I've worked with in the past. But oftentimes, when
you look at these terms, they're simpler than they sound. So loss in deep
learning just means air. And it's just a number. And categorical
cross entropy is just a fancy word for comparing
two distributions. So there's a distribution
that you predict, and there's a distribution
you should have predicted. And that will just
give you a number that says how different they were. I also wanted to let
you know that TensorFlow comes with a nice visualization
tool, called TensorBoard. And as you can see from
this animation that has failed
spectacularly, you can use TensorBoard to actually
visualize the neural network that you defined and actually
poke inside the layers. So the second way to get
started with deep learning is to leverage open
source, which is what I want to focus on today. And the first model I want
to talk about is Inception. And I'll show you
how can you use how you can use Inception
for image classification as well as Deep Dream
and Style Transfer. So before, we saw a
cartoon neural network. But here's what Inception
actually looks like. You can see it's much larger,
it has many more layers. And inside the layers,
there's more moving parts. And when you're defining
a network like Inception, this is somewhere
where TensorFlow really begins to shine. And tools like TensorBoard
make it really easy to spot any errors that you
have in your architecture. So first I want to talk
about image classification. So I'm from New York. So here's a picture
I snapped last week, looking north towards
Central Park from the office. And when you're doing
image classification, the easiest way to get
started is to use an API. And I just want to show
you what this looks like. So this is the Google
Cloud Vision API. It's drag and drop, and there's
also a RESTful interface. And basically, you can
classify the image, and you'll get a
set of tags back. So here, this is a skyline,
city, metropolis, whatever. But you can also do
this in open source. And you can do this,
actually, quite easily. So here's code for Keras,
running inside TensorFlow. And this is actually the
complete code on this slide. It's complete. And this code will
actually import Inception, which is a network
architecture that we saw in the previous slide. It will load the
pre-trained weights from a checkpoint
and pre-process and classify an image. And so this stuff is much,
much, more accessible than often seems. And this right here-- in
these, like, whatever-- five lines of code-- is much, much, much
better than what I could have done just a few
years ago with a huge amount of effort. And the funny thing about
Inception out of the box is if you classify this image,
you get nonsensical tags back. So here, Inception is
telling me that this is a valley and
possibly a doormat. And the reason why that is--
and it's not a problem-- the reason why that is
is the pre-trained model that comes with Inception
is from a data set called ImageNet. And ImageNet has a
million different images, from 1,000 different categories. But most of them are pictures
of dogs, cats, flowers, artwork. And so, of course,
image classifiers only can recognize the things
they've been trained on. So Inception doesn't know any
better when you get started. And in fact, the blog post that
announced the latest version of Inception is very proud
that, out of the box, Inception is actually
sensitive enough to tell the difference
between these two different species of dogs. One's a malamute
and one's a husky. And it's this sensitivity that,
when you retrain Inception on different data, makes it very
good in the medical diagnosis domain. And in fact, there's a good
expression-- don't trust, but verify-- if you run inception
on this image, you'll actually get output
that's more sensible. So it will identify
it correctly as a dog. So probably you don't care
about recognizing dogs. So how can you take Inception
and retrain it to recognize images that you care about? And the good news is
there's this concept in machine learning, called
transfer learning, that I want to briefly introduce
and then show you how accessible it is. So the idea in transfer
learning is simple. And that's-- if you have learned
a neural network to, say, recognize dogs, and
now you want to recog-- recognize, say,
cities, do you really need to start from scratch? While you're learning
to recognize dogs, the network has probably
identified useful features in images that might be
applicable to other things. So instead of throwing
away the network that you've previously learned
and retraining from scratch, instead you can do
surgery on the network. You can chop off the very
last layer of weights and create output nodes just
for the classes you care about. And then, instead of re-learning
the entire network, which might take a week
and a million images, you can re-learn just
the very last layer. And you can do
that in 30 seconds, and you might only
need 100 images. So with a very small
amount of data, you can actually create
your own image classifier. And I wanted to let you know
we have a code lab for this. And just a couple
of days ago, we published a second
version of it, called TensorFlow for Poets. And TensorFlow for
Poets is dead simple. It's basically going to run
a bunch of scripts for you that do that surgery
on the network. And the only thing you need
to train a very high accuracy custom classifier is literally
to collect directories of images that you care about. So what we're looking
at here is an app that I built at the Metropolitan
Museum of Art in New York. And I can point at
paintings and sculptures and it will tell
me which one it is. And Poets 1 and 2
will actually walk you through training a
classifier and installing it onto an Android. A hilarious piece of criticism
that I got about TensorFlow for Poets was some developers
felt that it was too easy. And TensorFlow for Poets
is a really easy code lab. It's not meant to go
into the details of how transfer learning works. But we have TensorFlow tutorials
for that if you're interested. But there's something
really important about TensorFlow for Poets. A lot of using machine
learning and practice is not about thinking
really carefully about how to define
models, and, like, using sophisticated
algorithms and lots of math. A lot of machine
learning is just like development in general. It's-- if you're
building an app, like this one at the museum, you
have to think carefully about how is this app going to
be used in production. So if a user picks it
up and they point it at a painting or
a sculpture, you have to think about things
like, what if it's dark? What if there's someone
in the background? What if there's an obstruction? And so a lot of the
thinking in machine learning is how can you
collect training data to cover these different cases. And if you do that
properly, then you'll have an app that works
very well in practice. And for Android
developers here, I wanted to let you know
we have three Android samples for TensorFlow. The one on the far
left is doing something called style transfer,
which I'll show in a sec. It's beautiful. The one in the middle
is an image classifier which runs Inception. And the one on the right
is a object detector, which can help you identify the
location of things in images. So one really, really
fascinating thing about deep learning-- that I would just want to
give you a preview of-- is that when you learn an
image classifier, in addition to being able to
classify images, that classifier actually
has gained some knowledge about images in general. And you can use that knowledge,
surprisingly, to generate art. I'm going to-- I'm
running behind, so I'm going to skip
that, and we're going to go straight into Deep Dream. So question-- how many
people have heard of or tried Deep Dream? So, like, half? Right. How many people know
why it's important? Much fewer. So, to me, the
point of Deep Dream is not that we can generate
beautiful psychedelic artwork. Like, that's really
wonderful, but that's not why Deep Dream is
fascinating to me. The reason that I find
Deep Dream interesting is it really begins to
investigate the question-- why is it that deep
neural networks are so good at recognizing images? And there's things we know about
deep learning and general-- why they're so effective. Of course, with,
like, GPUs and cloud, we can train larger
networks on much more data. So that's one reason why
networks are so good. Another reason is there's been
many algorithmic improvements over the last few years that
enable you to train networks more efficiently. These are things like different
initialization strategies, or optimizers, or batch norm,
but the reason, in my view, that-- why deep neural networks are
so good at recognizing images is they automatically
learn features. So I mentioned-- with that
cartoon diagram at the start-- that the network will find
useful features from images automatically. And in addition to
finding useful features, networks actually learn a
hierarchy of those features. So in most image
recognition networks, you're using something
called convolution. And I wanted to
give you a preview of what convolution looks like. So when I say convolution,
what I really mean is using a filter. And so on the left, we
have a gray scale picture of Manhattan. In the center we have a
three by three filter. And on the right, we
have another image that shows very roughly
the edges of where the buildings are. This kernel, or filter,
is a little edge detector that I wrote. It's very scrappy,
but if you write Python code to slide this
filter along the image and compute the dot
product of what's beneath it at each location,
it will produce that image which roughly shows the edges. So this filter is
hand engineered. But in convolutional
neural networks, we're going to learn a
hierarchy of these things. So here I've added
a line of keras that's going to learn
32 of these filters. And each of these
filters is going to be learned as a function
of the raw input image-- so as a function of the pixels. But if you have a stack of
these filters, what you're doing is you have-- the first layer is finding
features of the raw image. The second layer of filters is
learning features of features. And the third layer
is learning features of features of features. So the filters that you learn
become increasingly abstract. And you get this-- for free, by virtue of
training an image classifier-- you get this learned hierarchy
of feature-- of filters. And what I've done here is--
and you can repeat this with the code at the GooLink-- is I've started
from random noise, and I've optimized the noise
to increasingly excite filters at different layers
of the network. And the really
interesting thing-- and I think there's really
good science to be done here, and really understanding
this rigorously-- is that the lowest layer
filters generally learns to detect edges and colors. The next layer is
learning textures-- those are features
of edges and colors. And then as you move
higher up the network, you begin to learn
things like patterns. And if you go way high
up the network, which I'll show you later, that's
when you get psychedelic stuff. And so Deep Dream
doesn't have to be used to create trippy artwork. As you'll see in a
moment, I'm not an artist, but what I tried to
do is use Deep Dream to turn this image into
something like a firework display. And to do this, you
can actually select which filter you
want to optimize for. If you look at the Deep Dream
notebook at the GooLink link on the top, you're going
to see a whole lot of code. But the interesting
thing is that something like 90% of that code is fine
tuning, just to make sure that the images that are
produced are really beautiful. This is basically the core
of the Deep Dream algorithm, and it's really, really concise. So the basic idea
in TensorFlow here is that we're going
to pick some filter from some layer of the network. And we're going to
define a loss function. The loss is just a number. And here, the loss
is going to tell us, on average, how excited is
this filter by some image. So if we classify
any random image, the filter is going to
get some activation. And in Deep Dream, what
we can do is we can say, TensorFlow, please tell me
the gradient of this loss with respect to my image. And you're asking
TensorFlow, given an image, how should I modify that
to increasingly excite some filter in the network? And then what you can do is you
can literally modify the image according to the gradient. And here, I can actually
add it directly to the image and repeat. And so your image will
progressively incite-- excite a filter. And that's where Deep
Dream comes from. So here's another picture I took
from the reservoir in New York. And we're looking
south from 90th street. And I ran Deep
Dream on top of it using a filter at a very
high level of the network. And this particular filter--
actually, I'm sorry, I ran it on a whole layer
of the network, the second to last layer. And if you run Deep Dream on
an entire layer, what you're really saying is TensorFlow,
please progressively modify the image to
excite all the filters. So you're basically super
saturating it with things that it knows how to recognize. And that's why you see, if
you use Inception, that's why you see dogs,
and cats, and snakes, and stuff like that everywhere. But if you took Inception
and you retrained it-- say on artwork-- and then you ran Deep
Dream on top of that, you would no longer
have these dogs. Instead, you might
modify an image to contain elements of,
like, Monet and Picasso. So it's cool, but I think
there's good science there. I didn't want to
show you this here, but the notebook
actually contains code to produce your own
animated dreams. But it was borderline
not safe for work. Not in like a strange
sense, it was just-- was too psychedelic
and seizure-inducing. So there's another
absolutely beautiful thing you can do with
pre-trained image networks. And this is called
style transfer. And for people
new to it, I can't recommend this strongly enough. It is my favorite way to produce
really beautiful artwork. So what I want to
show you is something from a project called Magenta. There's going to be
a talk on Magenta tomorrow by Douglas Eck, who's
one of the project leads. Magenta is a wonderful project. And it investigates how
you can use deep learning to generate art and music. And they've done a great job. So here, in Style
Transfer, the idea is that you begin
with a painting. And here's one from the Museum
of Modern Art, in New York. And then you take a photograph. So here's one of the
Golden Gate Bridge. And the goal in
Style Transfer is to produce a new
piece of artwork that combines the style of the
painting with the content of the photograph. And by style, I mean things like
edges and shapes and colors. And by content, I mean things
that are semantic-- like here, I see a bridge. And so if you try
to do this by hand, or in Photoshop or whatever,
to produce a new painting, the result would
probably not be very good unless you're much better
at Photoshop than I am. But using Style Transfer,
you can actually generate images like this,
which are astoundingly good. And the reason I find
this so interesting is you're able to produce
images like this almost for free as an artifact of training
an image classifier. And in Deep Dream,
what we were doing is finding an image
that maximally excites some filter of the network. But in Style Transfer, we need
to somehow caption the notion of content and style. And here's a high level
diagram of how that works. We also get this for free. If you take, say, the
painting on the top and you use some
existing image classifier and you just classify the
painting, what you can do is look at the activations of
the lower half of the layers, roughly. Earlier, we said-- we saw
that the first layer detected edges, shapes, and
colors, and textures, say in the first three. And so when you
classify the image by looking at the activations
in those layers, in some sense, you can capture what
it means to have style. And then if you
classify the photograph, and you look at the activations
towards the top of the network, you have some sense of what
it means to have content. And then if you
combine those, you can actually come
up with a number that you can optimize
in TensorFlow. And that's how you actually
can start with random noise and progressively optimize
it to minimize a loss that combines content and style. And so at a high level
that's how it works, but I think the science
behind it is mind blowing. So this is a good example in
Deep Learning of something that never in 100 years would
I have thought that, ever, I could write a Python
program to produce images like that stylized bridge. It just never would
have happened. But it turns out that this
is something we get for free. And you'll see in
Douglas's talk, if you go-- which I
highly recommend-- you can actually do style
transfer in real time now. And here, what
they're doing is they have a collection of paintings. And they're saying, I'd like
to apply some more "Starry Night" to this dog. So that's Inception. Language is also,
obviously, really important. People that work in, like,
robotics or, like, video or images or games
have the best demos. So their work is
usually more well known, but obviously, almost all the
apps we write work with text. So here, I want to show
you a couple of models from the syntax net family. Actually just one, ParseySaurus. And ParseySaurus is currently
the world's most accurate natural language processor. And it's completely open
source in TensorFlow. So here's how we
can work with text. So imagine you just
have a simple sentence-- I love NYC. One thing you can do,
if you go the API route, is you can use something like
the cloud Natural Language API to analyze the sentence. It's drag and drop. There's a web demo, which
is actually really fun. And it's also RESTful,
if you write code. So here, we're
seeing the cloud API telling us that love
is a verb and it's looking at a dependency parse. So we can find out that
NYC is the direct object of a sentence. But you can also do this
in open source TensorFlow. And so here is a
single line of code to start up a Docker
container for syntax net. And this will contain the
ParseySaurus model as well as code for over 40
different languages. And using this, you can actually
do some really interesting things. So Parsi McParseface--
from last year-- McParseface was previously
the world's most accurate natural language parser. And McParseface has been
superseded by ParseySaurus. The main difference-- well, one
of the differences between them is that Parsi McParseface
operated on words. So the tokens that it
would analyse would be, the gostak distims. But ParseySaurus
operates on characters, so it's much finer grained. This is a quote from a
book called "The Meaning Of Meaning". And if you look at it, the
sentence is nonsensical. But what's really funny
is, for some reason, our brain actually knows a
whole lot about this sentence. So for instance, I could ask
you some questions about it. So what do you think the
part of speeches for distims? Is distims a noun? Is distims a verb? It's a verb. And it's funny that we know
that, because it's not a word. And the really
interesting thing is you can actually
use ParseySaurus to tell you that as well. So if you fire up
that Docker container, you'll find an ipython notebook
called the interactive text analyzer. And with just a
few lines of code, you can feed ParseySaurus
that sentence. You can get back
a bunch of tokens, which are protobuffs that
contain a whole bunch of information about the parse. You can iterate over it,
find the distims word, and print out the
part of speech tag. And ParseySaurus will tell
you that that's distims. You can also ask
the question who or what is doing the distimming? And is it the doshes that are
distimming or is it the gostak? And when I read this, I
see it-- it's the gostak-- is the thing that's
doing the distimming. And ParseySaurus can
tell you that as well. So if you iterate
over the tokens, we can find the
subject of a sentence. And you'll find out
that's the gostak. ParseySaurus can also
tell you a lot more about a sentence like this. So if you just print out the raw
protobuffs from all the tokens, you'll get a whole
slew of information. And only some of it is
actually on this slide. So it actually can give you
pretty fine-grained stuff. Like, for instance,
it can tell you that distims is in
the present tense. It can tell you that
the gostak is singular. And it can tell you that
the doshes are plural. And then the
Natural Language API has these beautiful
dependency trees, but you can actually
produce that as well. And so the notebook
actually contains code, using good old
graphviz, to produce something that looks exactly like this. And there you can
see that the doshes is the object of the sentence. So there's a lot of
value in open source. And the cool thing
about ParseySaurus is that multi-lingual support
is obviously really important. And I'm really, really impressed
by the syntax net team, as well as the Magenta team
and the Inception team, because out of the
box, they actually have a link where you can
download pre-trained models for 40 different languages. Or more than 40, actually. Loading these languages,
it's a little bit-- there's a few more
parameters, but it's really one or two lines of code. And so you can switch the
model to German just like that, and hear the languages
that I saw out of the box. So this is actually really
cool, because if you're building any kind of
application that works with text and is doing something
machine learning-y, you can think about
multi-lingual support from the start. There's two really good
use cases for this. The first is an easy
way to get started. And if you have an
existing system which is doing some sort of
text classification-- this could be
sentiment analysis, or you could be doing spam
detection, or whatever. And you're using part of
speech tags or dependency tags, you can use a model
like ParseySaurus to get a slight boost
in your accuracy. But I think more
importantly, you can use this as a
foundation for a new system. So anytime you're training
a model in Deep Learning, you're going to
need lots of data. And I think, by using these
features as additional input, you can reduce
the amount of data that you need because
you're not going to have to learn notions like
part of speech and dependency tags from scratch. Instead you get them for free. There's a really awesome term
that I found on Magenta's blog, and it's called Release++. And so this is some
text that I actually stole from their article, where
they're talking about something called Ensign, which is
actually a program that can generate sounds. And I thought this was a
really beautiful passage. So when they shared the code
for Ensign, they actually-- they explicitly called out-- sure, they're sharing the
code, but they're also sharing pre-trained models
and a Docker container. And because the team has done
such a wonderful job of doing this, it means that if you
want to try out their work, especially in style transfer,
you can do it out of the box, and it works very, very well. And it sounds simple,
but that's actually something really hard
to achieve in practice. Like, I can't tell
you how much time I spend playing with open source
software machine learning, whether it's written
in TensorFlow or a different toolkit. And it is so hard to
actually replicate -- results. And so I think some of the
work that I showed you today is good examples of work
that sets the bar on really how to share ideas. I think I've accelerated
too much in my speed-- in my rush to get
through all this content. But anyway, a
blessing and a curse of learning deep
learning today is that there are a huge number
of educational resources. There's actually an
entire medium blog post, where somebody--
actually very helpfully-- went through something like
30 different machine learning courses and looked
at all the reviews. So here just a couple
that I'd recommend. Stanford has a great
course-- it's actually student initiated, it's really
impressive-- on TensorFlow. CS20si-- it has an
intimidating title. The title is, like, "TensorFlow
for Deep Learning Research". But really a lot of this
course will actually walk you through the basics of
writing TensorFlow programs. If you're new to
neural networks, the course notes from
CS231n are outstanding. That course also has a
slightly intimidating title, its "Introduction to
Convolutional Neural Networks", but the beginning of
the class will actually walk you through-- just in Python, it doesn't
use TensorFlow at all-- really, what is
a classifier, how do you write one from scratch. And then it will take
you from something like k nearest
neighbors all the way up to a convolutional
neural network. If you like natural
language processing, although it's not on the
slides, Stanford also has, I believe, CS224n,
which is actually a course in doing
natural language processing with TensorFlow. And then there's
a wonderful blog. If you'd like to get some
intuition for how networks actually work under the
hood and why they're so good at recognizing images
and doing other things, Chris Ola's blog is
absolutely outstanding, and I highly recommend it. So thanks very much. One of my favorite
things about I/O is just actually
looking at the crowd. I see a bunch of folks that I
haven't seen for about a year. So I like this conference a lot. I'll be around if you
have any questions. And I'm happy to
chat afterwards. So thank you very
much guys, and I hope these resources are useful. [APPLAUSE] [MUSIC PLAYING]
