>> Hello. Yes, hi. Welcome to the latest in our
series of talks in advance topics on programming languages. Today we're very lucky to have
Thomas Wouters whose here to talk to us about features that are currently available in Python
but are advanced features. Last week, we had--we were very lucky to have Guido, that awesome
talk about upcoming features; they will be talking about existing features, right? Yes?
I wanted to--as always, this is the latest in our series of talks. I always want to make
a pitch so that people will actually give talks of their own. It's a really useful thing
to do. If you have specialized knowledge or if you have interesting knowledge, to share
that knowledge with as many Googlers as possible and potentially even the rest of the world.
So, please come and see me and we can set up a talk and you can give--yet another, in
our series of talks here. And with that, I will turn it over to Thomas who shall give
a very interesting topic. Thank you. >> WOUTERS: Is this on? Right. Although there's
an echo. So I'm going to give a talk about Advanced Python which is not an easy subject
in that there's a lot of Python that can be considered advanced. I'm not sure what your
level of advancement is so I'm going to cover basically everything from the start until
the end and we'll see how far we'll cover. These are the subjects I'll be covering. I'll
certainly be explaining what the Python design, objects, iterators and generators and hopefully
decorators. Everything after that is a bonus. I'm going to keep the pace pretty high, so
if there's any questions, just wave your arms and I'll stop and explain. But if everyone
can keep up then we can probably cover the more interesting topics at the end as well.
And if there's specific interest in any of the topics especially, for instance, Unicode,
I can skip over the advanced stuff and jump right to Unicode. So, first about Python,
Python was developed by Guido as a middle system language between shell scripting and
system programming. So it was intended to be easy to use by normal programmers but still
allow more complex structures than a shell script--scripting language. It turns out that
it's also convenient for library programmers. People doing the actual programming for end
users because you can hide a lot of smarts inside of objects or modules and just expose
a very simple API to end users. This has gotten progressively better with later releases of
Python. There are multiple implementations of Python; CPython is the one everyone uses
except for those that are developing the others. Jython is--has been around for quite awhile
it has--is actually in use by a lot of people but not as much as CPython. IronPython and
PyPy are still actively being developed although, IronPython is apparently very usable. It is
designed by implementation in that if you don't come up with an implementation of a
feature that you want implemented, it's not going to happen. But it's not a slave to the
implementation and it doesn't mean that if you implement an idea, it will get in. It's
still a very designed language. We have feature releases every one to two years. It used to
be every year or so and now, the last one was in the works for two years and there's
a lot of bugfix releases as well. An important note about the feature releases, if you do
not get any warnings in using a particular minor release, upgrading to newer minor release,
the one following that minor release will almost certainly work. Any--anything that
changes semantics or possibly breaks code will be warning for at least one release.
And bugfix releases have a similar requirement that you can always upgrade to a newer bugfix
release of the same minor version and nothing will break that wasn't already broken. Important
in realizing how Python works is that everything is runtime, even compiletime is actually runtime.
As you may know, there are cached files, .pyc files that are cached bytecode, that's basically
borrowed runtime, someone else's runtime that you can skip. Also important is that all execution
happens in a namespace and that namespace is actually a dict, it's a dictionary object
with keys and values. Modules, functions and classes all have their own little namespace
and they don't affect other namespaces unless they explicitly do so. And modules are executed
top to bottom. Script start running at line one and they run all the way down to the end
or until loops or whatever make them wait. So in that regard, it's very much like a script.
When you import a module, it does the same thing, it just starts executing line one and
runs on down until in the end and the import is done. And another import aspect is that
the def and class statements don't define a function or at compilation time, it's a
runtime thing. It actually happens when bad statement is reached. So some more about functions
and classes as well because they're important features. At compiletime which is runtime,
the code for the function will be compiled into a code object. It's a separate object
from whatever the rest of the code is compiled into. And then at def time when the def statement
is executed, it will be--the code object will be wrapped into a function object along with
whatever arguments it takes, any defaults to those arguments. And essentially def is
an assignment operation. If you look at the bytecode, you'll see that it just uses the
same bytecodes as normal assignment operations. So here's an example of a function. The red
part is the actual def statement that gets evaluated when it's reached including the
part in blue which is an argument default. That part gets evaluated at the moment of
def. So the Foo object is constructed or the Foo function is called, if it's a function,
at the moment the def is being executed and the green part has been compiled at that time
because compiletime has been in the past but it's not being executed until later which
is when you call the function. So this is when you call the function, there's an inner
function in there, so what happens is that the green lines you see here, they've been
compiled into their own code object and when you call func, the inner func gets created
at that time and again the default argument of the defaults for the three argument is
evaluated at the moment you call def. If you look after the definition of inner func, you'll
see that arg1 and arg2 are reassigned. That means that whatever arg2 was, will be assigned
to inner func's arc3 and not whatever it is at the moment you call inner func. However,
inside inner func, we also use arg1 which is using [INDISTINCT] scope which I'll be
mentioning later as well, which means that it will be using the outer functions arg1
and that will be whatever arg1 is at the moment you call inner func. So there's a very big
difference between those two uses. >> Question?
>> WOUTERS: Yes? >> That very final line, was that evaluating
inner func or [INDISTINCT] function? >> WOUTERS: No, it's just returning function
object. >> Okay, thanks.
>> WOUTERS: So it's not calling anything, only the parenthesis do actual calling. Yes?
>> [INDISTINCT] could you repeat the question... >> WOUTERS: Sorry? Yes.
>> [INDISTINCT] >> WOUTERS: The question was is the last line
actually evaluating inner func? It is not, it's just returning the function object that
is inner func. So class statements are very similar but also very different. A class statement
is followed by a block of code, a suite, that's executed when the class statement is reached
right away. It is however executed in separate namespace, a dict again. And then after that
code of block has been executed, the namespace is passed to the--or is used to create the
actual class object. Any functions that you define inside the code block then get turned
into methods by magic, basically. And--well, as people are already programmed in Python
will know the self argument of this, for Python, is passed implicitly when you call a method
but your method still have to receive it explicitly as the first argument. And an important thing
to realize is that the code block in a class is actually a normal code block. You can do
whatever you want in here, for instance, at the top I create a class attribute which is
just something stored in the class, result of calling func. I define a method which is
just a normal function but it takes self [INDISTINCT] argument and it assigns to an attribute of
self when it is called. I also define a helper function which just did some silly arithmetic
and call that helper function inside the class block itself to calculate or recalculate the
class header, and at the end I delete helper because otherwise, it would end up in the
class and since it doesn't have self [INDISTINCT] first argument, it wouldn't do very helpful
things. Other important parts about Python, variables aren't containers like in other
languages. They're not allocated memory that you can use to store data. People often say
about Python that everything is an object and they also sometimes say everything is
a reference. That's both true but both are not true when you apply them to variables
because variables are not objects and variables are not references. Variables are only labels.
Everything concrete, everything that allocates memory, everything you can toss around is
an object and whenever you hold an object, whenever you store it anywhere, you're actually
storing reference. You're not actually own an object, you only own references to an object.
Variables don't have type information, they don't have information on the scope or the
variable or how many objects live around or when they were created, the only thing they
do is refer to an object and that's it. If you want to look at it from an implementation
standpoint, it's a dict. It's a dictionary mapping names to objects, that's it. So, scopes
also related to namespaces. Python has a very simple two scope rule which is actually three
scopes. Names are either local or they are global or built in. Local means it's--it exists
in a current function or class or module but it doesn't exist outside. Global means it
exists in the current module, it doesn't mean exist everywhere, just in a current module.
And builtin is a special namespace that adduced by builtin functions that you can actually
modify. It's just a special module that you can modify yourself if you want to. When the
Python compiler examines a function and compiles a function, it keeps track of whatever names
you assign to and assumes correctly because by definition, those names you assign to are
local. You can change this by using the global declaration which is the only actual declaration
Python has. But don't do it too often, it's usually a sign that your code is incorrect.
All other names that the compiler finds are either global or builtin and the lookup is
it looks in global and then it looks in builtin. So if you have a global of the same name as
a builtin, all the code in the module will find the global of that name instead of the
builtin. You can actually use this on other modules as well. If you import a module and
then assign to an attribute of the module, you'll be assigning to a global name in that
module and you can mask whatever "itthings" is a builtin name by assigning to that attribute.
There's also a trick with nested scopes which were added later in Python. I think in Python
2.1 where you can--as I pointed out before, you can refer in an inner function to a variable
in an outer function but that is only read-only. You cannot assign to a name in an outer function.
This isn't really a mechanical problem in Python. It would be possible to assign if
we added it but there's no syntax to say and one I assigned to another--outer scope variable.
Global only assigns to a global name and not to outer scopes. Apparently, Python 3 might
be getting--Python 3.0 might be getting syntax for this. So I mentioned modules. Modules
are there to organize code. They're very convenient because they have their own namespace with
global names. They also keep the rest of the module tidy. They're always executed on first
import and then cached in sys.modules. This cache is just about everything that you import
in your program and it's also the only thing that keeps track of what you have imported.
So if you toss something out of sys.modules and import it again, it will get executed
again. Import is just syntactic sugar, just like--basically everything in Python is syntactic
sugar. It calls the __import__ builtin function. If you want to do an import over module whose
name you have in a string object, you use __import. If you want to replace __import,
you can just write your own function and replace the one in the builtin namespace. Another
trick is that sys.modules does not have to contain modules. It's a mapping of name to
whatever object you want import to return. So you can toss in any--any all object in
there and then importing that name will return your objects. Storing none in sys.modules
just means this module couldn't be found. So, if you want to prevent some module from
being imported, you can insert in sys.modules "none" under that name and then it'll raise
ImportError whenever you try to import it. Python objects in general are described under--in
various terms, Mutability is a common one. That means the object is changeable. List,
for instance, are mutable, Tuples are not. Mutability is very important in Python because
everything is a reference. If you have a mutable object and you end up changing it by accident
and changing it for everyone, that's rather inconvenient. So you have to keep in mind
whether objects are mutable or not. Fortunately, this mostly happens by accident correct, anyway.
Related concept is Hashability, whether you can take the hash of an object. In normal
operation, mutable objects are not hashable at most, immutable objects are hashable. For
instance, Tuples are hashable but they're only hashable if all their values are also
hashable. Hash--hashes are being used just for dicts and sets right now, so any dict
key has to be hashable in any set item, anything you want to store in the set has to be hashable
as well. But it's not inconceivable that more stuff uses hashes. And then there's the imaginary
abstract base classes that are often referred saying that an object is file-like or it's
a sequence or it's a mapping. Those are abstract concepts that are somewhat like protocols
or interfaces or whatever you want in another language but they're just informal. They're
just saying whenever an object is sequenced; it has the sequence operations implemented.
You can look in and you can iterate over it, etcetera. Some of them are overlapping, for
instance, sequences, mappings and files are all iterable so the iterable interface applies
to all of them. And actually all of the objects you define, all that you're defining is implementation
of syntax. Python defines syntax and you can plug in whatever implementation you want in
that syntax. And you don't do it by--in other languages you say you implement the less than
operator, in Python you say you implement the less than operation so that it can be
used even when you're not actually using the less than operator. Here's a short list of
hooks that Python supports. I'm not going to go over all of them. Most should be--should
be common enough. The one thing that's worth mentioning is conversions. You can define
a few hooks that define how your object gets converted to other types. That just works
for builtin types of course, because the hooks don't exist for arbitrary types. But most
arbitrary types that want to do something like this can just say if you have an __Foo
method, I'll call it to turn you into a Foo object if I want to. But it's not very common.
These things are not Python hooks. These things you cannot configure on your object. Assignment.
Assignment is an operation that changes the namespace and not the object. Since an object
is just the reference to an object, you have no control over where your references are
going. Another thing you cannot control are type checks. You cannot control when people
are comparing your type or doing instance checks, you cannot control what they get as
a result. Related identity comparison, the "is" operator, it checks whether two references
are pointing to or referring to the same objects, you have no control over that. "And" and "or"
and "not" are Boolean Operations and they just call the one truth value test that you're
operated--your object can implement and you have no control over what they actually return.
Some may know "and" and "or" in Python return, one of the two values were as "not" returns
always a Boolean. And Methodcalls, you cannot define in a single operation whatever will
happen when someone calls a method on you because methods are getting an attribute followed
by calling that object. They're two separate steps. So in order to implement methodcalls
in your object, you would have to implement getattr to return some in go-between object
and then have that go-between object do something when it's called. I'm sorry. So on to some
implementation details in C. This is just--applies to CPython of course. Python objects are implemented
in struct that holds various bookkeeping information like ref counts, the actual type that an object
is, as well as arbitrary C data. It can be pointers, it can be in C, it can be an array,
whatever you want. Types are what describes what an object is and how it behaves. There
are separate struct which is also a PyObject struct. The arbitrary data in a PyType struct
is the function pointers and various other things that describe the type. The PyObject
structs are not relocatable. You cannot move them around once you've given them out to
anyone else. It's a blog of memory that's allocated and that's it. If you want to move
it around, you have to destroy the object which means you have to get rid of all the
references to it. That also means that variable sized objects like lists that have a portion
that needs to be reallocated and might move around, are--just use an extra layer of indirection.
It's just a pointer stored somewhere in. And of course because it's C, it doesn't have
ref counting by nature so ref counting is on manually, whenever you receive a reference
you incref and when it's over, whenever you toss one out, you decref. It sounds simple,
it can get quite complicated but the Python C API is mostly clear enough that it's not
too hard once you get used to it. Another feature that is done reasonably manually is
weak references. Weak references are references that get informed when their object goes away
so that they can do some cleaning up and not crash. The weak references are just callbacks
basically in the PyObject struct arbitrary data. One of the major problems with ref counting
is reference cycles. That is two objects that refer to each other causing them both never
to be cleaned up. Two objects referring to each other as of course to the simple problem
and the complex problem is a huge loop of object referring to objects and everything.
Python has a cyclic garbage collector which keeps track of objects that might participate
in cycles, for instance, lists and dicts and walks them every now and then to find object
groups that are not reachable through anything but themselves and then it cleans up the entire
cycle. This is all done in C, if you write a C extension or a C type, you don't usually
have to bother with it too much. There are some methods you have to implement if you
want to participate in cyclic-GC, when you think you might be participating in cycles.
So in Python, when you have an object, what do you have? Well, you have an object that
has two special attributes, the __dict attribute which is a dictionary holding per object data,
and you have the __class attribute which refers to the class. And like in C, the class defines
behavior, and in newer Pythons the class is actually a PyType object and it all matches.
The way to define the behavior is not through function pointers because Python doesn't have
function pointers. It's with specially made methods. Methods that start and end with __,
those are specials Python. They're not special because they get allocated differently, they're
just special because some of them get called by Python automatically in some situations.
You can define your own __ methods and use them whenever you want. There's nothing special
on that regard, it's just the--some of them signals a Python, this needs to be called
whenever that happens. In general, you should not be calling another objects __ methods
yourself, not directly. You should always use them through whatever mechanism wraps
them. So for instance, if you want to have the string representation of an object, you
shouldn't be calling object .__str__, you should be just be calling StrObject. Another
feature of Python is that class attributes, that is attributes of the class object, are
also reachable through the instance. If you do the instance.attr and it's not defined
in the __dict, it would be fetched from the class and the class might have pair of classes
so they--those get searched as well. And of course in Python, the whole point of Python
is that you don't have to do stuff that Python can do for you, so refcounting, weak references
and cyclic-GC are all done automatically, you never have to worry about them. Typical
Python code does not use type checks because--partly because it was never possible until a couple
of years ago to sub class builtin types. So pretending to be a builtin type meant that
other people would not have to use type--must not use type checks or you could never pretend
to be a builtin type. It's also very convenient, it just means you can pretend to be whatever
type you want, implement the right methods and it'll just work. The C layer sometimes
needs specific types, if you want to multiply a string by an integer, it needs to have,
actually have a string and an integer or there won't be any multiplication in C. So the C
layer has conversions. When it wants an integer, there are special methods that say I need
an integer at this argument, it will convert whatever gets passed in--and it will convert
whatever get passed in to an integer or a string and do the multiplication that way.
If you really must use type checks, for instance, because you're faking C code or you're wrapping
C code or whatever, use isinstance instead of type. Checking type for a specific value
means you could--you only accept objects of that type where isinstance does proper instance
checks so that someone who does subclass whatever type you expect works the right way. Functions
become methods by magic in Python, it happens when you get an attribute off of an object
or rather when you get an attribute of a class through an object. Whenever you get an attribute
of a class that is a function, it get turn—-gets turn into unbound method which is a method
that knows it's a method and it knows which class it belongs to but it doesn't know which
instance, so when you call that method it knows that the first argument you passed must
be an instance of that class and then it'll complain if it's not an instance of the right
type. Of course that type check uses isinstance, so when you have the methods on bound--or
if you have an on bound method you can pass a subclass of whatever class it express--expects
and it works. Bound methods on the other hand are methods that know they're instance and
they will be passing that argument automatically so you call them without the first argument,
you start at the second argument and it all looks normal. So any questions so far? No?
All right, on to iterators then. [pause]
>> WOUTERS: So iterators in Python are really simple, they are helper objects for iteration;
they encapsulate, if you will, the iteration state and they're created and used automatically.
If you don't want to bother with them, you don't have to and it all just happens by magic.
If you do want to bother with them, you can do a lot of fun stuff with them, even more
so if you combine them with generators which I'll be talking about next. Iterators are
two methods, __iter__ and next, notice that there's no __ around next, because next is
actually intended to be called directly sometimes. So there's no __ around it or people would
think that they shouldn't be calling it. Because they're simple, iterators in Python are not
rewindable, they're not reversible, they're not copyable, none of that, they're the lowest
common denominator in iterators. There are however ways to write reversible iterators
if you want, you can just write your own iterator class and add methods to rewind or reverse
or whatever. You can also just nest iterators. And iterators themselves are iterable, they
are just—-actually an iterator is required in its own __iter method to return itself
or it wouldn't be an actual iterator. So in the example I have here, I create—-exclusively
create an iterator over range which is a list of number from 0 to 10, not including 10,
and then I call zip on it which takes a number of iterables and takes one element of each
iterable and wraps it in tuple and stores that in a list which it returns. So I created
a single iterator pass the same iterator to zip twice, and as you can see zip takes one
element of each iterator consuming the iterator as it goes and ends up with having two elements
at a time basically from the original list of zero through nine. So generators are a
convenient way to write iterators, they are lazy functions that get executed as you iterate
over the result basically. Generators use the yield key keyword instead of return, it
works very much the same way except after a yield statement execution continues they're--the
next time you call it or the next time you iterate. The function with yield in it will
return a generator, it will return an iterator that you can loop over and there's—-this
is terribly convenient for __iter__ methods because you can just write what you would
expect in it'll just work. In Python 2.5 generators were complexified and they can now also receive
data and--and you can use them to build co-routines very easily, nothing you cannot also do with
2.4 and earlier iterators, just more conveniently and with extra magic. There's a lot of generator-like
iterators in the iter tools module which is I think new in Python 2.2 or 2.3, there's
a whole bunch of stuff in there you can use to chain and filter and whatever with iterators
that are just really convenient for chaining their various combinations. So here's a generator
example, the bottom function is map, as you may--may know it accept--it only takes one
iterable that goes in map function, it creates a list of results from applying the function
to every item in the iterable--iterable. In the bottom, there's a two line function that
is the generator version and then there's a one line function that is the original map
implemented in terms of the generator. As you can see you just basically lose half the
function if you use a generator because you generate items--each item on the fly and you
return generator instead of an actual list, any questions?
>> [INDISTINCT] now you can do it like that? >> WOUTERS: Can I define what a co-routines
and how would I do it in Python? A co-routine is a routine that is basically like the generator,
it stops execution, passing data to someone else but where a generator returns results
to its caller, a co-routine returns results to someone else, another function. So you
can have two routines where they both consume the output of the other and then the end result
is handled data. How would you do them in Python? Well as I said you can do them in
Python 2.5 with the new sending data to a generator thing, before 2.5 you would write
a class that was itself an iterator and just write it in bits and pieces and--it wouldn't
be as convenient as co-routines in actual languages that have co-routines because you
don't have a single blog of code, you have a whole bunch of separate pieces of code that
get called at the right time, I wouldn't bother implementing them in Python right now. Maybe
with 2.5 outer 2.5 out and people getting used to the new stuff you can do with generators
will get actual coroutines in Python. Yes? >> [INDISTINCT]
>> WOUTERS: The question was if generators are lazy, can you write a generator that loops
infinitely and just keeps on yielding results as long as you ask for it? Yes, yes there
are... >> Then you just ask for finite number?
>> WOUTERS: Well it wouldn't—-do you ask for a finite number? You can if you use the
iter tools module, you can slice iterators, you can say up to so many items in this iterator
and it'll return in iterator that starts consuming the original iterator until the slice ends.
But you don't have to do that. The iterator is something you loop over, so if you have
a loop that loops over two things at once and it stops whenever the condition is met,
if you don't--and you can just loop over an infinite object and rely on the fact that
your loop will end itself for other reasons than that the iterator stops. There's actually
an infinite number of infinite iterators in the iter tools module like itertools.count
which starts counting in zero and just keeps on counting forever and ever, unless you stop
for some reason. [pause]
>> WOUTERS: So on to decorators. Decorators are syntax in Python 2.4 for a feature that's
been around forever which is function wrapping. Decorators are just funcs—-are functions
or callables rather, that take a function as argument and return replacement callable
or they can return the original function if they want. However, the syntax means that
it can get confusing if you have a decorator that also has to accept its own arguments
because now you have a function that should return a callable that should accept a callable
and then return the original callable, so we'll see some examples of that. Another problem
is that because functions are special when they're attributes of a class they become
methods, when you have the decorators that returns something that's not an actual Python
function but something that acts like a Python function for--in most regards, it won't get
turned into method unless you implement the same magic that methods are--that functions
have that turn them into methods which is __get which I'll maybe explain later. So if
you write decorators make sure their methods or make sure they're functions or they won't
get turned into methods. And anyway, as I said simple concept but the devil is in detail
and we'll see about that. Here's a simple decorator, it's a spam decorator that says
whenever the function that you decorate, the same function at the bottom is called, it
actually calls the wrapper function at the top which loops for ten times calling dif--original
function. So in this piece of code the original function get a called--gets called ten times
and there's no return value of the actual function call which is [INDISTINCT] then the
original either. Here's an example of a decorator with an argument, the decorator is spam ten
which is not--no longer--spam is no longer the decorator, it's rather the decorator generator
that takes a number which is a number of times to repeat. And then in spam, we define the
decorator function which accepts the function as an argument and then has a wrapper function
which codes the original function and then return--return wrapper and then of course
return the decorator from the decorator generator. So that will--looks--looks okay, I mean, it
takes some getting used to, the nesting and everything. But there's another problem, what
about interspection? Maybe Python users don't care about the name of their function but
some tools do, like Pydot for instance, which is the Python documentation tool, it actually
looks at function objects and looks the other name and their signature and their DocString
and whatever. And because we replaced the original function, when you asked for the
documentation of sing, it'll actually say, it's called the function wrapper and it has
a signature of *args and **kwargs and it has no DocString. That's probably not what you
want. Another thing is that if you have another decorator in there, you can chain the decorators
that changes attributes of the function; those changes will go away because you're not doing
anything with the attributes of the function. So, some people write code like this which
is the original spam with the repeats argument, with the decorator function in there, with
the wrapper function in there. And then after defining wrapper, we assigned the _name_,
the DocString, the _module_ and--and the dict of the original function back to wrapper so
that all those things will actually be the same for the new function and the old one.
And assigning dict like that actually works, you can--you can copy—-its not a copy it's
a reference assignment, the original function—-all of the original function dicts or attributes
will be accessible in the wrapper function and when you assign to either one of them
it'll--it'll appear in the other one as well, they just share their attributes space. Now
this is not the easiest way to write a decorator, so in Python 2.5 in the func tools module
there is a decorator for decorators that does this for you. So you have a decorator that
you apply to decorator or to decorator generators and then that decorator generated decorator
gets applied to whatever function you have. So as you can see, the devil is in the details,
they can get confusing somewhat quickly, any questions? All right, how are we for time?
>> You got [INDISTINCT]. >> WOUTERS: All right. New-style classes.
When I say new-style classes, when anyone says new-style classes they actually mean
all old new-style classes because they were added in Python 2.2 which was released I think
six or seven years ago something like that, they're old. And it's a unification of the
C types that I explained and the Python classes because before or actually still in classic
classes, instances and classes are distinct C types. There is a C type called class object
or class obj—-that implements all the magic that I talked about--about turning methods
into or turning functions into methods and there's the C type instance which make sure
that instances work as expected with the __dict and everything. So they're separate types
and if you ask for the type of an Int it will say it's an Int, but if you ask a type of—-about
the type of any class that tries to be an Int, it'll say "it's an instance". So you
have no way of checking all that. And another—-a problem with the original approach was that
you cannot sub class builtin types, so Guido worked on--in Python 2.2 on unifying C types
and Python classes and the end result is pretty good. You can mix them or match them and everything,
it worked good. But it only works because a lot of new general mechanisms were added.
They were necessary to bridge divide between C objects and Python types and things that
you assigned from Python have to be inserted in as a C data type in a C struct rather than
as a Python object pointer. Classic classes are still the default so if you write a new
class and you don't specifically make it a new-style class, it'll still be a classic
class and that was for compatibility reasons because there's a lot of stuff that's slightly
different between—-well a lot of stuff. There's a couple of things that are slightly
different between classic classes and new-style classes mostly with multiple inheritance and
you can check if any class or--actually any instance of a class is an instance of a new-style
class because it inherits from object instead of nothing. So you can do actually do is isinstance,
my thing object and you'll know that it's an instance of a new-style class. The first
of the mechanism that I am going to explain is descriptors.
[pause] >> WOUTERS: Descriptors are a generalization
of the magic that happened with functions to turn them into methods. A descriptor is
any object that lives in the class namespace that are class attribute space, so it's an
attribute of the class and that has __get__, __set__, __delete__ methods. It doesn't have
to be--have all of them I think, you can just do it __get or __set or __delete for specific
operations. Whenever an attribute is retrieved--attempted to be retrieved from an object whose class
has an attribute with a descriptor in it, those methods on the descriptor will be called
and the result of those calls will be passed back to the object. Same for setting, it'll
call the set method no result occurs and deleting they call the delete method. The delete method
is not called __del__ because that was already taken for some other hook apparently. It's
now the method behind--the mechanism behind methods. So if you want to have a function
like object that behaves the same way as functions do becoming a method, you can do that by defining
__get__ and it's also the mechanism behind property which is a trick of hiding accessories
behind normal attribute access. So here's an example of properties, I'm not going to
give an example of actual descriptors because it's too boring and you won't be using it
anyway, but here's a property. We have a class, we define the function get prop, it takes
a self argument even though it's not going to be a method, it takes a self argument,
it doesn't stop there and to return whatever the value is of the extra property. And then
we wrap it in property and store it in a local—-in a name that'll eventually be a class attribute.
Oh, I see I have an error right there. So we instantiate the class, I should have had
Foo instead of X there and then we do foo.prop, foo.prop calls to the get prop and because
it's a property, even though it's not a method because it's just a function inside the class
log, the property type knows that it needs to pass the instance for convenience, the
instance that it actually--is called on, onto the function that wraps it. If you look at
this you can say, "Oh, this can be a decorator, too," and it's true you can just say at property
at the top of def get_prop, except that property takes multiple arguments you can also pass
a set prop and the del prop if you want it, I didn't do it in this example for brevity,
but if you just have a get there, you can just say at the top at property instead of
prop = property(get_prop) at the bottom, any questions about this? All right. So the other
general mechanisms, kind of related, they're also descriptors, classmethods and staticmethods.
Before Python 2.2, Python only had "instance methods" that is normal methods, methods that
take self as the first argument, they got called in an instance and if you try to call
them on a class without having an instance, you get an error. So in the Python 2.2, we
got classmethods and staticmethods. Classmethods take class as the actual class objects as
the first argument and that's terribly useful, I'll show why in a moment. Staticmethods take
no magic argument and their not very useful even though Java and C++ programmers come
into Python often say, "Oh I need a staticmethod for this." Generally not, they're only useful
for one particular thing and I'll show that in a minute. So here's a classmethod. Again
it's--you can--if you're using Python 2.4, you can use classmethod at classmethod at
the top for the decorator syntax, if not you'll have to use it at the bottom. So say we have
a FancyDict, that sub class of dict, and we define a method to create a dict from a set
of keys with the single value, so we don't have to generate this list of key value pairs,
we can just say "generate it from keys". So what I have here is we create a list of that
key value pairs and pass that to the class and because it's a class method and gets the
actual class passed, we can call it on any sub class for FancyDict without anything,
in particular happening in the sub classes, and it'll create a sub class of FancyDict
instead of a FancyDict itself. So whenever you think, "Oh, I should have a staticmethod
and I'll do something with my own class," you should actually use a classmethod and
do something with the first argument. Now, this is a rather silly example because dict
already has this exact thing implemented. There's already a fromkeys method that is
a class method in the dict type and it's very useful whenever you sub class dict which is
not too often. Anyway at the bottom it's shown what happens when you use it. So staticmethods,
they're not very useful, the main use is protecting dependencies from becoming methods. When you
use dependency injection as I do here in the example, you don't know what you're actually
injecting into your class. If it happens to be a method or--if it happens to be a function
or something that does something magical when used as an attribute of a class, this won't
do what you want it to do; it won't do the same thing as calling sendmail.sendmail where
I'm now calling self.sendmail. So you can wrap it in a staticmethods to prevent it from
becoming an actual method. That's the only thing I've ever seen that makes sense for
using staticmethod. Although as we'll see later, Python actually has a staticmethod
itself which is a good example of something that should have been a classmethod. Another
new feature, __slots__ which is for omitting the __dict__ attribute for arbitrary attributes,
it basically prevents you from assigning arbitrary attributes to another object. It reduces memory
use because the dict isn't allocated and it's a more compact form to store the same number
of attributes as a dict, but it's not going to be much faster than a dict even for a few
attributes. The main reason to have it is when you want to emulate or actually implement
immutable Python classes like we add immutable types where you don't want attributes assigned
to them arbitrarily and then your well-—there's a tiny, tiny, tiny class for showing slots,
they're right there. If you actually try to assign to something other than self the value,
either in init or anywhere else, it'll actually throw an exception, except for stuff that's
already in the class but, I mean, __init the def statement won't be throwing an exception
of course, because Python knows that it's already there. Another new thing in Python
2.2 is the actual constructor. Before Python 2.2 there was just __init which is an initializer
and other constructor, it gets called after the object has been created and it's your
job to initialize it and set the right attributes and whatever, but the data is already there,
the objects are already there. So __new is called actually to construct an object, allocate
method--or allocate memory for it, make sure it's alright. In Python, it's used--actually
just for implementing immutable types because if you have an __init to set attributes, it's
too late because the types who are to be created so it can be immutable if you can assign to
it in __init. So you need to do it in __new. And this is the example of a staticmethod
that shouldn't actually be a staticmethod. It cannot be an instancemethod because its
job is to create the instance, so there's no instance to be a method of it. So, Guido
made it a staticmethod before he added classmethods in the development cycle of Python 2.2, I
think. It could have been a classmethod but, well it's too late now. It's the staticmethod
that takes the actual classes' first argument and you need to pass it around explicitly
whenever you call the super classes new. When you want to actually implement __new, you
generally always call object on __new or your super class __new to do the actual location
because there is no other way to allocate memory in Python. However, your __new method
or staticmethod can also return existing types, existing objects, you can just return any
object you want from __new whereas __init, either has to return self or return none because
there is nothing else--you can't actually change whatever it returns, __new you can
return whatever you want and that's the result of instantiating your class. There are--there's
one caveat there, when you return an instance of your own class, whether it's an actual
instance of your class or a sub class of your class, the __init is still called, even if
it's an already created object. That's because python can't know that you're __new is returning
an old object or a new object. So it always calls __init. Of course, if you return something
that's not your type, that's not a sub class of your type. It knows that it's already been
initialized, so it doesn't call under __init. So, here's an example of an __new, WrappingInt
which is an immutable type in Python. We set slots to the empty list so it doesn't get
arbitrary attributes. And then in __new we take the value which is whatever our constructor
was called with. We modular it so it doesn't go pass 255 or 256 or whatever. And then we
create--we actually create self by calling the parent class method. As you can see here,
it's a staticmethod because, even though we're calling it on a class, we're passing the class
we were passed as well exquisitely. Any questions so far? Yes.
>> You do you make a—-an object of view [INDISTINCT]. How do you define class? [INDISTINCT]
make sure if it is immutable. >> WOUTERS: How do you create class and make
sure it's immutable? By not providing any of the things that are--that mutate to the
object. So, for instance, this is an easy example because Int is its own value, so you're
not storing anything in the value. We don't have—-we don't accept arbitrary—-attributes
and we let our parent create the object and it's done. If you want to not sub class ability
to immutable type, you have to do a lot of more work because you need somewhere to store
the data and then provide properties to read the data but not write the data. That's basically
how you create. So, you do the same thing as here and you have some magic in here that
sets a hidden variable basically, that then properties can't get out but nobody else can
get right access to. It's not easy and it's not, usually not worth it. Mostly Python classes
just are--just implemented in terms of existing types and if you want an immutable type you
either want something that is int like, string like, or tuple like and you can just sub class
in string or tuple and be done with it. Any other questions? Alright, is there any interested
in Metaclasses? So I mentioned them--alright. So, Metaclasses are these theoretically, satisfying
and mind boggling concept where you can draw this graphs between what the classes and it's
Metaclass and what the class of the Metaclasses and then what instance-—where object its
an instance of the--the general ideas that the Metaclass is the class of a class. It's
the type that are classes, it's whatever implements a class. And of course the Metaclass is an
instance of something, so the Metaclass has a Metaclass and they are all in Python, the
base Metaclass is called type. And type's type is type. And type is actually an object;
the parent of the type of—-parent class of type is object. All objects have a base
class that's object, so you can see how it gets confusing. Of course, the type of object
is type, so you can draw very funny graphs that way, but it's all, you know, it doesn't
matter because in python everything is simple and you can just--you can just say, the class--the
Metaclasses, the class that creates the class. So, of course it doesn't apply to type or
object because they are created secretly in python by magic, but every other class you
define is Metaclasses that whose job it is to create the class from that name space that
is the cold walk of the class. So, we go [INDISTINCT], yes well, if we go back to all the way up
to the class, here, this is all done before the Metaclasses is even looked at. And then
when this piece of code, the blue parts and the green parts are all compiled and nicely
wrapped up in the name space executed, nicely wrapped up in the name space, then that dict
is passed to the Metaclass along with a name and the parents whatever you want to sub class.
And it says, you know, "Create me a class and return it." And then the result of that
instantiation is your new class. So, it's practically simple. And whatever
you want to use it for in Python you can use it for the stuff that you normally define
in a class to define how an instance of the class behaves. You can do the very same thing
with a Metaclass and it'll define how the class behaves. So the magic that creates,
for instance, functions which is hidden in the class and stuff that calls descriptors
which is hidden in the class is actually called by get at or get attribute which is—-I should
probably should have mentioned. __new and __init are called to construct the class in
to initialize the class, that all happens to same way that you would expect. So you
can overwrite them and you can do as many things as you want, the thing there most useful
for is post processing a class just doing some wonderful renaming or mangling or other
stuff with a class after it's been defined, before it's visible to any other Python code
without requiring an explicit post processing step. As I said, you can do evil stuff in
get at or get attribute if you want. It's probably not worth it. It'll just make your
code much more complex. So here's a Metaclass example in Python. We--sub plus type because
it's convenient and I think it's necessary as well. You define in __new which is the
staticmethod as usual. I forgot to mention, you don't actually have to explicitly make
new staticmethod but you can, if you want to. The under--the Metaclasses __new gets
passed—-the class that is actually ourselves because it's a staticmethod of ourselves or
a class method of ourselves. The name of the class that needs to be created, the basis
which is the tupule of the basis that are named in the class statement and the dict
that is the name space that--of the code object, it's just a dict with attributes. So what
we do here is we go over all the attributes, we skip any of that start with __ because
you don't want, by accident do the wrong thing with the magic methods. And then we name mangle
whatever attributes are leftover by appending or prepending Foo under--to them to make them
look funny, we delete the original attribute and then we--at the end we call the base class
new with the same arguments but with the modified attributes. And then to use it, to use the
Metaclass you have to explicitly tell Python to use a different Metaclass than whatever
is the default which is type. You do __metaclass__ is MangledType in the class dict or if you
want at the top of the module. Now Metaclasses is getting heritage, so if you sub class mangled
class you automatically get the mangled type as Metaclass. If you want to sub class and
have a different type your—-or different Metaclass--your Metaclass has to be a sub
class of your super classes Metaclass. So if I wanted to have a more mangled class with
a more mangled Metaclass, I have to sub class mangling type to get a more mangling type
and have that as my Metaclass. So, any questions there?
>> Are mangle type [INDISTINCT]? >> WOUTERS: Yes, sorry that's a typo. The—-it
should say mangling type at the bottom and not mangled type. Yes.
>> I think I remember that django is a Metaclass at the bottom [INDISTINCT], is that true?
>> WOUTERS: Yes. >> Do you know how would it do?
>> WOUTERS: Yes. I don't have an example right now. I have some django code and it's very
interesting how it works in django. Sorry. The question was django uses Metaclasses and
how is that done. Django has an API where you define a class with various attributes
to describe your data model. And then you can have some extra classes inside the class
to describe admin options and display options and whatever. What it does is it actually--just
like this, it goes over all the attributes that were the result of executing the code
inside the class statement and examines them. And the order doesn't matter to django for
the most part but where it does order it abuses the fact when compiling a class for executing
a class, it executes top--top to bottom. So it calls a field, let me see, you do--a field
is a particular type and the type is a--is an instantiation of a class. So you say, "field
is in field" and "field two is car field". And it keeps track of--in what order are those
type were created by keeping a global counter for instance, so I'd know which order the
fields are in the class statement. That's about the only magic thing that the django
thing does. And then, for the rest is just--is just examining whatever's in the others dict
that gets passed in the Metaclass to write down the sequel that's whatever the database
backhand needs to store and whatever options there are, etcetera. Does that answer your
question? >> Yes.
>> WOUTERS: Alright. Any other questions? All right. So, I'm going to cover multiple
inheritance unless everyone here goes, "No, no, don't ever use multiple inheritance,"
alright? So, multiple inheritance in Python and in other languages is something that's
frequently argued whether it's good enough or sane or insane. Well, it's generally not
a good idea but occasionally, especially in python, it could make stuff a lot easier.
New style objects have a different way of doing multiple inheritance in that they usually
C3 method resolution order which is an algorithm described, I think in a Dylan paper, describing
how to do when you have multiple inheritance, in particular, involving diamonds where multiple
super classes inherit from the same super, super class, how to handle that correctly.
And the algorithm is pretty simple, it just goes a depth-first, left-to-right search through
all the classes, but then it removes all duplicates except the last one. So, if a class appears
two times, it'll be visited, not the first time it appears, but the last time it appears.
And in Python, we also have a convenience--contingency object called super which can help you continue
method resolution... All your parent classes are visited after you're visited are therefore
visited before you're visited. But, you might—-they might not be the next phase class, if you're
super class might not be the next super class to be visited and your sub classes might not
have been visited right before you. That's rather important to realize. So calling your
base class method directly saying "my parent.foo" whatever, is never right. Because there's
always going to be some situation where that will do wrong thing and skip classes when
visiting methods. So, the way to do it in Python is you have a single base class with
a full interface of whatever your tree is going to implement, so that any object can
always call whatever method they want to call within that tree on any other class within
that tree. But--in usual cases those--those functions will probably be do nothing functions.
They shouldn't raise errors because then you can't safely call them all the time but they
should—-if anything, they should do nothing. The signature of those methods should never
change because you cannot know which order of classes will be called in. If you have
to change a signature of a method in a particular part of the multiple inheritances tree, you
should have a second master basically, in the multiple inheritance tree and make sure
that it's a separate section of the multiple inheritance tree. And then you should have-—you
should use super everywhere you want to have anything from a base class, anywhere. And
that can be annoying because all of your code has to follow it in all of the classes and
you're usually not the only one developing all those classes, so you have to convince
everyone to use super everywhere. And as I can show here, using super is not actually
convenient right now. You call super, passing this class, the current class the class you're
coding in, and the instance or class if you want-—if you have class methods that you
were passed, you were called on. That's--that returns a proxy object and then you can, you
can use that as if it was self to call the original method. It's not too convenient and
I hope we'll have syntax for doing super calls in python 3.O, maybe sooner, but, I'm not
holding my breathe. Any questions about these? All right. I'm going to cover Unicode then
if we have time. >> [INDISTINCT]
>> WOUTERS: This [INDISTINCT] >> [INDISTINCT]
>> WOUTERS: So, Unicode, it's somewhat longer topic though.
>> [INDISTINCT] >> WOUTERS: No, just twice as long as the
previous topic or whatever. So, Unicode is a standard for describing characters. The
way byte strings describe bytes, you say, "A is represented by 65", a Unicode you say,
"A is code point 65" and--there's no relation between Unicode as such and bytes on the disk.
In Python that means the old store type that string holds bytes, it's a byte string, we
call it a byte string nowadays. And Unicode object or Unicode strings they hold characters
which is for ASCII is actually the same thing. But for non ASCII, it's entirely different.
A Unicode has no on disk representation as such, so if you want to store Unicode in disk
or even in memory, you need to use an encoding. Python internally uses either UTF-16 or UCS-4
encodings or UCS-2 and UCS-4 depending on how you exactly define what Python does. But
it uses either 2 or 4 bytes to represent each character, whereas, bytes strings they always
use 1 byte for every character or byte. When you had a byte string and you want to have
a Unicode string, a Unicode object, you have to decode the byte string into Unicode. And
if you--if you have a Unicode string and you want to store it somewhere, you want to pass
it over to the network or when you write it to disk; you have to in code Unicode into
bytes. A convenient encoding is UTF-8, and some people confused UTF-8 with Unicode, for
instance, Postgres, the database, has an encoding code Unicode which is actually UTF-8 and not
Unicode. It's one of those mistakes many people make but UTF-8 is not Unicode. UTF-8 is a
Unicode encoding and it's Unicode encoding that looks like ASCII to all Americans who
are people who don't care about accents or funny characters. But it can actually encode
all of Unicode and it does so by basically screwing Chinese and Japanese people by having
all of their characters be like 7 or 8 different bytes. So, in Python, Unicode is pretty convenient,
except for--it mixes the strings. You can have Unicode literals which look just like
string literals except you have a couple of more escapes besides the normal backslash
X escapes and backslash O, and backslash 0, I mean. You can have backslash U which is
a short Unicode escape and a backslash capital U which is a long Unicode escape. The short
U has, as you can see, 2 bytes and then the long U has 4 bytes. And the long U isn't very
necessary until you start working with the higher code planes that were added last to
Unicode and anything. Also, instead of Chr, the heart to create a single character, you
have Unichr which creates from a number or any character, any Unicode character. And
we support in the compiler at compile time Unicode names. You can use backslash and then
in the curly braces, the name for any Unicode code point. Then the code defines all these
names, we have them all in the interpreter at compile time, that actually results in
a single character Unicode string with a euro sign in there. It's a single character Unicode,
but when you encode it in—-in an encoding that supports the euro sign, it'll look actually
an entirely different character or multiple bytes or whatever. If you want to do this
name look ups in reverse, if you have a character you want to look up in the Unicode name, the
Unicode data module does that for you. If you have the name and you want the actual
character, Unicode data does that, too. So Unicode objects, they behave exactly like
strings, it's very convenient, you can slice them and you actually slicing characters instead
of encoded data. The length is right, you can iterate every character, everything is
great, except when they mix with non-ASCII byte strings. When they mix with ASCII byte
strings, Python will automatically upgrade the byte string into a Unicode object which
is with the ASCII encoding. So that works for ASCII bytes strings, but if there's a
funny character in the byte string that's not ASCII, it'll blow up, because it tries
to interpret it as ASCII and it sees that it's actually not ASCII and it doesn't know
what you want to do with it, so don't do that. Another problem with the python approach is
that the decode and encode methods of strings and Unicode objects are generalized. They
don't just do encoding to byte string or decoding into Unicode object, you can actually convert
strings into strings and byte strings into byte strings and integers into whatever you
want or two integers. It's--it's inconvenient and I'm not entirely sure if that should be
fixed or not, but it's inconvenient when you only care about Unicode. On the other hand
they do have convenient features. So, using Unicode in python is very simple. Never mix
Unicode objects and byte strings. It's a very simple rule, if you do that everything would
be great, except of course it's not always easy not to mix byte strings in Unicode. If
you write libraries, you might get pass the string when you don't expect it or you get--might
get pass a Unicode object when you don't expect it. If you have your application--if you write
your application, you have to make sure that anywhere you get a string; you get it as Unicode
object or you get it as a byte string and you translate it yourself. So decode--the
best way to do it is decode byte strings into Unicode when you get them and in code Unicode
into whatever output you have when you're outputting. And of course you have to remember
to use the right encoding, so you have to remember what the encoding would be like when
you get input or should output and there's no way to detect and encoding. Because it's
just bytes and there's no marker in there that says "this is UTF-8" or "this is a latin-1"
or whatever, or UTF-16 for that matter. And it all looks vaguely familiar when you're
actually looking at the bytes, but that might not mean that's the correct thing to decode
with that encoding. Fortunately, if you can figure out which encoding to use Python does
have some conveniency functions and modules, the codecs module, in particular. Codecs module
has an open function that behaves like the builtin open function, except it takes on
encoding and it'll automatically decode data as a—-as if you just read from it. So when
you read from codecs with open objects, you're actually reading Unicode and you write Unicode
to it it'll automatically encode it into whatever encode you passed in. If you want to wrap
existing streams, like sockets or files you partially read, you can use codecs to get
reader or get writer to transform the on the file transform the string from byte string
to Unicode or the other way around. And lastly, when you do write stuff like it and you're
debugging your code and there's some problem with mixing Unicode and byte strings, pay
attention to the exceptions because there's two exceptions you could get; there's Unicode
decoder which you get when decoding a byte string into Unicode goes wrong and there's
Unicode encoder which is the other way around. And if you use str.encode, so you're trying
to encode a byte string into a Unicode encoding, what it'll actually do is silently try to
decode the string first into Unicode and then encode it with the encoding you gave it. So,
that actually tries to apply the default encoding which is ASCII to str, and even though you
call str at encode, you will be getting a decode error if it ends up that str is not
an ASCII string. So, that was it. That was all my topics I'm glad we went over all them.
Here's some more information if you want; descriptors, metaclasses and super all describe
in Guido's original descrintro tutorial for python 2.2 which is still very relevant. Iterators
and generators are well described in a--under kooklinks tutorial on functional programming,
you don't have to follow the whole thing if you don't like functional programming, but
the parts about iterators and generators are very good. And if you want to learn about
writing C types, the standard documentation is a great source, as well as the Python's
source, the actual Python modules are all in the same API and they're very readable
source even if I do say so myself and it's highly recommended. And always the python
standard library and the Python C code are all great sources. That was it. Any more questions?
>> [INDISTINCT] somewhere, that we can get up?
>> WOUTERS: I can put it up somewhere. >> How about the previous presentation about
the upcoming [INDISTINCT]. >> WOUTERS: Sorry?
>> The previous presentation, I guess, it was [INDISTINCT] wheel about the feature of
Python? Is there any record of that somewhere that we can look at up?
>> WOUTERS: There's like a four or five different movies, sorry.
>> It's on my laptop you can upload it. >> Okay great.
>> WOUTERS: Any other questions? >> What's a [INDISTINCT]-—what's the good resource
for a sort of module import resolution? You know, like, when you're changing—-when you're
moving Python [INDISTINCT] from one place to another. [INDISTINCT] and is there, like
a, standard way of how you do of all that. >> WOUTERS: So, you mean from one string to
another? >> From, you know, one string to another or
what [INDISTINCT] code base [INDISTINCT]. You start mixing it [INDISTINCT] and things
like that or in like [INDISTINCT] libraries. It's got to be like when you do [INDISTINCT]...
>> WOUTERS: Usually a byte store...
