home

about

photos

computing

contact

syndicate

training

Metaclasses Demystified Saturday, January 10, 2009

This article originally appeared in the July 2008 issue of Python Magazine and was adapted from Jonathan LaCours CLEVERtrain
professional Python training services.
The word metaclass often strikes fear into the hearts of Python programmers. This article explores metaclasses in an
approachable way, in order to take the mystery out of metaclasses, and enable you to leverage metaprogramming
techniques to solve real problems.
There are few things in the Python world that are so misunderstood and feared as metaclasses. In fact, when I recently
was training a group of seasoned programmers about metaclasses, I asked the room what sprung to mind when they
heard the word metaclass. The students shouted out their feelings in chorus confusing, magical, and worst of all,
difficult.
Well, as I told my students that day, when it comes to metaclasses, Id recommend that we called upon the wise words
of Douglas Adams in his famed book The Hitchhikers Guide to the Galaxy.
Dont panic!
Once you take a closer look at how object-oriented programming works in Python, youll quickly discover that not only
are metaclasses easy, theyre also extremely useful. Here, well do our best to take all of the mystery out of metaclasses,
and show you that theres nothing to be afraid of!

Revisiting Classes
Most Python programmers are familiar with creating classes, and might even create or modify their own classes several
times a week in the course of their work. Many of us were even taught about classes and object-oriented programming
in the course of our education. But, when was the last time any of us really thought about classes in depth since we
were first introduced to the concept? Id wager that most of us take classes for granted, and dont really think about
what they are providing for us, and more importantly, how they provide it to us.
Understanding metaclasses can be greatly simplified by taking another look at Python classes, from a slightly different
perspective. In order to do this, lets pose a simple question. What exactly does a class do?
My first instinct when attempting to answer this question is to rely on my education, which unearths big, fancy,
computer-sciency words like encapsulation and abstraction. Its tempting to allow this to cloud our thinking, so lets
do our best to think at a much simpler level when considering this question. So, I ask again what does a class do?
Fundamentally, a class is used to construct objects called instances. This is essentially all a class does creates instances.
The process of creating instances using a class is called instantiation. For example, given a class Person, we can
instantiate it to create an instance of that Personclass.
class Person(object):
pass
jonathan = Person()
This instance is related back to its class in that the class is what constructs the instance. What do we mean by
construct? Well, when a class is instantiated, it constructs an instance by providing it with its namespace. We all
know from Tim Peters The Zen of Python that namespaces are a honking great idea, and classes are a great

example of why! The attributes in the namespace of a class are used to define the namespace of its instances, thus
providing those instances with behavior and state. Lets enhance our Personclass, to see how this works.
>>> class Person(object):
...
greeting = 'Hello'
...
...
def greet(self, who):
...
print self.greeting, who
>>> jonathan = Person()
>>> jonathan.greet('Readers')
'Hello Readers'
The Personclass now has two attributes in its namespace greetingand greet. When we instantiate our
Personclass, our instance is given both state, in the form of the greetingattribute, and behavior, in the form of the
greetmethod. In this way, a class is of critical and primary importance in defining how its instances will behave.
So, what does a class do? Lets summarize. In a nutshell, a class is an object with constructs instances through a process
called instantiation, therefore defining the namespace, behavior, and state of that instance.

Defining Metaclasses
Now that weve established clearly what a class is, we can confidently pose the question what exactly is a metaclass?
When sifting through documentation, the most common answer youll find to that question is short, but often difficult to
unpack:
A metaclass is the class of a class.
While this answer is certainly correct and concise, and tells us what a metaclass is, it still doesnt tell us what a metaclass
does. Lets look at an alternate answer to the question:
A metaclass is a class whose instances are classes.
I will concede that this answer is just as much of a mouthful as its counterpart, but it has the benefit of giving us a clue
as to what a metaclass does! In short, a metaclass constructs classes. We also know that a metaclass is just another class,
and weve just spent some time outlining what a class does. As a result, we should be able to build upon what weve
learned about classes earlier to determine a bit more about metaclasses.
Recall that a class helps define the behavior and state of its instances. Metaclasses provide the same basic capability to
classes, giving you the ability to change the way a class will behave at runtime. This technique is commonly referred to
as metaprogramming.
Having a definition of metaclasses and metaprogramming is useful, but lets take a deeper look by investigating an existing
metaclass.

The typeMetaclass
In Python, classes which inherit from objectare called new-style classes. All such new-style classes have a default
metaclass called type. We can prove that typeis the metaclass of objectthrough simple introspection in the
Python interpreter. Recall, a metaclass is the class of a class, so what is the class of object:
>>> print object.__class__
<type 'type'>
Just as we expected! The metaclass of objectis type, and thus all classes which inherit from objectwill be
provided with this metaclass by default. Classes that do not inherit from objectare called old-style classes and will
disappear in Python 3.0. While old-style classes also support metaclasses and metaprogramming, well focus on newstyle classes in this article for the sake of simplicity.
Typically, classes are defined using the classstatement in Python, as we saw in our earlier Personexample.
However, we have just learned that metaclasses create classes when they are instantiated. This means that we should be
able to define a class by instantiating the typemetaclass manually.

Lets define our original Personclass again, but this time, lets do it without using the classstatement by instantiating
the typemetaclass:
>>> def greet(self, who):
...
print self.greeting, who
>>> Person = type(
...
'Person',
...
(object,),
...
{'greet': greet, 'greeting': 'Hello'}
... )
>>>
>>> jonathan = Person()
>>> jonathan.greet('Readers')
'Hello Readers'
This method of creating classes is equivalent to using the classstatement, and reveals quite a bit about how
metaclasses work. The constructor for a metaclass expects very specific arguments:
1. The first argument is the name of the class.
2. The second argument is a tuple of the base classes for the class.
3. The last argument is a dictionary representing the namespace of the class. This dictionary contains all of the
attributes that would typically appear within the body of a class statement.
Now, weve seen a metaclass in action, and we know how to instantiate them to create classes. Armed with this
knowledge of the default typemetaclass, we can now tackle the much more interesting problem of creating our own
metaclasses.

Defining Metaclasses
Defining metaclasses in Python is as simple as creating a class that inherits from the built-in typemetaclass. The
constructor for our metaclass will take the same arguments as the constructor for the typemetaclass:
1. The class name.
2. A tuple of the class bases.
3. A dictionary representing the namespace of the class.
Constructors typically perform some action on their instances, so lets make our constructor set a flag on the instance
that we can inspect to make sure that our metaclass is being used.
class MyFirstMeta(type):
def __init__(cls, name, bases, ns):
cls.uses_my_metaclass = True
One important thing to note here is that the first argument to the constructor of a class is typically called self, as it
refers to the instance being constructed. It is conventional to name the first argument of a metaclass constructor clsas
the metaclass instance is actually a class.
Now that weve defined our metaclass, we know that we can construct a new class called MyClasssimply by
instantiating the metaclass:
>>> MyClass = MyFirstMeta('MyClass', (object,), {})
>>> print MyClass.uses_my_metaclass
True
This very simple metaclass has given us our first glimpse into the power of metaclasses. Within our metaclass
constructor, we gave our class some state in the form of a boolean attribute uses_my_metaclass. Metaclasses have
the power to add, remove, or modify any attribute of the class being constructed. Metaclasses will frequently add or
replace methods on their instances, based upon the data in the namespace of the class. Many Python object-relational
mappers use metaclasses to transform the attributes of a class into database table definitions, for example.
While you can certainly construct classes by manually instantiating custom metaclasses, it is much more convenient to
use the classstatment to create your classes. Python allows you to define the metaclass for a class by using the special
__metaclass__attribute in your class statement:
class MyClass(object):

__metaclass__ = MyFirstMeta
This is the preferred method of attaching metaclasses to your classes. An important thing to note about this syntax is
that while you are not manually instantiating the metaclass when defining your classes this way, the Python interpeter will
instantiate the metaclass. The metaclass instantiation will occur immediately after the class statement has been fully
executed. As a result, bugs in metaclasses often are triggered during imports. In a way, the class statement is simply
syntactic sugar for instantiating metaclasses!

Mystery Metaclass Methods

Before we move onto some practical examples of metaclasses, lets investigate metaclass definition a bit further. Our
Personclass defines a method called greetwithin its namespace. Instances of Personwill thus have an instance
method called greet. We know what happens when we define methods on a class, but what happens if we define a
method on a metaclass?
Methods defined on a class become instance methods. Since instances of metaclasses are classes, methods defined on
metaclasses become _class methods_. Lets take a look at this in practice:
>>> class MysteryMeta(type):
>>> ... def mystery_method(cls):
>>> ...
return 'I am a class method!'
>>>
>>> class Mystery(object):
__metaclass__ = MysteryMeta
>>>
>>> print Mystery.mystery_method()
I am a class method!
This revelation often surprises people, but is a logical outcome of the fact that metaclasses are simply classes which
produce classes. Many metaclasses utilize this capability of defining class methods, but it is often preferable to define
such class methods on a base class, which is easier to document and understand.
Weve now established what metaclasses are, how they work, and how we can define our own by inheriting from type.
But, what about practical use cases for metaclasses? Lets take a look at several examples of how we might come to use
metaprogramming in practice.

Example: The Enforcer

When I am teaching Python to programmers with strong backgrounds in bondage and discipline languages like Java, I
frequently hear the same complaints about Python. One such complaint is that class definitions are loose and you
cannot enforce the type of variables by declaring them up-front. In order to illustrate how to use metaclasses, lets define
a little library that will address this particular complaint.
Our goal is to create a base class called Enforcerthat will enforce the types of attributes on its subclasses. Lets say
we wanted to enforce that the nameattribte of our Personclass was a string, and that the ageattribute was an
integer. Attempting to set attribtues with the wrong type should trigger a TypeErrorto be raised, just like the Java
compiler would catch such errors.
class Person(Enforcer):
name = Field(str)
age = Field(int)
The first thing we need to do is define the Fieldclass, which we will use to hold onto the type of the variable were
attempting to restrict. Lets also give it the ability to validate whether or not a value is of the right type for that particular
attribute:
class Field(object):
def __init__(self, ftype):
self.ftype = ftype
def is_valid(self, value):
return isinstance(value, self.ftype)
Now that we have our Fieldclass, we need to create a metaclass that will look at the namespace of our class,
searching for Fielddefinitions, and then storing them in a dictionary for later use. Recall that the last argument to the
constructor of a metaclass is a dictionary containing the namespace of the class. We can loop through this namespace

to find Fielddefinitions:
class EnforcerMeta(type):
def __init__(cls, name, bases, ns):
# store the field definitions on the class as a dictionary
# mapping the field name to the Field instance.
cls._fields = {}
# loop through the namespace looking for Field instances
for key, value in ns.items():
if isinstance(value, Field):
cls._fields[key] = value
Our metaclass first attaches a _fieldsdictionary to the class itself. This data structure is where we will store Field
definitions for later use. We then loop through the items in the namespace looking for Fieldinstances, and finally we
store them in our _fieldsdictionary.
Next up is the Enforcerbase class itself. The Enforcerbase class first needs to attach the EnforcerMeta
metaclass weve just defined. This is a very common way for libraries to distribute their metaclasses, by defining a base
class to inherit from, rather than requiring users to even know that a metaclass is being used, or how to attach the
metaclass to their classes.
The second thing the Enforcerbase class needs to do is to override the __setattr__method. This is a special
method on Python classes that allows you to override the default attribute setting behavior on your Python objects. The
__setattr__method takes in the name of the attribute being set, and the value being set.
class Enforcer(object):
# attach the metaclass
__metaclass__ = EnforcerMeta
def __setattr__(self, key, value):
if key in self._fields:
if not self._fields[key].is_valid(value):
raise TypeError('Invalid type for field!')
super(Enforcer, self).__setattr__(key, value)
Our Enforcerclass first attaches the metaclass. Then, it overrides the __setattr__method so that it can watch
for field assignments. First, we check to see if the attribute being set is one of our defined fields. Then, we ask the field
definition if the value that is being passed is valid for the field definition. If it is not a valid type for the field, we raise a
TypeError.
The last line of the Enforcerclass is extremely important. This line is instructing the Python interpreter to call the
__setattr__implementation on the appropriate superclass definition, in this case object. Without this line,
attribute setting would fail on all Enforcersubclasses.
Lets try out our new creation from the Python interactive prompt:
>>> class Person(Enforcer):
...
name = Field(str)
...
age = Field(int)
...
>>> jonathan = Person()
>>> jonathan.name = 3
TypeError: Invalid type
>>> jonathan.age = 'Old'
TypeError: Invalid type
>>> jonathan.name = 'Jonathan'
>>> jonathan.age = 28
Our metaclass has completely changed the way that our class behaves, transforming it into a more rigid class definition
that keeps our Java-loving friends happy. While such restrictive enforcement is atypical in Python, it certainly shows how
powerful metaclasses can be.
This example also illustrates a common pattern for metaclasses, in which a class describes how it wants to behave in a
declarative way without actually writing any code to implement that behavior. The metaclass subsequently takes that

metadata, and uses it to reprogram the class. This is the essence of what metaprogramming brings to the table, and is
the most popular use-case for metaclasses.

Example: Auto Decorator

Since being given a special syntax in Python 2.4, decorators have become a very commonly used feature. A decorator is
essentially a wrapper around a function or method. In versions of Python prior to 2.4, decorators were applied by
manually replacing the method definition with a decorated version of the method. Starting in Python 2.4, decorators
could be applied using the now popular @decoratorsyntax.
Sometimes, I find myself in the situation where I need to decorate all of the methods of a class with the same decorator.
Lets create a metaclass that will simplify this process by automatically applying a decorator that is declared in the
namespace of the class:
import inspect
class AutoDecorateMeta(type):
def __init__(cls, name, bases, ns):
deco = ns.get('decorator', lambda f: f)
for key, value in ns.items():
# skip the decorator and constructor
if key in ('decorator', '__init__'): continue
# skip objects in the namespace that aren't methods
if not inspect.isfunction(value): continue
# apply the decorator
setattr(cls, key, deco(value))
Our metaclass first looks up an attribute named decoratorin the namespace of the class. This is the decorator that
we will apply to every method of the class. Note that if such a decorator cannot be found in the namespace of the class,
we define a fake decorator that will simply return the existing method definition.
Next, we loop through the namespace of the class looking for methods. If we encounter the decorator or the
constructor, we skip them, as we dont want to decorate our constructor or the decorator itself. We then make use of
the inspectmodule to determine if the value is a method, and if it is, we replace the method definition with a
decorated version of that method.
Lets put our metaclass to the test by creating a class that defines several properties. Typically, wed have to decorate
every method with the @propertydecorator, which could add a lot of verbosity to our class, depending on the
number of methods wed have to decorate.
class Person(object):
__metaclass__ = AutoDecorateMeta
decorator = property
def __init__(self, first, middle, last):
self.first = first
self.middle = middle
self.last = last
def name(self):
return '%s %s' % (self.first, self.last)
def full_name(self):
return '%s %s %s' % (self.first, self.middle, self.last)
def initials(self):
return '%s%s%s' % (self.first[0], self.middle[0], self.last[0])
Now that weve created our class, we should be able to try it out from our Python interactive prompt to see if all of our
methods were transformed into properties:
>>> mlk = Person('Martin', 'Luther', 'King')

>>> print mlk.name

Martin King
>>> print mlk.full_name
Martin Luther King
>>> print mlk.initials
MLK
Not only is this example useful in practice, it also illustrates another use case for metaclasses and metaprogramming
automating repetitive tasks that occur in class definition. This metaclass could be used to wrap every method of a class
in a transaction, to add debugging hooks, or even to synchronize all of the methods using a thread lock.

Cautionary Notes
Metaclasses are fantastically cool, and as weve seen, they can be pretty useful. The rise of object-relational mappers and
web frameworks have put metaclasses into use by an increasing number of Python users, and has increased the visibility
of metaclasses substantially. That being said, metaclasses are a feature of Python that must be used carefully. Because
metaclasses do their work at class definition time, bugs in your metaclasses can result in errors that are triggered at
import time. In addition, metaclasses are often hidden from the programmer behind a base class, which can cause
confusion.
Applied judiciously, metaclasses can be a great tool for solving problems, and give Python programmers the ability to
take advantage of metaprogramming techniques in their own code. I hope that this article has shed some light on what
metaclasses are, and has taken some of the mystery out of metaclasses.

Comment
1. Great article. I dont think I really understood meta classes until now. I also liked that you included a note on
how you could shoot yourself in the foot. Every cool feature has its draw backs, but most people dont include
these in their tutorials.
-Mark
Mark Roddy

1475 days ago

2. I frequently use metaclasses to keep track of a collection of subclasses so that I creating a new subclass of my
class can (often extend) existing behavior declaratively.
class LeafClassesMeta(type):
"""
A metaclass for classes that keeps track of all of them that
aren't base classes.
"""
_all_classes = set()
def __init__(cls, name, bases, attrs):
cls._all_classes.add(cls)
# remove any base classes
cls._all_classes -= set(bases)
Jason R. Coombs

1474 days ago

3. Ive read you examples but I have one question. It seems to me that in the examples you gave I could use
subclass to achieve the same result. For example create a class AutoDecoratorMeta and then use that to create
my class Person(AutoDecoratorMeta):
Would there be a some easy piratical case where metaclass would work but subclass would not.
Thanks,

Lucas
Lucas

1471 days ago

4. Thanks for the tutorial! This is by far the best descriptions of what metaclasses are, and how theyre used.
I agree with Lucas, though, that Im not clear on exactly WHY Id want to use them yet.
Fitzgerald Steele

1462 days ago

5. Jonathan, thanks for this post! I went to your talk at pyatl. It was a great talk: easy to follow, entertaining, and it
was my first exposure to meta classes in Python. I have made use of meta classes several times since then.
Toby Ho

1450 days ago

commenting closed for this article