Specifying the Encoding of a Template File¶

This is the most basic encoding-related setting, and it is equivalent to Python’s “magic encoding comment”, as described in pep-0263. Any template that contains non-ascii characters requires that this comment be present so that Mako can decode to unicode (and also make usage of Python’s AST parsing services). Mako’s lexer will use this encoding in order to convert the template source into a unicode object before continuing its parsing:

## -*- coding: utf-8 -*-

Alors vous imaginez ma surprise, au lever du jour, quand
une drôle de petite voix m’a réveillé. Elle disait:
 « S’il vous plaît… dessine-moi un mouton! »

For the picky, the regular expression used is derived from that of the abovementioned pep:

#.*coding[:=]\s*([-\w.]+).*\n

The lexer will convert to unicode in all cases, so that if any characters exist in the template that are outside of the specified encoding (or the default of ascii), the error will be immediate.

As an alternative, the template encoding can be specified programmatically to either Template or TemplateLookup via the input_encoding parameter:

t = TemplateLookup(directories=['./'], input_encoding='utf-8')

The above will assume all located templates specify utf-8 encoding, unless the template itself contains its own magic encoding comment, which takes precedence.

Handling Expressions¶

The next area that encoding comes into play is in expression constructs. By default, Mako’s treatment of an expression like this:

${"hello world"}

looks something like this:

context.write(unicode("hello world"))

In Python 3, its just:

context.write(str("hello world"))

That is, the output of all expressions is run through the ``unicode`` builtin. This is the default setting, and can be modified to expect various encodings. The unicode step serves both the purpose of rendering non-string expressions into strings (such as integers or objects which contain __str()__ methods), and to ensure that the final output stream is constructed as a unicode object. The main implication of this is that any raw bytestrings that contain an encoding other than ascii must first be decoded to a Python unicode object. It means you can’t say this in Python 2:

${"voix m’a réveillé."}  ## error in Python 2!

You must instead say this:

${u"voix m’a réveillé."}  ## OK !

Similarly, if you are reading data from a file that is streaming bytes, or returning data from some object that is returning a Python bytestring containing a non-ascii encoding, you have to explcitly decode to unicode first, such as:

${call_my_object().decode('utf-8')}

Note that filehandles acquired by open() in Python 3 default to returning “text”, that is the decoding is done for you. See Python 3’s documentation for the open() builtin for details on this.

If you want a certain encoding applied to all expressions, override the unicode builtin with the decode builtin at the Template or TemplateLookup level:

t = Template(templatetext, default_filters=['decode.utf8'])

Note that the built-in decode object is slower than the unicode function, since unlike unicode its not a Python builtin, and it also checks the type of the incoming data to determine if string conversion is needed first.

The default_filters argument can be used to entirely customize the filtering process of expressions. This argument is described in The default_filters Argument.

Defining Output Encoding¶

Now that we have a template which produces a pure unicode output stream, all the hard work is done. We can take the output and do anything with it.

As stated in the “Usage” chapter, both Template and TemplateLookup accept output_encoding and encoding_errors parameters which can be used to encode the output in any Python supported codec:

from mako.template import Template
from mako.lookup import TemplateLookup

mylookup = TemplateLookup(directories=['/docs'], output_encoding='utf-8', encoding_errors='replace')

mytemplate = mylookup.get_template("foo.txt")
print mytemplate.render()

render() will return a bytes object in Python 3 if an output encoding is specified. By default it performs no encoding and returns a native string.

render_unicode() will return the template output as a Python unicode object (or string in Python 3):

print mytemplate.render_unicode()

The above method disgards the output encoding keyword argument; you can encode yourself by saying:

print mytemplate.render_unicode().encode('utf-8', 'replace')

Saying to Heck with it: Disabling the usage of Unicode entirely¶

Some segements of Mako’s userbase choose to make no usage of Unicode whatsoever, and instead would prefer the “passthru” approach; all string expressions in their templates return encoded bytestrings, and they would like these strings to pass right through. The only advantage to this approach is that templates need not use u"" for literal strings; there’s an arguable speed improvement as well since raw bytestrings generally perform slightly faster than unicode objects in Python. For these users, assuming they’re sticking with Python 2, they can hit the disable_unicode=True flag as so:

# -*- encoding:utf-8 -*-
from mako.template import Template

t = Template("drôle de petite voix m’a réveillé.", disable_unicode=True, input_encoding='utf-8')
print t.code

The disable_unicode mode is strictly a Python 2 thing. It is not supported at all in Python 3.

The generated module source code will contain elements like these:

# -*- encoding:utf-8 -*-
#  ...more generated code ...

def render_body(context,**pageargs):
    context.caller_stack.push_frame()
    try:
        __M_locals = dict(pageargs=pageargs)
        # SOURCE LINE 1
        context.write('dr\xc3\xb4le de petite voix m\xe2\x80\x99a r\xc3\xa9veill\xc3\xa9.')
        return ''
    finally:
        context.caller_stack.pop_frame()

Where above that the string literal used within Context.write() is a regular bytestring.

When disable_unicode=True is turned on, the default_filters argument which normally defaults to ["unicode"] now defaults to ["str"] instead. Setting default_filters to the empty list [] can remove the overhead of the str call. Also, in this mode you cannot safely call render_unicode() - you’ll get unicode/decode errors.

The h filter (html escape) uses a less performant pure Python escape function in non-unicode mode (note that in versions prior to 0.3.4, it used cgi.escape(), which has been replaced with a function that also escapes single quotes). This because MarkupSafe only supports Python unicode objects for non-ascii strings.

Rules for using disable_unicode=True

• don’t use this mode unless you really, really want to and you absolutely understand what you’re doing
• don’t use this option just because you don’t want to learn to use Unicode properly; we aren’t supporting user issues in this mode of operation. We will however offer generous help for the vast majority of users who stick to the Unicode program.
• Python 3 is unicode by default, and the flag is not available when running on Python 3.