Notes about supported databases¶
Django attempts to support as many features as possible on all database backends. However, not all database backends are alike, and we’ve had to make design decisions on which features to support and which assumptions we can make safely.
This file describes some of the features that might be relevant to Django usage. Of course, it is not intended as a replacement for server-specific documentation or reference manuals.
PostgreSQL notes¶
Transaction handling¶
By default, Django starts a transaction when a database connection is first used and commits the result at the end of the request/response handling. The PostgreSQL backends normally operate the same as any other Django backend in this respect.
MySQL notes¶
Django expects the database to support transactions, referential integrity, and Unicode (UTF-8 encoding). Fortunately, MySQL has all these features as available as far back as 3.23. While it may be possible to use 3.23 or 4.0, you’ll probably have less trouble if you use 4.1 or 5.0.
MySQL 4.1¶
MySQL 4.1 has greatly improved support for character sets. It is possible to set different default character sets on the database, table, and column. Previous versions have only a server-wide character set setting. It’s also the first version where the character set can be changed on the fly. 4.1 also has support for views, but Django currently doesn’t use views.
MySQL 5.0¶
MySQL 5.0 adds the information_schema
database, which contains detailed
data on all database schema. Django’s inspectdb
feature uses this
information_schema
if it’s available. 5.0 also has support for stored
procedures, but Django currently doesn’t use stored procedures.
Storage engines¶
MySQL has several storage engines (previously called table types). You can change the default storage engine in the server configuration.
The default engine is MyISAM [1]. The main drawback of MyISAM is that it doesn’t currently support transactions or foreign keys. On the plus side, it’s currently the only engine that supports full-text indexing and searching.
The InnoDB engine is fully transactional and supports foreign key references.
The BDB engine, like InnoDB, is also fully transactional and supports foreign key references. However, its use seems to be deprecated.
Other storage engines, including SolidDB and Falcon, are on the horizon. For now, InnoDB is probably your best choice.
[1] | Unless this was changed by the packager of your MySQL package. We’ve had reports that the Windows Community Server installer sets up InnoDB as the default storage engine, for example. |
MySQLdb¶
MySQLdb is the Python interface to MySQL. Version 1.2.1p2 or later is required for full MySQL support in Django.
Note
If you see ImportError: cannot import name ImmutableSet
when trying to
use Django, your MySQLdb installation may contain an outdated sets.py
file that conflicts with the built-in module of the same name from Python
2.4 and later. To fix this, verify that you have installed MySQLdb version
1.2.1p2 or newer, then delete the sets.py
file in the MySQLdb
directory that was left by an earlier version.
Creating your database¶
You can create your database using the command-line tools and this SQL:
CREATE DATABASE <dbname> CHARACTER SET utf8;
This ensures all tables and columns will use UTF-8 by default.
Collation settings¶
The collation setting for a column controls the order in which data is sorted as well as what strings compare as equal. It can be set on a database-wide level and also per-table and per-column. This is documented thoroughly in the MySQL documentation. In all cases, you set the collation by directly manipulating the database tables; Django doesn’t provide a way to set this on the model definition.
By default, with a UTF-8 database, MySQL will use the
utf8_general_ci_swedish
collation. This results in all string equality
comparisons being done in a case-insensitive manner. That is, "Fred"
and
"freD"
are considered equal at the database level. If you have a unique
constraint on a field, it would be illegal to try to insert both "aa"
and
"AA"
into the same column, since they compare as equal (and, hence,
non-unique) with the default collation.
In many cases, this default will not be a problem. However, if you really want
case-sensitive comparisons on a particular column or table, you would change
the column or table to use the utf8_bin
collation. The main thing to be
aware of in this case is that if you are using MySQLdb 1.2.2, the database backend in Django will then return
bytestrings (instead of unicode strings) for any character fields it returns
receive from the database. This is a strong variation from Django’s normal
practice of always returning unicode strings. It is up to you, the
developer, to handle the fact that you will receive bytestrings if you
configure your table(s) to use utf8_bin
collation. Django itself should work
smoothly with such columns, but if your code must be prepared to call
django.utils.encoding.smart_unicode()
at times if it really wants to work
with consistent data – Django will not do this for you (the database backend
layer and the model population layer are separated internally so the database
layer doesn’t know it needs to make this conversion in this one particular
case).
If you’re using MySQLdb 1.2.1p2, Django’s standard
CharField
class will return unicode strings even
with utf8_bin
collation. However, TextField
fields will be returned as an array.array
instance (from Python’s standard
array
module). There isn’t a lot Django can do about that, since, again,
the information needed to make the necessary conversions isn’t available when
the data is read in from the database. This problem was fixed in MySQLdb
1.2.2, so if you want to use TextField
with
utf8_bin
collation, upgrading to version 1.2.2 and then dealing with the
bytestrings (which shouldn’t be too difficult) is the recommended solution.
Should you decide to use utf8_bin
collation for some of your tables with
MySQLdb 1.2.1p2, you should still use utf8_collation_ci_swedish
(the
default) collation for the django.contrib.sessions.models.Session
table (usually called django_session
) and the
django.contrib.admin.models.LogEntry
table (usually called
django_admin_log
). Those are the two standard tables that use
TextField
internally.
Connecting to the database¶
Refer to the settings documentation.
Connection settings are used in this order:
DATABASE_OPTIONS
.DATABASE_NAME
,DATABASE_USER
,DATABASE_PASSWORD
,DATABASE_HOST
,DATABASE_PORT
- MySQL option files.
In other words, if you set the name of the database in DATABASE_OPTIONS
,
this will take precedence over DATABASE_NAME
, which would override
anything in a MySQL option file.
Here’s a sample configuration which uses a MySQL option file:
# settings.py
DATABASE_ENGINE = "mysql"
DATABASE_OPTIONS = {
'read_default_file': '/path/to/my.cnf',
}
# my.cnf
[client]
database = DATABASE_NAME
user = DATABASE_USER
password = DATABASE_PASSWORD
default-character-set = utf8
Several other MySQLdb connection options may be useful, such as ssl
,
use_unicode
, init_command
, and sql_mode
. Consult the
MySQLdb documentation for more details.
Creating your tables¶
When Django generates the schema, it doesn’t specify a storage engine, so tables will be created with whatever default storage engine your database server is configured for. The easiest solution is to set your database server’s default storage engine to the desired engine.
If you’re using a hosting service and can’t change your server’s default storage engine, you have a couple of options.
After the tables are created, execute an
ALTER TABLE
statement to convert a table to a new storage engine (such as InnoDB):ALTER TABLE <tablename> ENGINE=INNODB;
This can be tedious if you have a lot of tables.
Another option is to use the
init_command
option for MySQLdb prior to creating your tables:DATABASE_OPTIONS = { "init_command": "SET storage_engine=INNODB", }
This sets the default storage engine upon connecting to the database. After your tables have been created, you should remove this option.
Another method for changing the storage engine is described in AlterModelOnSyncDB.
Notes on specific fields¶
Boolean fields¶
Since MySQL doesn’t have a direct BOOLEAN
column type, Django uses a
TINYINT
column with values of 1
and 0
to store values for the
BooleanField
model field. Refer to the documentation
of that field for more details, but usually this won’t be something that will
matter unless you’re printing out the field values and are expecting to see
True
and False.
.
Character fields¶
Any fields that are stored with VARCHAR
column types have their
max_length
restricted to 255 characters if you are using unique=True
for the field. This affects CharField
,
SlugField
and
CommaSeparatedIntegerField
.
Furthermore, if you are using a version of MySQL prior to 5.0.3, all of those
column types have a maximum length restriction of 255 characters, regardless
of whether unique=True
is specified or not.
SQLite notes¶
SQLite provides an excellent development alternative for applications that are predominantly read-only or require a smaller installation footprint. As with all database servers, though, there are some differences that are specific to SQLite that you should be aware of.
String matching for non-ASCII strings¶
SQLite doesn’t support case-insensitive matching for non-ASCII strings. Some
possible workarounds for this are documented at sqlite.org, but they are
not utilised by the default SQLite backend in Django. Therefore, if you are
using the iexact
lookup type in your queryset filters, be aware that it
will not work as expected for non-ASCII strings.
SQLite 3.3.6 or newer strongly recommended¶
Versions of SQLite 3.3.5 and older contains a bug when handling ORDER BY
parameters. This can cause problems when you use the select
parameter for
the extra()
QuerySet method. The bug can be identified by the error message
OperationalError: ORDER BY terms must not be non-integer constants
.
SQLite 3.3.6 was released in April 2006, so most current binary distributions
for different platforms include newer version of SQLite usable from Python
through either the pysqlite2
or the sqlite3
modules.
However, some platform/Python version combinations include older versions of SQLite (e.g. the official binary distribution of Python 2.5 for Windows, 2.5.4 as of this writing, includes SQLite 3.3.4).
Version 3.5.9¶
The Ubuntu “Intrepid Ibex” (8.10) SQLite 3.5.9-3 package contains a bug that causes problems with the evaluation of query expressions. If you are using Ubuntu “Intrepid Ibex”, you will need to update the package to version 3.5.9-3ubuntu1 or newer (recommended) or find an alternate source for SQLite packages, or install SQLite from source.
At one time, Debian Lenny shipped with the same malfunctioning SQLite 3.5.9-3 package. However the Debian project has subsequently issued updated versions of the SQLite package that correct these bugs. If you find you are getting unexpected results under Debian, ensure you have updated your SQLite package to 3.5.9-5 or later.
The problem does not appear to exist with other versions of SQLite packaged with other operating systems.
Version 3.6.2¶
SQLite version 3.6.2 (released August 30, 2008) introduced a bug into SELECT
DISTINCT
handling that is triggered by, amongst other things, Django’s
DateQuerySet
(returned by the dates()
method on a queryset).
You should avoid using this version of SQLite with Django. Either upgrade to 3.6.3 (released September 22, 2008) or later, or downgrade to an earlier version of SQLite.
“Database is locked” errors¶
SQLite is meant to be a lightweight database, and thus can’t support a high
level of concurrency. OperationalError: database is locked
errors indicate
that your application is experiencing more concurrency than sqlite
can
handle in default configuration. This error means that one thread or process has
an exclusive lock on the database connection and another thread timed out
waiting for the lock the be released.
Python’s SQLite wrapper has
a default timeout value that determines how long the second thread is allowed to
wait on the lock before it times out and raises the OperationalError: database
is locked
error.
If you’re getting this error, you can solve it by:
Switching to another database backend. At a certain point SQLite becomes too “lite” for real-world applications, and these sorts of concurrency errors indicate you’ve reached that point.
Rewriting your code to reduce concurrency and ensure that database transactions are short-lived.
Increase the default timeout value by setting the
timeout
database option option:DATABASE_OPTIONS = { # ... "timeout": 20, # ... }
This will simply make SQLite wait a bit longer before throwing “database is locked” errors; it won’t really do anything to solve them.
Oracle notes¶
Django supports Oracle Database Server versions 9i and
higher. Oracle version 10g or later is required to use Django’s
regex
and iregex
query operators. You will also need at least
version 4.3.1 of the cx_Oracle Python driver.
Note that due to a Unicode-corruption bug in cx_Oracle
5.0, that
version of the driver should not be used with Django;
cx_Oracle
5.0.1 resolved this issue, so if you’d like to use a
more recent cx_Oracle
, use version 5.0.1.
In order for the python manage.py syncdb
command to work, your Oracle
database user must have privileges to run the following commands:
- CREATE TABLE
- CREATE SEQUENCE
- CREATE PROCEDURE
- CREATE TRIGGER
To run Django’s test suite, the user needs these additional privileges:
- CREATE USER
- DROP USER
- CREATE TABLESPACE
- DROP TABLESPACE
- CONNECT WITH ADMIN OPTION
- RESOURCE WITH ADMIN OPTION
Connecting to the database¶
Your Django settings.py file should look something like this for Oracle:
DATABASE_ENGINE = 'oracle'
DATABASE_NAME = 'xe'
DATABASE_USER = 'a_user'
DATABASE_PASSWORD = 'a_password'
DATABASE_HOST = ''
DATABASE_PORT = ''
If you don’t use a tnsnames.ora
file or a similar naming method that
recognizes the SID (“xe” in this example), then fill in both
DATABASE_HOST
and DATABASE_PORT
like so:
DATABASE_ENGINE = 'oracle'
DATABASE_NAME = 'xe'
DATABASE_USER = 'a_user'
DATABASE_PASSWORD = 'a_password'
DATABASE_HOST = 'dbprod01ned.mycompany.com'
DATABASE_PORT = '1540'
You should supply both DATABASE_HOST
and DATABASE_PORT
, or leave both
as empty strings.
Tablespace options¶
A common paradigm for optimizing performance in Oracle-based systems is the
use of tablespaces to organize disk layout. The Oracle backend supports
this use case by adding db_tablespace
options to the Meta
and
Field
classes. (When you use a backend that lacks support for tablespaces,
Django ignores these options.)
A tablespace can be specified for the table(s) generated by a model by
supplying the db_tablespace
option inside the model’s class Meta
.
Additionally, you can pass the db_tablespace
option to a Field
constructor to specify an alternate tablespace for the Field
‘s column
index. If no index would be created for the column, the db_tablespace
option is ignored:
class TablespaceExample(models.Model):
name = models.CharField(max_length=30, db_index=True, db_tablespace="indexes")
data = models.CharField(max_length=255, db_index=True)
edges = models.ManyToManyField(to="self", db_tablespace="indexes")
class Meta:
db_tablespace = "tables"
In this example, the tables generated by the TablespaceExample
model
(i.e., the model table and the many-to-many table) would be stored in the
tables
tablespace. The index for the name field and the indexes on the
many-to-many table would be stored in the indexes
tablespace. The data
field would also generate an index, but no tablespace for it is specified, so
it would be stored in the model tablespace tables
by default.
Use the DEFAULT_TABLESPACE
and DEFAULT_INDEX_TABLESPACE
settings to specify default values for the db_tablespace options.
These are useful for setting a tablespace for the built-in Django apps and
other applications whose code you cannot control.
Django does not create the tablespaces for you. Please refer to Oracle’s documentation for details on creating and managing tablespaces.
Naming issues¶
Oracle imposes a name length limit of 30 characters. To accommodate this, the backend truncates database identifiers to fit, replacing the final four characters of the truncated name with a repeatable MD5 hash value.
NULL and empty strings¶
Django generally prefers to use the empty string (‘’) rather than
NULL, but Oracle treats both identically. To get around this, the
Oracle backend coerces the null=True
option on fields that have
the empty string as a possible value. When fetching from the database,
it is assumed that a NULL value in one of these fields really means
the empty string, and the data is silently converted to reflect this
assumption.
TextField
limitations¶
The Oracle backend stores TextFields
as NCLOB
columns. Oracle imposes
some limitations on the usage of such LOB columns in general:
- LOB columns may not be used as primary keys.
- LOB columns may not be used in indexes.
- LOB columns may not be used in a
SELECT DISTINCT
list. This means that attempting to use theQuerySet.distinct
method on a model that includesTextField
columns will result in an error when run against Oracle. A workaround to this is to keepTextField
columns out of any models that you foresee performingdistinct()
queries on, and to include theTextField
in a related model instead.
Using a 3rd-party database backend¶
In addition to the officially supported databases, there are backends provided by 3rd parties that allow you to use other databases with Django:
The Django versions and ORM features supported by these unofficial backends vary considerably. Queries regarding the specific capabilities of these unofficial backends, along with any support queries, should be directed to the support channels provided by each 3rd party project.