Release: 2.0.31 | Release Date: June 18, 2024

# SQLAlchemy 2.0 Documentation

# What’s new in SQLAlchemy 0.4?

About this Document

This document describes changes between SQLAlchemy version 0.3, last released October 14, 2007, and SQLAlchemy version 0.4, last released October 12, 2008.

Document date: March 21, 2008

## First Things First

If you’re using any ORM features, make sure you import from `sqlalchemy.orm`:

    from sqlalchemy import *
    from sqlalchemy.orm import *

Secondly, anywhere you used to say `engine=`, `connectable=`, `bind_to=`, `something.engine`, `metadata.connect()`, use `bind`:

    myengine = create_engine("sqlite://")

    meta = MetaData(myengine)

    meta2 = MetaData()
    meta2.bind = myengine

    session = create_session(bind=myengine)

    statement = select([table], bind=myengine)

Got those ? Good! You’re now (95%) 0.4 compatible. If you’re using 0.3.10, you can make these changes immediately; they’ll work there too.

## Module Imports

In 0.3, “`from sqlalchemy import *`” would import all of sqlalchemy’s sub-modules into your namespace. Version 0.4 no longer imports sub-modules into the namespace. This may mean you need to add extra imports into your code.

In 0.3, this code worked:

    from sqlalchemy import *

    class UTCDateTime(types.TypeDecorator):
        pass

In 0.4, one must do:

    from sqlalchemy import *
    from sqlalchemy import types

    class UTCDateTime(types.TypeDecorator):
        pass

## Object Relational Mapping

### Querying

#### New Query API

Query is standardized on the generative interface (old interface is still there, just deprecated). While most of the generative interface is available in 0.3, the 0.4 Query has the inner guts to match the generative outside, and has a lot more tricks. All result narrowing is via `filter()` and `filter_by()`, limiting/offset is either through array slices or `limit()`/`offset()`, joining is via `join()` and `outerjoin()` (or more manually, through `select_from()` as well as manually-formed criteria).

To avoid deprecation warnings, you must make some changes to your 03 code

User.query.get_by( **kwargs )

    User.query.filter_by(**kwargs).first()

User.query.select_by( **kwargs )

    User.query.filter_by(**kwargs).all()

User.query.select()

    User.query.filter(xxx).all()

#### New Property-Based Expression Constructs

By far the most palpable difference within the ORM is that you can now construct your query criterion using class-based attributes directly. The “.c.” prefix is no longer needed when working with mapped classes:

    session.query(User).filter(and_(User.name == "fred", User.id > 17))

While simple column-based comparisons are no big deal, the class attributes have some new “higher level” constructs available, including what was previously only available in `filter_by()`:

    # comparison of scalar relations to an instance
    filter(Address.user == user)

    # return all users who contain a particular address
    filter(User.addresses.contains(address))

    # return all users who *dont* contain the address
    filter(~User.address.contains(address))

    # return all users who contain a particular address with
    # the email_address like '%foo%'
    filter(User.addresses.any(Address.email_address.like("%foo%")))

    # same, email address equals 'foo@bar.com'.  can fall back to keyword
    # args for simple comparisons
    filter(User.addresses.any(email_address="foo@bar.com"))

    # return all Addresses whose user attribute has the username 'ed'
    filter(Address.user.has(name="ed"))

    # return all Addresses whose user attribute has the username 'ed'
    # and an id > 5 (mixing clauses with kwargs)
    filter(Address.user.has(User.id > 5, name="ed"))

The `Column` collection remains available on mapped classes in the `.c` attribute. Note that property-based expressions are only available with mapped properties of mapped classes. `.c` is still used to access columns in regular tables and selectable objects produced from SQL Expressions.

#### Automatic Join Aliasing

We’ve had join() and outerjoin() for a while now:

    session.query(Order).join("items")

Now you can alias them:

    session.query(Order).join("items", aliased=True).filter(Item.name="item 1").join(
        "items", aliased=True
    ).filter(Item.name == "item 3")

The above will create two joins from orders->items using aliases. the `filter()` call subsequent to each will adjust its table criterion to that of the alias. To get at the `Item` objects, use `add_entity()` and target each join with an `id`:

    session.query(Order).join("items", id="j1", aliased=True).filter(
        Item.name == "item 1"
    ).join("items", aliased=True, id="j2").filter(Item.name == "item 3").add_entity(
        Item, id="j1"
    ).add_entity(
        Item, id="j2"
    )

Returns tuples in the form: `(Order, Item, Item)`.

#### Self-referential Queries

So query.join() can make aliases now. What does that give us ? Self-referential queries ! Joins can be done without any `Alias` objects:

    # standard self-referential TreeNode mapper with backref
    mapper(
        TreeNode,
        tree_nodes,
        properties={
            "children": relation(
                TreeNode, backref=backref("parent", remote_side=tree_nodes.id)
            )
        },
    )

    # query for node with child containing "bar" two levels deep
    session.query(TreeNode).join(["children", "children"], aliased=True).filter_by(
        name="bar"
    )

To add criterion for each table along the way in an aliased join, you can use `from_joinpoint` to keep joining against the same line of aliases:

    # search for the treenode along the path "n1/n12/n122"

    # first find a Node with name="n122"
    q = sess.query(Node).filter_by(name="n122")

    # then join to parent with "n12"
    q = q.join("parent", aliased=True).filter_by(name="n12")

    # join again to the next parent with 'n1'.  use 'from_joinpoint'
    # so we join from the previous point, instead of joining off the
    # root table
    q = q.join("parent", aliased=True, from_joinpoint=True).filter_by(name="n1")

    node = q.first()

#### `query.populate_existing()`

The eager version of `query.load()` (or `session.refresh()`). Every instance loaded from the query, including all eagerly loaded items, get refreshed immediately if already present in the session:

    session.query(Blah).populate_existing().all()

### Relations

#### SQL Clauses Embedded in Updates/Inserts

For inline execution of SQL clauses, embedded right in the UPDATE or INSERT, during a `flush()`:

    myobject.foo = mytable.c.value + 1

    user.pwhash = func.md5(password)

    order.hash = text("select hash from hashing_table")

The column-attribute is set up with a deferred loader after the operation, so that it issues the SQL to load the new value when you next access.

#### Self-referential and Cyclical Eager Loading

Since our alias-fu has improved, `relation()` can join along the same table *any number of times*; you tell it how deep you want to go. Lets show the self-referential `TreeNode` more clearly:

    nodes = Table(
        "nodes",
        metadata,
        Column("id", Integer, primary_key=True),
        Column("parent_id", Integer, ForeignKey("nodes.id")),
        Column("name", String(30)),
    )

    class TreeNode(object):
        pass

    mapper(
        TreeNode,
        nodes,
        properties={"children": relation(TreeNode, lazy=False, join_depth=3)},
    )

So what happens when we say:

    create_session().query(TreeNode).all()

? A join along aliases, three levels deep off the parent:

    SELECT
    nodes_3.id AS nodes_3_id, nodes_3.parent_id AS nodes_3_parent_id, nodes_3.name AS nodes_3_name,
    nodes_2.id AS nodes_2_id, nodes_2.parent_id AS nodes_2_parent_id, nodes_2.name AS nodes_2_name,
    nodes_1.id AS nodes_1_id, nodes_1.parent_id AS nodes_1_parent_id, nodes_1.name AS nodes_1_name,
    nodes.id AS nodes_id, nodes.parent_id AS nodes_parent_id, nodes.name AS nodes_name
    FROM nodes LEFT OUTER JOIN nodes AS nodes_1 ON nodes.id = nodes_1.parent_id
    LEFT OUTER JOIN nodes AS nodes_2 ON nodes_1.id = nodes_2.parent_id
    LEFT OUTER JOIN nodes AS nodes_3 ON nodes_2.id = nodes_3.parent_id
    ORDER BY nodes.oid, nodes_1.oid, nodes_2.oid, nodes_3.oid

Notice the nice clean alias names too. The joining doesn’t care if it’s against the same immediate table or some other object which then cycles back to the beginning. Any kind of chain of eager loads can cycle back onto itself when `join_depth` is specified. When not present, eager loading automatically stops when it hits a cycle.

#### Composite Types

This is one from the Hibernate camp. Composite Types let you define a custom datatype that is composed of more than one column (or one column, if you wanted). Lets define a new type, `Point`. Stores an x/y coordinate:

    class Point(object):
        def __init__(self, x, y):
            self.x = x
            self.y = y

        def __composite_values__(self):
            return self.x, self.y

        def __eq__(self, other):
            return other.x == self.x and other.y == self.y

        def __ne__(self, other):
            return not self.__eq__(other)

The way the `Point` object is defined is specific to a custom type; constructor takes a list of arguments, and the `__composite_values__()` method produces a sequence of those arguments. The order will match up to our mapper, as we’ll see in a moment.

Let’s create a table of vertices storing two points per row:

    vertices = Table(
        "vertices",
        metadata,
        Column("id", Integer, primary_key=True),
        Column("x1", Integer),
        Column("y1", Integer),
        Column("x2", Integer),
        Column("y2", Integer),
    )

Then, map it ! We’ll create a `Vertex` object which stores two `Point` objects:

    class Vertex(object):
        def __init__(self, start, end):
            self.start = start
            self.end = end

    mapper(
        Vertex,
        vertices,
        properties={
            "start": composite(Point, vertices.c.x1, vertices.c.y1),
            "end": composite(Point, vertices.c.x2, vertices.c.y2),
        },
    )

Once you’ve set up your composite type, it’s usable just like any other type:

    v = Vertex(Point(3, 4), Point(26, 15))
    session.save(v)
    session.flush()

    # works in queries too
    q = session.query(Vertex).filter(Vertex.start == Point(3, 4))

If you’d like to define the way the mapped attributes generate SQL clauses when used in expressions, create your own `sqlalchemy.orm.PropComparator` subclass, defining any of the common operators (like `__eq__()`, `__le__()`, etc.), and send it in to `composite()`. Composite types work as primary keys too, and are usable in `query.get()`:

    # a Document class which uses a composite Version
    # object as primary key
    document = query.get(Version(1, "a"))

#### `dynamic_loader()` relations

A `relation()` that returns a live `Query` object for all read operations. Write operations are limited to just `append()` and `remove()`, changes to the collection are not visible until the session is flushed. This feature is particularly handy with an “autoflushing” session which will flush before each query.

    mapper(
        Foo,
        foo_table,
        properties={
            "bars": dynamic_loader(
                Bar,
                backref="foo",
                # <other relation() opts>
            )
        },
    )

    session = create_session(autoflush=True)
    foo = session.query(Foo).first()

    foo.bars.append(Bar(name="lala"))

    for bar in foo.bars.filter(Bar.name == "lala"):
        print(bar)

    session.commit()

#### New Options: `undefer_group()`, `eagerload_all()`

A couple of query options which are handy. `undefer_group()` marks a whole group of “deferred” columns as undeferred:

    mapper(
        Class,
        table,
        properties={
            "foo": deferred(table.c.foo, group="group1"),
            "bar": deferred(table.c.bar, group="group1"),
            "bat": deferred(table.c.bat, group="group1"),
        },
    )

    session.query(Class).options(undefer_group("group1")).filter(...).all()

and `eagerload_all()` sets a chain of attributes to be eager in one pass:

    mapper(Foo, foo_table, properties={"bar": relation(Bar)})
    mapper(Bar, bar_table, properties={"bat": relation(Bat)})
    mapper(Bat, bat_table)

    # eager load bar and bat
    session.query(Foo).options(eagerload_all("bar.bat")).filter(...).all()

#### New Collection API

Collections are no longer proxied by an {{{InstrumentedList}}} proxy, and access to members, methods and attributes is direct. Decorators now intercept objects entering and leaving the collection, and it is now possible to easily write a custom collection class that manages its own membership. Flexible decorators also replace the named method interface of custom collections in 0.3, allowing any class to be easily adapted to use as a collection container.

Dictionary-based collections are now much easier to use and fully `dict`-like. Changing `__iter__` is no longer needed for `dict``s, and new built-in ``dict` types cover many needs:

    # use a dictionary relation keyed by a column
    relation(Item, collection_class=column_mapped_collection(items.c.keyword))
    # or named attribute
    relation(Item, collection_class=attribute_mapped_collection("keyword"))
    # or any function you like
    relation(Item, collection_class=mapped_collection(lambda entity: entity.a + entity.b))

Existing 0.3 `dict`-like and freeform object derived collection classes will need to be updated for the new API. In most cases this is simply a matter of adding a couple decorators to the class definition.

#### Mapped Relations from External Tables/Subqueries

This feature quietly appeared in 0.3 but has been improved in 0.4 thanks to better ability to convert subqueries against a table into subqueries against an alias of that table; this is key for eager loading, aliased joins in queries, etc. It reduces the need to create mappers against select statements when you just need to add some extra columns or subqueries:

    mapper(
        User,
        users,
        properties={
            "fullname": column_property(
                (users.c.firstname + users.c.lastname).label("fullname")
            ),
            "numposts": column_property(
                select([func.count(1)], users.c.id == posts.c.user_id)
                .correlate(users)
                .label("posts")
            ),
        },
    )

a typical query looks like:

    SELECT (SELECT count(1) FROM posts WHERE users.id = posts.user_id) AS count,
    users.firstname || users.lastname AS fullname,
    users.id AS users_id, users.firstname AS users_firstname, users.lastname AS users_lastname
    FROM users ORDER BY users.oid

### Horizontal Scaling (Sharding) API

[browser:/sqlalchemy/trunk/examples/sharding/attribute_shard .py]

### Sessions

#### New Session Create Paradigm; SessionContext, assignmapper Deprecated

That’s right, the whole shebang is being replaced with two configurational functions. Using both will produce the most 0.1-ish feel we’ve had since 0.1 (i.e., the least amount of typing).

Configure your own `Session` class right where you define your `engine` (or anywhere):

    from sqlalchemy import create_engine
    from sqlalchemy.orm import sessionmaker

    engine = create_engine("myengine://")
    Session = sessionmaker(bind=engine, autoflush=True, transactional=True)

    # use the new Session() freely
    sess = Session()
    sess.save(someobject)
    sess.flush()

If you need to post-configure your Session, say with an engine, add it later with `configure()`:

    Session.configure(bind=create_engine(...))

All the behaviors of `SessionContext` and the `query` and `__init__` methods of `assignmapper` are moved into the new `scoped_session()` function, which is compatible with both `sessionmaker` as well as `create_session()`:

    from sqlalchemy.orm import scoped_session, sessionmaker

    Session = scoped_session(sessionmaker(autoflush=True, transactional=True))
    Session.configure(bind=engine)

    u = User(name="wendy")

    sess = Session()
    sess.save(u)
    sess.commit()

    # Session constructor is thread-locally scoped.  Everyone gets the same
    # Session in the thread when scope="thread".
    sess2 = Session()
    assert sess is sess2

When using a thread-local `Session`, the returned class has all of `Session's` interface implemented as classmethods, and “assignmapper“‘s functionality is available using the `mapper` classmethod. Just like the old `objectstore` days….

    # "assignmapper"-like functionality available via ScopedSession.mapper
    Session.mapper(User, users_table)

    u = User(name="wendy")

    Session.commit()

#### Sessions are again Weak Referencing By Default

The weak_identity_map flag is now set to `True` by default on Session. Instances which are externally deferenced and fall out of scope are removed from the session automatically. However, items which have “dirty” changes present will remain strongly referenced until those changes are flushed at which case the object reverts to being weakly referenced (this works for ‘mutable’ types, like picklable attributes, as well). Setting weak_identity_map to `False` restores the old strong-referencing behavior for those of you using the session like a cache.

#### Auto-Transactional Sessions

As you might have noticed above, we are calling `commit()` on `Session`. The flag `transactional=True` means the `Session` is always in a transaction, `commit()` persists permanently.

#### Auto-Flushing Sessions

Also, `autoflush=True` means the `Session` will `flush()` before each `query` as well as when you call `flush()` or `commit()`. So now this will work:

    Session = sessionmaker(bind=engine, autoflush=True, transactional=True)

    u = User(name="wendy")

    sess = Session()
    sess.save(u)

    # wendy is flushed, comes right back from a query
    wendy = sess.query(User).filter_by(name="wendy").one()

#### Transactional methods moved onto sessions

`commit()` and `rollback()`, as well as `begin()` are now directly on `Session`. No more need to use `SessionTransaction` for anything (it remains in the background).

    Session = sessionmaker(autoflush=True, transactional=False)

    sess = Session()
    sess.begin()

    # use the session

    sess.commit()  # commit transaction

Sharing a `Session` with an enclosing engine-level (i.e. non-ORM) transaction is easy:

    Session = sessionmaker(autoflush=True, transactional=False)

    conn = engine.connect()
    trans = conn.begin()
    sess = Session(bind=conn)

    # ... session is transactional

    # commit the outermost transaction
    trans.commit()

#### Nested Session Transactions with SAVEPOINT

Available at the Engine and ORM level. ORM docs so far:

https://www.sqlalchemy.org/docs/04/session.html#unitofwork_managing

#### Two-Phase Commit Sessions

Available at the Engine and ORM level. ORM docs so far:

https://www.sqlalchemy.org/docs/04/session.html#unitofwork_managing

### Inheritance

#### Polymorphic Inheritance with No Joins or Unions

New docs for inheritance: https://www.sqlalchemy.org/docs/04 /mappers.html#advdatamapping_mapper_inheritance_joined

#### Better Polymorphic Behavior with `get()`

All classes within a joined-table inheritance hierarchy get an `_instance_key` using the base class, i.e. `(BaseClass, (1, ), None)`. That way when you call `get()` a `Query` against the base class, it can locate subclass instances in the current identity map without querying the database.

### Types

#### Custom Subclasses of `sqlalchemy.types.TypeDecorator`

There is a New API for subclassing a TypeDecorator. Using the 0.3 API causes compilation errors in some cases.

## SQL Expressions

### All New, Deterministic Label/Alias Generation

All the “anonymous” labels and aliases use a simple <name>_<number> format now. SQL is much easier to read and is compatible with plan optimizer caches. Just check out some of the examples in the tutorials: https://www.sqlalchemy.org/docs/04/ormtutorial.html https://www.sqlalchemy.org/docs/04/sqlexpression.html

### Generative select() Constructs

This is definitely the way to go with `select()`. See htt p://www.sqlalchemy.org/docs/04/sqlexpression.html#sql_transf orm .

### New Operator System

SQL operators and more or less every SQL keyword there is are now abstracted into the compiler layer. They now act intelligently and are type/backend aware, see: https://www.sqlalchemy.org/docs/04/sqlexpression.html#sql_operators

### All `type` Keyword Arguments Renamed to `type_`

Just like it says:

    b = bindparam("foo", type_=String)

### in_ Function Changed to Accept Sequence or Selectable

The in_ function now takes a sequence of values or a selectable as its sole argument. The previous API of passing in values as positional arguments still works, but is now deprecated. This means that

    my_table.select(my_table.c.id.in_(1, 2, 3))
    my_table.select(my_table.c.id.in_(*listOfIds))

should be changed to

    my_table.select(my_table.c.id.in_([1, 2, 3]))
    my_table.select(my_table.c.id.in_(listOfIds))

## Schema and Reflection

### `MetaData`, `BoundMetaData`, `DynamicMetaData`…

In the 0.3.x series, `BoundMetaData` and `DynamicMetaData` were deprecated in favor of `MetaData` and `ThreadLocalMetaData`. The older names have been removed in 0.4. Updating is simple:

    +-------------------------------------+-------------------------+
    |If You Had                           | Now Use                 |
    +=====================================+=========================+
    | ``MetaData``                        | ``MetaData``            |
    +-------------------------------------+-------------------------+
    | ``BoundMetaData``                   | ``MetaData``            |
    +-------------------------------------+-------------------------+
    | ``DynamicMetaData`` (with one       | ``MetaData``            |
    | engine or threadlocal=False)        |                         |
    +-------------------------------------+-------------------------+
    | ``DynamicMetaData``                 | ``ThreadLocalMetaData`` |
    | (with different engines per thread) |                         |
    +-------------------------------------+-------------------------+

The seldom-used `name` parameter to `MetaData` types has been removed. The `ThreadLocalMetaData` constructor now takes no arguments. Both types can now be bound to an `Engine` or a single `Connection`.

### One Step Multi-Table Reflection

You can now load table definitions and automatically create `Table` objects from an entire database or schema in one pass:

    >>> metadata = MetaData(myengine, reflect=True)
    >>> metadata.tables.keys()
    ['table_a', 'table_b', 'table_c', '...']

`MetaData` also gains a `.reflect()` method enabling finer control over the loading process, including specification of a subset of available tables to load.

## SQL Execution

### `engine`, `connectable`, and `bind_to` are all now `bind`

### `Transactions`, `NestedTransactions` and `TwoPhaseTransactions`

### Connection Pool Events

The connection pool now fires events when new DB-API connections are created, checked out and checked back into the pool. You can use these to execute session-scoped SQL setup statements on fresh connections, for example.

### Oracle Engine Fixed

In 0.3.11, there were bugs in the Oracle Engine on how Primary Keys are handled. These bugs could cause programs that worked fine with other engines, such as sqlite, to fail when using the Oracle Engine. In 0.4, the Oracle Engine has been reworked, fixing these Primary Key problems.

### Out Parameters for Oracle

    result = engine.execute(
        text(
            "begin foo(:x, :y, :z); end;",
            bindparams=[
                bindparam("x", Numeric),
                outparam("y", Numeric),
                outparam("z", Numeric),
            ],
        ),
        x=5,
    )
    assert result.out_parameters == {"y": 10, "z": 75}

### Connection-bound `MetaData`, `Sessions`

`MetaData` and `Session` can be explicitly bound to a connection:

    conn = engine.connect()
    sess = create_session(bind=conn)

### Faster, More Foolproof `ResultProxy` Objects