Aggregation Framework

The aggregation framework is MongoDB's analytics engine: a pipeline of stages that transforms documents in sequence - match, group, project, sort, join - and returns the final shape. MongoEngine hands you the raw pipeline API via objects.aggregate([...]), plus a few convenience methods (sum, average, count) for the simple cases. Almost every "report" query in an application is an aggregation pipeline, and this chapter is its vocabulary.

The Pipeline, End to End

from mongoengine import Document, IntField, ReferenceField, BooleanField

class Post(Document):
    title = StringField()
    author = ReferenceField(User)
    views = IntField(default=0)
    published = BooleanField(default=False)

# total views per author, top 10, published posts only
results = Post.objects.aggregate([
    {"$match": {"published": True}},
    {"$group": {
        "_id": "$author",
        "total_views": {"$sum": "$views"},
        "post_count": {"$sum": 1},
    }},
    {"$sort": {"total_views": -1}},
    {"$limit": 10},
    {"$project": {
        "author": "$_id",
        "total_views": 1,
        "post_count": 1,
        "_id": 0,
    }},
])
for r in results:
    print(r["author"], r["total_views"], r["post_count"])

Each stage receives the previous stage's documents and emits transformed ones; the pipeline is the unit of work, executed server-side in a single operation. The stage order matters - $match first filters early (cheap, indexable), $group collapses, $sort and $limit shape, $project names the output. The result is a list of plain dicts, not document instances.

The Stages You Will Use

# $lookup: attach each author's username to the per-author totals
with_usernames = Post.objects.aggregate([
    {"$match": {"published": True}},
    {"$group": {"_id": "$author", "total_views": {"$sum": "$views"}}},
    {"$lookup": {
        "from": "users",
        "localField": "_id",
        "foreignField": "_id",
        "as": "author_doc",
    }},
    {"$project": {"username": {"$arrayElemAt": ["$author_doc.username", 0]},
                  "total_views": 1, "_id": 0}},
    {"$sort": {"total_views": -1}},
])

$lookup is the aggregation JOIN - the one place MongoDB lets you pull in another collection's fields in a single server operation. It is the standard way to enrich a grouped report (totals per author, plus the author's name) without the application doing the second query.

The Shortcut Methods

# simple aggregates as one-liners
total = Post.objects(published=True).sum("views")
avg = Post.objects(published=True).average("views")
n = Post.objects(published=True).count()

sum, average, and count are sugar over a two-stage pipeline ($match + $group/$count) - fine for a single number, but the moment you need two grouped values, a top-N, or a join, write the explicit pipeline: it is the same engine with the full vocabulary.

Performance Notes

• Match early. The first $match can use an index; everything after $group operates on the reduced set. Push filters before aggregation, not after.
• Group keys should be indexed. The grouping field (author above) as an index makes the group stage an indexed scan.
• Watch memory. $sort without a limit holds all matched documents in the 100 MB sort memory cap; top-N sorts ($sort + $limit) can use an index instead.
• Inspect with explain(). collection.aggregate(pipeline, explain=True) (via as_pymongo()) shows the stages and where indexes are used - the aggregation equivalent of a query plan.

Summary

Write reports as pipelines: $match to scope (indexed), $group to aggregate, $sort/$limit for top-N, $project to name the output, and $lookup to enrich from another collection - all executed server-side by objects.aggregate([...]). Use sum/average/count for single numbers, keep the match stage first and its fields indexed, and reach for explain() when a pipeline gets slower than its input suggests.