Indexes

Indexes are the difference between a query that scans a thousand documents and one that touches two. MongoEngine declares indexes in the document's meta block - single-field, compound, text, and unique - and creates them with ensure_indexes(), which reconciles the declared set with what exists on the server. The skill is designing the index set from the query patterns your application actually runs, not from a gut feeling about "important" fields.

Declaring Indexes

class Post(Document):
    author = ReferenceField(User)
    published = BooleanField(default=False)
    published_at = DateTimeField()
    content = StringField()

    meta = {
        "collection": "posts",
        "indexes": [
            "author",                          # single-field
            "published",                       # single-field
            ("author", "-published_at"),       # compound: equality + sort
            "text(content, title)",            # text index
            "slug",                            # will be unique (below)
        ],
    }

    slug = StringField(unique=True)            # unique index, same mechanism

The string forms map directly to MongoDB index keys: "field" is ascending, "-field" descending, a tuple is compound (order matters - the first element is the equality field, later ones serve the sort), and "text(...)" builds a text index over the listed fields. unique=True on a field declares a unique index alongside the field itself.

Creating Them

Post.ensure_indexes()          # creates missing indexes from meta + field specs

# see what exists on the server
print(Post.list_indexes())     # [{'name': '_id_', ...}, {'name': 'author_1', ...}]

# drop a specific one
Post.drop_indexes(["content_text"])   # or drop_indexes() for all except _id

Indexes are not created automatically on save - ensure_indexes() is the explicit, idempotent step that creates whatever is missing. Run it in application startup, in a deploy hook, or in a migration; it is safe to call repeatedly (it creates only what is absent) and it is the single place where the declared index set and the server state are reconciled.

Designing From the Query Patterns

The index design rule: every filter field your queries hit, in the order the queries use them, gets an index entry - and the sort field goes last in a compound index so the sort is free.

Query pattern	Index that serves it
`Post.objects(author=ada)`	`["author"]`
`Post.objects(author=ada).order_by("-published_at")`	`["author", "-published_at"]`
`Post.objects(published=True, published_at__gte=last_week)`	`["published", "published_at"]` (equality first, range second)
`User.objects(username__istartswith="a")`	a text index, or accept the scan (prefix regex on plain strings uses the index only for `^` anchors)

Two compounding rules worth memorizing. First, the E SR order: Equality fields first, then Sort fields, then Range fields - a compound index serves the query only when the query uses the fields in that prefix order. Second, fewer, targeted indexes beat many speculative ones: every index costs write amplification and storage, and an index no query uses is pure overhead.

Unique and Covering

class Slug(Document):
    value = StringField(unique=True)     # the database now enforces it

# a covering index: the query returns only indexed fields - no document fetch
Post.objects(author=ada, published=True).only("published_at").count()
# served entirely from the (author, published, published_at) compound

Unique indexes are the only constraint MongoDB enforces at the database level - they are the real guarantee behind unique=True fields and the correct tool for natural keys (slugs, emails, skus). When a query's filter, sort, and projection all live in one compound index, the server can answer without touching the documents at all - the covering-index case, the fastest form a query can take.

Measuring Whether an Index Helps

from mongoengine import get_db

coll = get_db()["posts"]
plan = coll.find({"author": ada.id}).explain()
print(plan["queryPlanner"]["winningPlan"]["stage"])   # IXSCAN = using an index
print(plan["executionStats"]["totalDocsExamined"])    # vs nReturned: the ratio that matters

explain() is the ground truth: IXSCAN means an index is used, COLLSCAN means a full scan, and totalDocsExamined / nReturned is the efficiency ratio - 2 examined for 1 returned is a good index, 50,000 examined for 1 returned is a scan wearing an index's costume. Run it for your real query shapes at real data volumes, and design the index set from what it shows.

Summary

Declare indexes in meta["indexes"] (single, compound with E-S-R order, text, and unique via fields), create them with ensure_indexes() at startup or deploy, and design the set from the queries you actually run - not from importance. Use explain() to verify IXSCAN and a healthy examined-to-returned ratio, keep unique constraints as unique indexes (the only server-enforced rule in the system), and prune indexes no query uses - every one of them taxes every write.