Python Extensions

For high-performance needs, Python allows writing extensions in C/C++. This enables integration with existing C libraries or optimization of performance-critical parts of code.

This topic covers: the Python C API, the setup module, compiling extensions, and a simple example of a C function called from Python.

Although Python is famous for being easy and quick to write, it is slow to run for CPU-intensive operations such as tight loops, large array manipulation, or numerical algorithms. By writing a C extension, critical parts can be executed at native C speed — a potential speedup of 10-100x for the right code. Many popular libraries such as NumPy, Pandas, and OpenCV use this approach: a comfortable Python interface with a super-fast C core.

In addition to the Python C API, there are modern alternatives such as Cython (writing Python that is compiled to C), ctypes (calling C libraries directly from Python without writing a C extension), CFFI, and PyBind11 (for C++). Each approach has tradeoffs between ease of use, performance, and flexibility. The Python C API is the most fundamental and gives full control, but it requires an understanding of reference counting, the PyObject type, and manual memory management.

The extension development process is usually: write C code with the Python.h header, create a method table that defines which functions are exposed to Python, create a setup.py with setuptools.Extension, then compile with python setup.py build. The compilation output is a .so file (Linux/Mac) or .pyd file (Windows) that can be imported directly like a regular Python module.

Code Example

// example.c — a simple C extension
// #include <Python.h>
//
// static PyObject* hello(PyObject* self, PyObject* args) {
//     const char* name;
//     if (!PyArg_ParseTuple(args, "s", &name))
//         return NULL;
//     return PyUnicode_FromFormat("Hello, %s!", name);
// }
//
// static PyMethodDef methods[] = {
//     {"hello", hello, METH_VARARGS, "Greet someone"},
//     {NULL, NULL, 0, NULL}
// };

# setup.py
# from setuptools import setup, Extension
# module = Extension("example", sources=["example.c"])
# setup(name="example", ext_modules=[module])

# $ python setup.py build

Code Explanation

The C code wraps the hello function so it can be called from Python. PyArg_ParseTuple reads a string argument from Python ("s" means string) and stores it in name. If it fails, the function returns NULL (error). PyUnicode_FromFormat creates a Python string from the given format.

The PyMethodDef table tells Python what functions are available in this module — one "hello" function that uses METH_VARARGS (accepts positional arguments). The last line {NULL, NULL, 0, NULL} is a sentinel marking the end of the table.

On the Python side, setup.py uses setuptools.Extension to compile example.c into a module named "example". The python setup.py build command compiles and produces the extension file. After that, import example and call example.hello("John") like a regular Python module.

Everyday Example

A fintech startup needs to validate thousands of account numbers per second using a checksum algorithm written in C. Instead of rewriting the entire algorithm in Python (which is slow), they build a C extension that wraps the existing C function. The result: validation that usually takes 5 seconds in Python drops to 0.05 seconds — 100x faster.

Another example: a game company uses Cython to optimize its physics engine. The collision detection part, which initially took 30ms per frame in pure Python, drops to 2ms after being Cythonized — enough to reach 60 FPS.

Summary

C/C++ extensions give Python native performance capabilities for heavy computation tasks. Ideal for optimizing critical sections, integrating existing C libraries, and building numerical/scientific modules. Start with the Python C API or ctypes for simple cases, then move to Cython or PyBind11 for larger projects. Remember: only optimize the parts proven to be slow (profile first!), because C extensions are far more complex and error-prone than pure Python.