Shared Memory Between Processes

Queues copy data through pickling, which is the wrong tool when processes need to read and write the same large buffer - for example, a numeric array produced by one worker and consumed by many. Shared memory solves that: one region of OS memory, mapped into several processes, without copying. Python offers two levels: the small, typed Value/Array objects and the large, general-purpose SharedMemory (Python 3.8+).

Value and Array for Small State

import multiprocessing as mp

def bump(shared):
    with shared.get_lock():
        shared.value += 1

if __name__ == "__main__":
    counter = mp.Value("i", 0)          # 'i' = signed int
    procs = [mp.Process(target=bump, args=(counter,)) for _ in range(4)]
    [p.start() for p in procs]
    [p.join() for p in procs]
    print(counter.value)                # 4

mp.Array("d", [0.0] * 100) creates a shared array of 100 doubles. Both expose get_lock(), which you must hold across any read-modify-write; the underlying buffer is just a ctypes block, so item access is a plain index operation.

SharedMemory for Large Buffers

For arrays that NumPy should touch directly, multiprocessing.shared_memory.SharedMemory creates a named POSIX shared-memory segment that any process can attach to:

import multiprocessing.shared_memory as shm
import numpy as np

def worker(name, n, size):
    existing = shm.SharedMemory(name=name)
    arr = np.ndarray((size,), dtype="float64", buffer=existing.buf)
    arr[:] = np.arange(size) * (name[1])     # each worker writes its slice
    existing.close()

if __name__ == "__main__":
    block = shm.SharedMemory(create=True, size=1000 * 8)
    arr = np.ndarray((1000,), dtype="float64", buffer=block.buf)
    arr[:] = 0
    procs = [mp.Process(target=worker, args=(f"W{i}", block.name, 1000))
             for i in range(1, 3)]
    [p.start() for p in procs]
    [p.join() for p in procs]
    print(arr[:5])
    block.close()
    block.unlink()     # remove the OS segment - do this in the creator

Lifecycle Rules

• The creator must unlink() the segment when nobody needs it; close() only detaches the local mapping.
• Pass the name (a string) through a queue or constructor argument so other processes can attach.
• There is no locking built in: use an mp.Lock passed alongside the name, or partition the buffer so each writer owns disjoint indices.

When to Use Shared Memory

Reach for it when the payload is large (megabytes and up) and is mutated or read repeatedly across processes. For one-off handoffs, a queue is simpler; for read-mostly big data, write the array to a temporary file or use a memory-mapped file (mmap) instead - shared memory segments do not survive a machine restart and need careful cleanup after crashes.

Summary

Use Value/Array plus its lock for small shared counters and buffers, and SharedMemory with NumPy for large numeric payloads shared across processes. Always document who unlinks the segment, and protect concurrent writes with an explicit lock or by partitioning the data.