Process Communication

Processes do not share memory, so they communicate by sending pickled values over explicit channels: the multiprocessing.Queue for many-to-many messaging and multiprocessing.Pipe for a two-way connection between two processes. Everything that crosses the boundary must be picklable, which shapes how you structure your data.

multiprocessing.Queue

import multiprocessing

def producer(q):
    for i in range(5):
        q.put(i)
    q.put(None)            # sentinel

def consumer(q):
    while True:
        item = q.get()
        if item is None:
            break
        print("got", item)

if __name__ == "__main__":
    q = multiprocessing.Queue()
    p = multiprocessing.Process(target=producer, args=(q,))
    c = multiprocessing.Process(target=consumer, args=(q,))
    p.start(); c.start()
    p.join(); c.join()

The API is the same as queue.Queue (put, get, task_done, join), but every item is pickled into a background feeder thread and unpickled in the receiving process. Two consequences: values are copied (mutations on one side are invisible to the other), and the queue is safe to pass to any number of processes.

Pipe

A pipe is a point-to-point duplex channel - the right tool when exactly two processes need to talk, for example a parent dispatching commands to one worker and collecting its replies:

import multiprocessing

def child(conn):
    while True:
        op, x = conn.recv()
        if op == "stop":
            break
        conn.send((op, x * x))

if __name__ == "__main__":
    parent_conn, child_conn = multiprocessing.Pipe()
    p = multiprocessing.Process(target=child, args=(child_conn,))
    p.start()

    parent_conn.send(("square", 6))
    print(parent_conn.recv())   # ('square', 36)
    parent_conn.send(("stop", None))
    p.join()
    child_conn.close()

Limits and Pitfalls

• Picklability: lambdas, closures, and objects with open sockets or database handles cannot cross the boundary - send plain data (dicts, tuples, dataclasses) instead.
• Ordering: a single queue preserves FIFO order, but two producers interleave arbitrarily; tag messages with a sequence number when order per producer matters.
• Backpressure: Queue(maxsize=N) blocks producers when full, just like the thread-safe queue.
• Cost: pickling large arrays through a queue is slow; for big payloads prefer shared memory (next chapter) or shared files.

Summary

Use a multiprocessing.Queue when many processes exchange messages and a Pipe for a dedicated two-process channel. Everything sent must be picklable, data is copied not shared, and large payloads belong in shared memory rather than in the pipe.