Video Capture and Processing

Video processing is the image pipeline in a loop: open a capture source (a file or a camera index), read frames one at a time, apply the per-frame operations, and write or display the results. The API is VideoCapture for input and VideoWriter for output, and the frame loop - read, process, write, check the key - is the same skeleton as the display chapter with a source attached. The new concerns are the frame rate, the codec, and the per-frame budget: every operation you add is multiplied by 30 frames a second.

Reading Video

import cv2

# from a file
cap = cv2.VideoCapture("clip.mp4")
print(cap.isOpened())
print("fps:", cap.get(cv2.CAP_PROP_FPS))
print("frames:", cap.get(cv2.CAP_PROP_FRAME_COUNT))
print("size:", cap.get(cv2.CAP_PROP_FRAME_WIDTH),
      cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

while True:
    ret, frame = cap.read()          # ret False at end of file
    if not ret:
        break
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    edges = cv2.Canny(cv2.GaussianBlur(gray, (5, 5), 0), 100, 200)
    cv2.imshow("Edges", edges)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()

cap.read() returns (ret, frame) - check ret, because at end-of-file (and on a failed open) the frame is None and processing it is the classic crash. The CAP_PROP_* getters report fps, frame count, and resolution; setting them (width, fps) works for some backends and is ignored by others, which is why "set the camera to 1280x720" silently does nothing on certain drivers. For a webcam, pass the index (0 for the default device) instead of a path; on Linux you can also pass a v4l2 device node directly.

Writing Video

import cv2

cap = cv2.VideoCapture("clip.mp4")
w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = cap.get(cv2.CAP_PROP_FPS) or 30.0

# the fourcc encodes the codec; mp4v works in most mp4 outputs
fourcc = cv2.VideoWriter_fourcc(*"mp4v")
out = cv2.VideoWriter("out.mp4", fourcc, fps, (w, h))

while True:
    ret, frame = cap.read()
    if not ret:
        break
    out.write(cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY))

cap.release()
out.release()

VideoWriter takes a codec fourcc, the fps, and the frame size - the size must match every frame you write, or the writer misbehaves. mp4v in an .mp4 container is the safe default; avc1 (H.264) gives smaller files but is not always available in the bundled build, and on Linux the XVID/FLV codecs in .avi are the reliable fallbacks. A gray frame written into a color-sized writer works but check the container - codec and extension must agree, which is the source of most "the file is 0 bytes" reports.

Per-Frame Processing and the Budget

import cv2
import time

cap = cv2.VideoCapture(0)
last = time.perf_counter()
fps_actual = 0.0

while True:
    ret, frame = cap.read()
    if not ret:
        break

    # a realistic per-frame chain, kept deliberately cheap
    small = cv2.resize(frame, (640, 360))
    gray = cv2.cvtColor(small, cv2.COLOR_BGR2GRAY)
    blurred = cv2.GaussianBlur(gray, (5, 5), 0)
    edges = cv2.Canny(blurred, 100, 200)

    now = time.perf_counter()
    dt = now - last
    last = now
    fps_actual = 0.9 * fps_actual + 0.1 * (1.0 / dt if dt > 0 else 0)

    cv2.putText(edges, f"{fps_actual:.0f} fps", (10, 24),
                cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 255, 0), 2)
    cv2.imshow("Video", edges)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()

Every operation in the chain runs 30 times a second, so the budget discipline differs from stills: downscale early (processing at 640x360 instead of 1920x1080 is a 9x reduction), keep kernels small, and measure - the running fps in the corner is the one widget every video tool should have. When the fps drops, the fix is almost always resolution or a heavy filter (bilateral, non-local means, DNN inference), not the Canny or the color conversion. Frame timing with perf_counter and an exponential moving average is the standard instrumentation.

Frame Differencing: A Free Detector

import cv2

cap = cv2.VideoCapture(0)
prev = None

while True:
    ret, frame = cap.read()
    if not ret:
        break
    gray = cv2.cvtColor(cv2.resize(frame, (640, 360)), cv2.COLOR_BGR2GRAY)
    blur = cv2.GaussianBlur(gray, (5, 5), 0)

    if prev is not None:
        diff = cv2.absdiff(blur, prev)
        _, thresh = cv2.threshold(diff, 25, 255, cv2.THRESH_BINARY)
        thresh = cv2.dilate(thresh, None, iterations=2)
        contours, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL,
                                       cv2.CHAIN_APPROX_SIMPLE)
        for c in contours:
            if cv2.contourArea(c) > 500:
                x, y, w, h = cv2.boundingRect(c)
                cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 0, 255), 2)
    prev = blur

    cv2.imshow("Motion", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()

Frame differencing - the absolute difference between consecutive blurred frames, thresholded, dilated, and run through findContours - is the zero-model motion detector: anything that changed becomes a contour. It is the first "video understanding" result you can get in twenty lines, the same six-line contour pipeline from earlier applied to a difference image. Its limits are well known - lighting changes trigger it, and it sees motion but not objects - which is exactly what motivates the DNN detectors in the next chapters.

Summary

Build the loop: VideoCapture (file path or device index), a while that reads (ret, frame) and checks ret, processes, and writes via VideoWriter with a matching codec, fps, and size - releasing both at the end. Treat the per-frame budget as the design constraint: downscale early, keep filters light, and instrument with a running fps. The frame-differencing recipe (absdiff, threshold, dilate, findContours) is the entry point to video understanding - motion as contours - and it sets the stage for the learned detectors that follow.