Camera Calibration and 3D

Camera calibration measures the fixed relationship between the camera and the world: the intrinsic matrix (focal length, principal point) and the distortion coefficients (how the lens bends straight lines). Calibrating lets you undo the distortion, project 3D points into the image, and - the payoff - recover the pose of objects in the scene. The standard input is a checkerboard: a known geometric pattern photographed from many angles, from which OpenCV solves for all the parameters at once.

What Calibration Produces

import cv2

# after calibrateCamera:
# camera_matrix - [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
# dist_coeffs - [k1, k2, p1, p2, k3, ...] radial + tangential
print(camera_matrix)      # fx, fy in pixels; cx, cy ~ image center
print(dist_coeffs)        # usually k1, k2 dominate; the rest near 0

# undistort: map distorted pixels to their ideal positions
undist = cv2.undistort(distorted_img, camera_matrix, dist_coeffs)
cv2.imwrite("undistorted.jpg", undist)

# a faster per-frame variant when the ROI is fixed
map1, map2 = cv2.initUndistortRectifyMap(camera_matrix, dist_coeffs,
                                         None, camera_matrix,
                                         (w, h), cv2.CV_16SC2)
fast = cv2.remap(distorted_img, map1, map2, cv2.INTER_LINEAR)

The intrinsics live in camera_matrix: fx/fy are the focal lengths in pixels and (cx, cy) the principal point, usually near the image center. The distortion model is a set of coefficients - radial terms (barrel/pincushion, strongest at the edges) and tangential terms (lens not perfectly seated). undistort corrects a whole image in one call; initUndistortRectifyMap plus remap precomputes the lookup so the per-frame cost is a single remap, the right pattern for video where the matrix is constant.

Calibrating with a Checkerboard

import cv2
import numpy as np

criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
checker = (9, 6)                     # interior corners: 9x6

obj_points = []                       # 3D: the board in its own frame
img_points = []                       # 2D: where the corners land in each photo

objp = np.zeros((checker[0] * checker[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:checker[0], 0:checker[1]].T.reshape(-1, 2)

ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(
    obj_points, img_points, (w, h), None, None)

print("reprojection error:", ret)     # mean pixels off - < 0.5 is good
print(mtx, dist)

For each photo, findChessboardCorners locates the interior corner intersections and returns their 2D positions; the 3D object points are fixed - the grid spacing in the board's own coordinate system. Collect 10-20 photos from varied angles and positions (the board should fill the frame from corner to corner across the set), then calibrateCamera returns the matrix, the distortion, and a per-image reprojection error. That error - the mean distance between detected corners and the corners projected back through the solved model - is the single number that tells you whether the calibration is trustworthy; sub-pixel refinement with cornerSubPix before calibrating is what gets you there.

Drawing the Board and Verifying

import cv2

# the standard QA pass: project the solved 3D corners back and draw
img = cv2.imread("board_photo.jpg")
corners, _ = cv2.findChessboardCorners(img, (9, 6), None)
term = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
corners = cv2.cornerSubPix(img, corners, (11, 11), (-1, -1), term)

objp = np.zeros((9 * 6, 3), np.float32)
objp[:, :2] = np.mgrid[0:9, 0:6].T.reshape(-1, 2)
projected, _ = cv2.projectPoints(objp, rvecs[0], tvecs[0], mtx, dist)

img = cv2.drawChessboardCorners(img, (9, 6), corners, True)
cv2.circle(img, tuple(projected[0].ravel()), 5, (0, 0, 255), -1)
cv2.imwrite("verify.jpg", img)

The verification habit: take one calibration photo, re-detect its corners, project the 3D object points back through the solved intrinsics and that photo's pose, and overlay. If the red projections sit on the detected corners, the model is right. drawChessboardCorners renders the grid with the board's orientation for the visual check. Do this on a few photos from different parts of the frame - a calibration that fits the center but misses the corners has a distortion model that is under-constrained.

Pose: Solving for 3D Position

import cv2
import numpy as np

# ArUco markers: a known square with a code - detect, then solve pose
detector = cv2.aruco.ArucoDetector()
corners, ids, _ = detector.detectMarkers(frame)
if ids is not None:
    for c, idv in zip(corners, ids):
        pts2d = c.reshape(-1, 2).astype(np.float32)
        side = 0.05                      # meters, the marker's real size
        objp = np.array([[-side/2, -side/2, 0], [side/2, -side/2, 0],
                         [side/2, side/2, 0], [-side/2, side/2, 0]],
                        np.float32)
        ok, rvec, tvec = cv2.solvePnP(objp, pts2d, mtx, dist,
                                      flags=cv2.SOLVEPNP_ITERATIVE)
        if ok:
            rot, _ = cv2.Rodrigues(rvec)
            print("marker center:", tvec.ravel())

solvePnP is the inverse of projection: given the known 3D shape, its 2D image points, and the calibrated intrinsics, it solves for the object's rotation and translation in the camera frame. ArUco markers make this trivial - four known corners, instant pose - and it is the backbone of AR anchoring and robot localization with a camera. With a calibration in hand, every "where is that thing" question becomes a solvePnP call.

Summary

Calibrate by photographing a checkerboard from many angles, detecting corners with sub-pixel refinement, and running calibrateCamera - the output is the intrinsic matrix, the distortion coefficients, and a reprojection error you should verify is sub-pixel. Undo lens distortion with undistort (single images) or a precomputed remap (video), and verify by projecting known 3D corners back into a test photo. The payoff is solvePnP: with the calibration known, any object with known geometry (an ArUco marker) yields its 3D pose from a single frame - calibration is what turns a camera from an image source into a measurement instrument.