Skip, xfail, and Expected Failures

The skip and the xfail are the two ways a test does not run as the plain pass: the skip is the not-run (the environment, the not-implemented) and the xfail is the run-but-expected-fail (the known bug, the platform quirk). The skipif is the conditional skip (the sys.version_info, the sys.platform, the os.environ), and the xfail carries the strict flag (the XPASS the failure when the bug is fixed, the mark forced to be removed) and the run flag (the run=False the not-run, the xfail as the skip). This chapter covers the skip/skipif (the condition, the allow_module_level), the xfail (the strict, the run, the raises), and the report's SKIPPED/XFAIL/XPASS counts.

The skip and the skipif

# the SKIP: the unconditional (the reason the report)
import pytest, sys

@pytest.mark.skip(reason="needs the display (the CI's
                          the headless, the no X)")
def test_gui():
    ...

# the SKIPIF: the conditional (the the environment)
@pytest.mark.skipif(sys.version_info < (3, 10),
                    reason="the match-statement the
                            the 3.10+")
def test_match():
    match word_count(""):
        case 0: pass

@pytest.mark.skipif(sys.platform != "linux",
                    reason="the /proc the the linux")
def test_proc():
    ...
# the COLLECTION: the SKIP (the reason in the
#   report's line) - the NO RUN, the item shown as
#   SKIPPED with the reason

The skip is the unconditional (the reason the report's line, the test collected but not run) and the skipif is the conditional (the sys.version_info, the sys.platform, the os.environ - the environment the test needs, the condition the module-level or the test-level). The allow_module_level=True (the ini) permits the pytestmark = pytest.mark.skipif(...) at the module level (the whole file the skip, the no per-test repetition), and the report's SKIPPED count plus the -rs flag (the pytest -rs the skip reason shown) are the visibility - the skip is not the silence, the reason is the tracked. The rule is the one: the skip is the environment or the not-implemented (the reason the why), never the "flaky so skip" (the hide, the swallowed bug) - the next section's xfail is the run-but-expected-fail, the other half of the not-plain-pass.

The xfail: The strict and the run

# the XFAIL: the expected-fail (the known bug, the
#   platform quirk) - the FAIL is the XFAIL (green),
#   the PASS is the XPASS (the bug fixed, the mark stale)
import pytest
from texttools import word_count

@pytest.mark.xfail(reason="bug #42: the spaced edge")
def test_spaced_edge():
    assert word_count("  a  ") == 1   # the current 2

# the STRICT: the XPASS is the FAILURE (the
#   the bug FIXED the mark must be REMOVED,
#   the suite FORCES the cleanup)
@pytest.mark.xfail(reason="bug #43", strict=True)
def test_strict():
    ...   # the PASS -> the FAILED (the XPASS(strict))

# the RUN=False: the xfail AS the skip (the
#   the not-run, the)
@pytest.mark.xfail(run=False, reason="the
                              the")
def test_not_run():
    ...   # the NO RUN (the
         #   the)
# the the REPORT: the the XFAIL (the the green),
#   the the XPASS (the), the the
#   the XPASS(strict) (the the FAILED)

The xfail is the expected-fail: the test runs, the failure is the XFAIL (the green, the known bug documented with the reason), and the pass is the XPASS (the bug fixed, the mark stale). The strict=True turns the XPASS into the failure (the XPASS(strict) - the bug fixed but the mark not removed, the suite forces the cleanup, the no stale xfail rot), and the run=False makes the xfail a skip (the not-run, the platform the test cannot even start). The raises argument restricts the xfail to the specific exception (the xfail(raises=ValueError) - the other exception is the real failure, not the expected). The rule is the one: the xfail is the known bug with the issue reference (the reason="bug #42"), the strict=True the default the new project (the xfail_strict ini), and the skip is the not-run while the xfail is the run-but-fail - the two halves of the not-plain-pass, the SKIPPED/XFAIL/XPASS the report's counts the -rxs shows.

The Working Rules

The skip is the not-run, the skipif the conditional. The reason the report's line (the -rs shown), the condition the sys.version_info/sys.platform/os.environ, and the allow_module_level the module-level pytestmark - the skip the environment or the not-implemented, never the flaky-hide.
The xfail is the run-but-expected-fail, the reason the issue. The fail is the XFAIL (the green), the pass is the XPASS (the stale), and the reason="bug #42" the tracked reference - the xfail documents, the skip defers.
The strict=True forces the cleanup. The XPASS(strict) the failure (the bug fixed, the mark not removed), the xfail_strict = true ini the project default - the no stale xfail the suite rots by.
The run=False is the xfail-as-skip, the raises the specific. The run=False the not-run (the platform the cannot-start), the raises=ValueError the other-exception the real failure - the two refine the xfail's boundary.

Summary

The skip is the not-run (the reason the report's line, the SKIPPED the count) and the skipif is the conditional (the sys.version_info, the sys.platform, the os.environ - the environment the test needs, the allow_module_level the module-level pytestmark). The xfail is the run-but-expected-fail: the failure is the XFAIL (the green, the known bug the reason="bug #42" documents), the pass is the XPASS (the stale mark), the strict=True (the xfail_strict ini) turns the XPASS into the failure (the cleanup forced), and the run=False makes the xfail a skip (the not-run) while the raises restricts the expected exception. The -rxs flag shows the SKIP/XFAIL/XPASS reasons in the report, and the rule is the split: the skip is the environment or the not-implemented (the reason the why), the xfail is the known bug (the issue the reference), and neither is the flaky-hide - the next chapter runs the unittest tests the pytest collects, the other framework's tests in the same suite.