Skillversitas

IO Caching

Notebook workflows are full of expensive I/O: downloading datasets, reading large files, querying slow services. IPython's %cache magic and the %%store magic let you persist values between sessions so a restart does not force a full re-download or recompute.

%%store - Persist Variables Across Sessions

%%store
big_list = list(range(1000))

In a new session (or a restarted kernel):

%store -r
%store -r big_list
print(len(big_list))  # 1000

Storing works per kernel session; combine it with checkpoints for full session recovery.

%cache - Filesystem Caching for Functions

The more common pattern is a function-level cache that reads once and serves from disk afterward. IPython's %cache decorator (via the ipykernel cache tools) and functools.lru_cache cover in-memory needs:

import functools, json, os, time

@functools.lru_cache(maxsize=None)
def load_config(name: str):
    time.sleep(2)  # simulate slow disk/network read
    with open(f"configs/{name}.json") as f:
        return json.load(f)

print(load_config("app"))   # slow first call
print(load_config("app"))   # instant, from cache

A Manual Disk Cache

import hashlib, pathlib, requests

CACHE = pathlib.Path("cache")
CACHE.mkdir(exist_ok=True)

def get(url: str, ttl: int = 3600):
    key = hashlib.sha256(url.encode()).hexdigest()[:16]
    p = CACHE / key
    if p.exists() and time.time() - p.stat().st_mtime < ttl:
        return p.read_text()
    text = requests.get(url, timeout=30).text
    p.write_text(text)
    return text

When to Cache

  • Remote data that changes daily (reference tables, exchange rates): cache with a TTL.
  • Expensive preprocessing: cache the transformed DataFrame on disk (parquet) and reload in seconds.
  • Never cache live monitoring data or anything the analysis must observe changing.

Summary

Use %%store to keep variables alive across kernel restarts, functools.lru_cache for repeated function calls, and a small disk-cache helper with a TTL for remote data. Caching turns a ten-minute notebook startup into a two-second one without changing the analysis.