Skillversitas

Datasets and DataLoaders

The Dataset is the data's interface: the __len__ the count, the __getitem__ the (image, label) the index, and the transform the pipeline (the ToTensor, the Normalize, the Resize). The DataLoader is the batch's engine: the batch_size the group, the shuffle the train's random, the num_workers the process (the GPU's feed), and the collate_fn the batch the custom. The torchvision.datasets (the MNIST, the CIFAR10, the ImageFolder) is the ready, and the Subset/WeightedRandomSampler the split plus the imbalance. This chapter covers the Dataset (the __getitem__, the custom), the transform (the Compose, the RandomCrop), the DataLoader (the batch, the shuffle, the num_workers), and the ImageFolder (the dir the label).

The Dataset and the transform

# the DATASET (the):
from torch.utils.data import Dataset

class TextDataset(Dataset):
    def __init__(self, texts, labels, transform=None):
        self.texts = texts
        self.labels = labels
        self.transform = transform
    def __len__(self):
        return len(self.texts)
    def __getitem__(self, i):
        x = self.texts[i]          # the
        y = self.labels[i]         #   the
        if self.transform:
            x = self.transform(x)  # the
        return x, y

# the TRANSFORM (the):
#   the
import torchvision.transforms as T
tf = T.Compose([
    T.Resize((32, 32)),        # the
    T.RandomHorizontalFlip(),  #   the
    T.ToTensor(),              #   the
    T.Normalize((0.5,), (0.5,))])
#   the

The Dataset is the two: the __len__ (the count) and the __getitem__ (the (x, y) the index) - the custom the TextDataset (the texts the list, the labels the list, the transform the optional), and the torchvision.datasets.MNIST the ready (the download the auto, the transform the arg). The transform is the pipeline: the Compose the list (the Resize, the RandomHorizontalFlip, the ToTensor, the Normalize) - the Random the train only (the test the no the Random, the deterministic), and the ToTensor the PIL to the (C, H, W) the 0 to the 1 - the next section's DataLoader is the batch's engine, and the num_workers the GPU's feed.

The DataLoader

# the DATALOADER (the):
from torch.utils.data import DataLoader

loader = DataLoader(
    dataset,
    batch_size=64,        # the
    shuffle=True,         #   the
    num_workers=4,        #   the
    pin_memory=True,      #   the
)
for x, y in loader:
    print(x.shape)   # the the (64, 1, 28, 28) (the
                     #   the)
#   the
#   the

# the COLLATE (the):
#   the
def collate(batch):
    xs, ys = zip(*batch)
    return torch.stack(xs), torch.tensor(ys)
loader = DataLoader(dataset, batch_size=8,
    collate_fn=collate)   # the

# the IMAGEFOLDER (the):
#   the
from torchvision.datasets import ImageFolder
ds = ImageFolder("data/dogs", transform=tf)
#   the
#   the
#   the

The DataLoader is the batch's engine: the batch_size the group (the 64 the (64, 1, 28, 28)), the shuffle the train's random (the test the False the no the order), the num_workers the process (the 4 the CPU core, the GPU's feed the overlap), and the pin_memory the CUDA's fast (the page-locked, the H2D the speed). The collate_fn is the batch the custom (the variable the padding, the RNN the length), and the ImageFolder is the dir the label (the data/dogs/<class>/*.jpg the class the name, the ds.class_to_idx the map) - the train the shuffle=True, the test the shuffle=False, the num_workers the 4, and the rule is the one: the Dataset the __getitem__, the DataLoader the batch the shuffle the worker, the transform the pipeline the train only the Random - the next chapter's CNN is the model the loader feeds.

The Working Rules

  • The Dataset is the __len__ plus the __getitem__. The (x, y) the index, the custom the TextDataset, and the torchvision.datasets the ready (the MNIST, the CIFAR10, the ImageFolder the dir the label) - the transform the arg.
  • The transform is the Compose, the Random the train only. The Resize plus the RandomHorizontalFlip plus the ToTensor plus the Normalize the pipeline, and the test the no the Random (the deterministic, the eval the fair).
  • The DataLoader is the batch, the shuffle the train. The batch_size the group, the shuffle=True the train (the test the False), the num_workers the process (the 4 the GPU's feed), and the pin_memory the CUDA's fast.
  • The collate_fn is the custom, the ImageFolder the dir. The variable the padding (the RNN the length), and the ImageFolder the data/<class>/*.jpg the class (the class_to_idx the map, the transform the arg) - the no the annotation the file.

Summary

The Dataset is the data's interface: the __len__ the count, the __getitem__ the (x, y) the index, and the transform the pipeline (the Compose the Resize, the RandomHorizontalFlip, the ToTensor, the Normalize - the Random the train only). The torchvision.datasets (the MNIST, the CIFAR10, the ImageFolder the dir the label) is the ready, and the custom the TextDataset (the texts the list, the transform the optional). The DataLoader is the batch's engine: the batch_size the group (the (64, 1, 28, 28)), the shuffle the train's random (the test the False), the num_workers the process (the 4 the GPU's feed), the pin_memory the CUDA's fast, and the collate_fn the custom (the variable the padding). The rule is the one: the Dataset the __getitem__, the DataLoader the batch the shuffle the worker, the transform the pipeline - the next chapter's CNN is the model the loader feeds, and the Conv2d the feature the spatial the learn.