Skillversitas

Recurrent Neural Networks

The RNN is the sequence's model: the time the step the recur (the h_t = f(x_t, h_{t-1})), the hidden the memory (the past the carry), and the vanish the gradient (the long the fade). The GRU is the gate (the update plus the reset, the 2 the gate), the LSTM the cell (the input, the forget, the output, the 3 the gate), and the clip the stability (the explode the guard). The Embedding is the token (the 100K vocab the 300 dim), the seq the pad (the variable the length), and the text the next the predict (the LM the task). This chapter covers the GRU (the gate, the hidden), the LSTM (the cell, the gate), the Embedding (the token), the pad (the seq), and the LM (the next the predict).

The GRU and the LSTM

# the GRU (the):
import torch.nn as nn

gru = nn.GRU(input_size=300, hidden_size=256,
    num_layers=2, batch_first=True)
x = torch.randn(8, 20, 300)   # the the (B, T, F)
out, h_n = gru(x)
print(out.shape)   # the the (8, 20, 256) (the the
                   #   the)
print(h_n.shape)   # the the (2, 8, 256) (the the
                   #   the)
#   the
#   the

# the LSTM (the):
#   the
lstm = nn.LSTM(300, 256, num_layers=2,
    batch_first=True)
out, (h_n, c_n) = lstm(x)
print(c_n.shape)   # the the (2, 8, 256) (the the
                   #   the)
#   the

The GRU is the gate: the update (the z, the past the keep or the new) plus the reset (the r, the past the no the look), the hidden the memory (the (B, T, 256) the out, the (2, B, 256) the h_n the layer), and the batch_first=True the (B, T, F) (the no the (T, B, F)). The LSTM is the cell: the input (the i, the new the write), the forget (the f, the old the drop), the output (the o, the cell the read), the c_n the cell state (the (2, B, 256)) - the vanish the LSTM's the less (the cell the highway, the gradient the flow). The bidirectional is the 2x the hidden (the past plus the future, the NLP the task the no the stream), and the clip is the stability (the clip_grad_norm_(parameters, 1.0) the explode the guard, the RNN chapter the required) - the next section's Embedding is the token, and the LM the task.

The Embedding and the LM

# the EMBEDDING (the):
emb = nn.Embedding(100000, 300)   # the
x = torch.randint(0, 100000, (8, 20))  # the the
v = emb(x)   # the the (8, 20, 300) (the
             #   the)
#   the
#   the

# the LM (the):
#   the
class LM(nn.Module):
    def __init__(self, vocab=100000, dim=300, hid=256):
        super().__init__()
        self.emb = nn.Embedding(vocab, dim)
        self.gru = nn.GRU(dim, hid, batch_first=True)
        self.head = nn.Linear(hid, vocab)
    def forward(self, x):
        v = self.emb(x)
        out, _ = self.gru(v)
        return self.head(out)   # the the (B, T, V)
#   the
#   the

The Embedding is the token: the nn.Embedding(100000, 300) the 100K vocab the 300 dim (the index the vector, the 300 the meaning the learn), the (8, 20) the (8, 20, 300) (the token the dense), and the pad is the seq the variable (the pad the 0, the pack_padded_sequence the no the compute). The LM is the next the predict: the emb the v, the gru the out, the head the 100K class (the token the t+1 the predict, the shift the x[:-1] the y[1:]), and the cross_entropy the vocab the 100K the softmax (the GPU the only, the CPU the slow) - the transformer (the next chapter) is the RNN the replace: the parallel the train, the attention the memory the long, and the LM the GPT the scale.

The Working Rules

  • The GRU is the gate, the hidden the memory. The update plus the reset (the 2 the gate), the out the (B, T, 256), the h_n the (L, B, 256), and the batch_first=True the (B, T, F) (the no the (T, B, F) the transpose).
  • The LSTM is the cell, the vanish the less. The input plus the forget plus the output (the 3 the gate), the c_n the cell (the highway the gradient), and the bidirectional the 2x the hidden (the no the stream, the NLP the task).
  • The Embedding is the token, the pad the seq. The 100K vocab the 300 dim (the index the vector), the (8, 20) the (8, 20, 300), and the pad the variable (the pack_padded_sequence the no the compute, the length the mask).
  • The LM is the next, the clip the stability. The emb the gru the head (the 100K class the token), the shift the x[:-1] the y[1:], and the clip_grad_norm_(parameters, 1.0) the explode the guard (the RNN the required, the GPU the only the vocab the 100K).

Summary

The RNN is the sequence's model: the time the step the recur (the h_t = f(x_t, h_{t-1})), the hidden the memory (the past the carry), and the vanish the gradient (the long the fade). The GRU is the gate (the update plus the reset, the out the (B, T, 256), the h_n the (L, B, 256)), the LSTM the cell (the input plus the forget plus the output, the c_n the highway), and the clip the stability (the clip_grad_norm_ the 1.0 the explode the guard). The Embedding is the token (the 100K vocab the 300 dim, the pad the seq the variable), and the LM is the next the predict (the emb the gru the head the 100K class, the shift the x[:-1] the y[1:]) - the transformer (the next chapter) is the RNN the replace: the parallel the train, the attention the memory the long, the GPT the scale.