Recurrent Neural Network
An RNN applies the same transition to a sequence, threading a hidden state through time: h_t = f(h_{t−1}, x_t). Plain RNNs struggle to learn long dependencies.
Definition
A recurrent net shares weights across time and carries a state. In theory it can map arbitrary-length sequences to outputs. In practice vanishing/exploding gradients made vanilla RNNs poor at long context — the problem LSTM and later attention solved.
Why it matters
Sequence problems (language, tick data, speech) need a memory of the past. RNNs were the first scalable answer. Understanding their failure mode is why the field moved to gated recurrences and then Transformers.
Case
A vanilla RNN language model forgets the subject of a sentence by the time it reaches the verb in a long clause. LSTM gates, then attention, were invented for that exact failure.
How to read it
If you still ship an RNN, measure performance versus context length. If quality dies after a few dozen steps, you do not have a sequence model — you have a short FIR filter with extra ceremony.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- Finding Structure in TimeElman · 1990 · Cognitive ScienceSimple recurrent nets as sequence models.