Long Short-Term Memory Cell

The long short-term memory LSTM in the following cell, which is shown schematically in Fig. 17, is a more involved type of recurrent cell. The LSTM cell maintains an inner cell state c in addition to the state h that was already used for the basic RNN cells, as well as additional inner layers called input gate i, forget gate f and output gate o. With that, the LSTM cell is capable of deciding whether to retain particular events over longer periods of time steps than basic RNN cells.

With an LSTM cell, singular events at a previous point in time can influence the output of the cell possibly as long as the whole time span of interest. LSTM cells are therefore well suited to model scenarios where non-periodic or single events much earlier remain important over long periods of time. This comes at the cost of longer training times, as the complexity of the cell, and with it the number of parameters, increases substantially.

Fig. 17: Schematic layout of one LSTM cell. Input x_t and previous cell state h_t-1 are fed to the cells’ input activation function a1, as well as to the input gate i, forget gate f and output gate o. The cells’ inner state is symbolized by a larger node c. Element-wise multiplication and combination with the cells’ inner state c at both steps t and t−1 and the output activation a2 produce the cells’ output. This output can then be fed to an output projection layer p in order to reduce the amount of trainable parameters. This operation then results in the output y_t and cell state h_t at time step t. Blue dashed connections symbolize so-called peephole connections as explained in the text.

In addition to the regular layout of an LSTM cell, Fig. 17 shows blue dashed lines, which symbolize so-called peephole connections. Peephole connections mean that activations for the input and forget gates now process the current inner cell state c at time step t − 1, while the output gate activation gets c after it has been updated, that is at time step t.