Neural Network Memory Criticality: from Echo State to HiPPO and Beyond

Using artificial intelligence (AI) systems to represent, model, and predict sequential scientific data is of growing interest. Neural networks can often effectively model such sequences if information is propagated through the network at criticality, neither decaying nor exploding exponentially. Applying the dynamical systems approach, we use the Lyapunov exponent to define criticality. Criticality analysis can be used for models at different levels of complexity, including echo state networks (ESN), long short-term memory (LSTM), and high-order polynomial projection operators (HiPPO) being used for numerous scientific AI applications. We relate the echo state property to criticality in the ESN. We observe in LSTM how gated memory constant error propagation solves problems with fully connected recurrence by preserving criticality in sub-networks. The HiPPO memory generalizes findings from gated memory, with essential operators preserving criticality. Our analysis can inform criticality constraints on future scientific memory models at different complexities and scales.