Energy-reliability limits in nanoscale neural networks

New device technologies such as spintronics, carbon nanotubes, and nanoscale CMOS incur random transient failures, where the failure probability is governed by the energy consumption through energy-failure functions. At the same time, there is growing use of deep neural networks for many inference applications, and specialized hardware is being developed with these nanotechnologies as physical substrates. It is important to understand the basic energy-reliability limits. Using Pippenger's mutual information propagation technique (extended to directed acyclic graphs), together with optimization, we obtain a lower bound on energy consumption in multilayer binary neural networks for a given reliability. We also obtain a simple energy allocation rule for neurons in the different layers of the neural network. The mathematical results also provide insight into mammalian neuroenergetics of brain regions involved in sensory processing.