Networks of spiking neurons (SNNs) are frequently studied as models for
networks of neurons in the brain, but also as paradigm for novel energy
efficient computing hardware. In principle they are especially suitable for
computations in the temporal domain, such as speech processing, because their
computations are carried out via events in time and space. But so far they have
been lacking the capability to preserve information for longer time spans
during a computation, until it is updated or needed - like a register of a
digital computer. This function is provided to artificial neural networks
through Long Short-Term Memory (LSTM) units. We show here that SNNs attain
similar capabilities if one includes adapting neurons in the network.
Adaptation denotes an increase of the firing threshold of a neuron after
preceding firing. A substantial fraction of neurons in the neocortex of rodents
and humans has been found to be adapting. It turns out that if adapting neurons
are integrated in a suitable manner into the architecture of SNNs, the
performance of these enhanced SNNs, which we call LSNNs, for computation in the
temporal domain approaches that of artificial neural networks with LSTM-units.
In addition, the computing and learning capabilities of LSNNs can be
substantially enhanced through learning-to-learn (L2L) methods from machine
learning, that have so far been applied primarily to LSTM networks and
apparently never to SSNs.
This preliminary report on arXiv will be replaced by a more detailed version
in about a month.
Despite being originally inspired by the central nervous system, artificial
neural networks have diverged from their biological archetypes as they have
been remodeled to fit particular tasks. In this paper, we review several
possibilites to reverse map these architectures to biologically more realistic
spiking networks with the aim of emulating them on fast, low-power neuromorphic
hardware. Since many of these devices employ analog components, which cannot be
perfectly controlled, finding ways to compensate for the resulting effects
represents a key challenge. Here, we discuss three different strategies to
address this problem: the addition of auxiliary network components for
stabilizing activity, the utilization of inherently robust architectures and a
training method for hardware-emulated networks that functions without perfect
knowledge of the system's dynamics and parameters. For all three scenarios, we
corroborate our theoretical considerations with experimental results on
accelerated analog neuromorphic platforms.