
In physics, biology and engineering, network systems abound. How does the
connectivity of a network system combine with the behavior of its individual
components to determine its collective function? We approach this question for
networks with linear timeinvariant dynamics by relating internal network
feedbacks to the statistical prevalence of connectivity motifs, a set of
surprisingly simple and local statistics of connectivity. This results in a
reduced order model of the network inputoutput dynamics in terms of motifs
structures. As an example, the new formulation dramatically simplifies the
classic ErdosRenyi graph, reducing the overall network behavior to one
proportional feedback wrapped around the dynamics of a single node. For general
networks, higherorder motifs systematically provide further layers and types
of feedback to regulate the network response. Thus, the local connectivity
shapes temporal and spectral processing by the network as a whole, and we show
how this enables robust, yet tunable, functionality such as extending the time
constant with which networks remember past signals. The theory also extends to
networks composed from heterogeneous nodes with distinct dynamics and
connectivity, and patterned input to (and readout from) subsets of nodes. These
statistical descriptions provide a powerful theoretical framework to understand
the functionality of realworld network systems, as we illustrate with examples
including the mouse brain connectome.

A major obstacle to understanding neural coding and computation is the fact
that experimental recordings typically sample only a small fraction of the
neurons in a circuit. Measured neural properties are skewed by interactions
between recorded neurons and the "hidden" portion of the network. To properly
interpret neural data and determine how biological structure gives rise to
neural circuit function, we thus need a better understanding of the
relationships between measured effective neural properties and the true
underlying physiological properties. Here, we focus on how the effective
spatiotemporal dynamics of the synaptic interactions between neurons are
reshaped by coupling to unobserved neurons. We find that the effective
interactions from a presynaptic neuron $r'$ to a postsynaptic neuron $r$ can
be decomposed into a sum of the true interaction from $r'$ to $r$ plus
corrections from every directed path from $r'$ to $r$ through unobserved
neurons. Importantly, the resulting formula reveals when the hidden units
haveor do not havemajor effects on reshaping the interactions among
observed neurons. As a particular example of interest, we derive a formula for
the impact of hidden units in random networks with "strong"
couplingconnection weights that scale with $1/\sqrt{N}$, where $N$ is the
network size, precisely the scaling observed in recent experiments. With this
quantitative relationship between measured and true interactions, we can study
how network properties shape effective interactions, which properties are
relevant for neural computations, and how to manipulate effective interactions.

The Hawkes process is a class of point processes whose future depends on
their own history. Previous theoretical work on the Hawkes process is limited
to a special case in which a past event can only increase the occurrence of
future events, and the link function is linear. However, in neuronal networks
and other realworld applications, inhibitory relationships may be present, and
the link function may be nonlinear. In this paper, we develop a new approach
for investigating the properties of the Hawkes process without the restriction
to mutual excitation or linear link functions. To this end, we employ a
thinning process representation and a coupling construction to bound the
dependence coefficient of the Hawkes process. Using recent developments on
weakly dependent sequences, we establish a concentration inequality for
secondorder statistics of the Hawkes process. We apply this concentration
inequality to crosscovariance analysis in the highdimensional regime, and we
verify the theoretical claims with simulation studies.

Unraveling the interplay of excitation and inhibition within
rhythmgenerating networks remains a fundamental issue in neuroscience. We use
a biophysical model to investigate the different roles of local and longrange
inhibition in the respiratory network, a key component of which is the
preB\"otzinger complex inspiratory microcircuit. Increasing inhibition within
the microcircuit results in a limited number of outofphase neurons before
rhythmicity and synchrony degenerate. Thus, unstructured local inhibition is
destabilizing and cannot support the generation of more than one rhythm. A
twophase rhythm requires restructuring the network into two microcircuits
coupled by longrange inhibition in the manner of a halfcenter. In this
context, inhibition leads to greater stability of the two outofphase rhythms.
We support our computational results with in vitro recordings from mouse
preB\"otzinger complex. Partial excitation block leads to increased rhythmic
variability, but this recovers following blockade of inhibition. Our results
support the idea that local inhibition in the preB\"otzinger complex is
present to allow for descending control of synchrony or robustness to adverse
conditions like hypoxia. We conclude that the balance of inhibition and
excitation determines the stability of rhythmogenesis, but with opposite roles
within and between areas. These different inhibitory roles may apply to a
variety of rhythmic behaviors that emerge in widespread pattern generating
circuits of the nervous system.

Recent experimental advances are producing an avalanche of data on both
neural connectivity and neural activity. To take full advantage of these two
emerging datasets we need a framework that links them, revealing how collective
neural activity arises from the structure of neural connectivity and intrinsic
neural dynamics. This problem of {\it structuredriven activity} has drawn
major interest in computational neuroscience. Existing methods for relating
activity and architecture in spiking networks rely on linearizing activity
around a central operating point and thus fail to capture the nonlinear
responses of individual neurons that are the hallmark of neural information
processing. Here, we overcome this limitation and present a new relationship
between connectivity and activity in networks of nonlinear spiking neurons by
developing a diagrammatic fluctuation expansion based on statistical field
theory. We explicitly show how recurrent network structure produces pairwise
and higherorder correlated activity, and how nonlinearities impact the
networks' spiking activity. Our findings open new avenues to investigating how
singleneuron nonlinearitiesincluding those of different cell
typescombine with connectivity to shape population activity and function.

An essential step toward understanding neural circuits is linking their
structure and their dynamics. In general, this relationship can be almost
arbitrarily complex. Recent theoretical work has, however, begun to identify
some broad principles underlying collective spiking activity in neural
circuits. The first is that local features of network connectivity can be
surprisingly effective in predicting global statistics of activity across a
network. The second is that, for the important case of large networks with
excitatoryinhibitory balance, correlated spiking persists or vanishes
depending on the spatial scales of recurrent and feedforward connectivity. We
close by showing how these ideas, together with plasticity rules, can help to
close the loop between network structure and activity statistics.

Sensory neurons give highly variable responses to stimulation, which can
limit the amount of stimulus information available to downstream circuits. Much
work has investigated the factors that affect the amount of information encoded
in these population responses, leading to insights about the role of
covariability among neurons, tuning curve shape, etc. However, the
informativeness of neural responses is not the only relevant feature of
population codes; of potentially equal importance is how robustly that
information propagates to downstream structures. For instance, to quantify the
retina's performance, one must consider not only the informativeness of the
optic nerve responses, but also the amount of information that survives the
spikegenerating nonlinearity and noise corruption in the next stage of
processing, the lateral geniculate nucleus. Our study identifies the set of
covariance structures for the upstream cells that optimize the ability of
information to propagate through noisy, nonlinear circuits. Within this optimal
family are covariances with "differential correlations", which are known to
reduce the information encoded in neural population activities. Thus,
covariance structures that maximize information in neural population codes, and
those that maximize the ability of this information to propagate, can be very
different.

The primate visual system has an exquisite ability to discriminate partially
occluded shapes. Recent electrophysiological recordings suggest that response
dynamics in intermediate visual cortical area V4, shaped by feedback from
prefrontal cortex (PFC), may play a key role. To probe the algorithms that may
underlie these findings, we build and test a model of V4 and PFC interactions
based on a hierarchical predictive coding framework. We propose that
probabilistic inference occurs in two steps. Initially, V4 responses are driven
solely by bottomup sensory input and are thus strongly influenced by the level
of occlusion. After a delay, V4 responses combine both feedforward input and
feedback signals from the PFC; the latter reflect predictions made by PFC about
the visual stimulus underlying V4 activity. We find that this model captures
key features of V4 and PFC dynamics observed in experiments. Specifically, PFC
responses are strongest for occluded stimuli and delayed responses in V4 are
less sensitive to occlusion, supporting our hypothesis that the feedback
signals from PFC underlie robust discrimination of occluded shapes. Thus, our
study proposes that area V4 and PFC participate in hierarchical inference, with
feedback signals encoding topdown predictions about occluded shapes.

Wholebrain neural connectivity data are now available from viral tracing
experiments, which reveal the connections between a source injection site and
elsewhere in the brain. These hold the promise of revealing spatial patterns of
connectivity throughout the mammalian brain. To achieve this goal, we seek to
fit a weighted, nonnegative adjacency matrix among 100 $\mu$m brain "voxels"
using viral tracer data. Despite a multiyear experimental effort, injections
provide incomplete coverage, and the number of voxels in our data is orders of
magnitude larger than the number of injections, making the problem severely
underdetermined. Furthermore, projection data are missing within the injection
site because local connections there are not separable from the injection
signal.
We use a novel machinelearning algorithm to meet these challenges and
develop a spatially explicit, voxelscale connectivity map of the mouse visual
system. Our method combines three features: a matrix completion loss for
missing data, a smoothing spline penalty to regularize the problem, and
(optionally) a low rank factorization. We demonstrate the consistency of our
estimator using synthetic data and then apply it to newly available Allen Mouse
Brain Connectivity Atlas data for the visual system. Our algorithm is
significantly more predictive than current state of the art approaches which
assume regions to be homogeneous. We demonstrate the efficacy of a low rank
version on visual cortex data and discuss the possibility of extending this to
a wholebrain connectivity matrix at the voxel scale.

Highly connected recurrent neural networks often produce chaotic dynamics,
meaning their precise activity is sensitive to small perturbations. What are
the consequences for how such networks encode streams of temporal stimuli? On
the one hand, chaos is a strong source of randomness, suggesting that small
changes in stimuli will be obscured by intrinsically generated variability. On
the other hand, recent work shows that the type of chaos that occurs in spiking
networks can have a surprisingly lowdimensional structure, suggesting that
there may be "room" for fine stimulus features to be precisely resolved. Here
we show that strongly chaotic networks produce patterned spikes that reliably
encode timedependent stimuli: using a decoder sensitive to spike times on
timescales of 10's of ms, one can easily distinguish responses to very similar
inputs. Moreover, recurrence serves to distribute signals throughout chaotic
networks so that small groups of cells can encode substantial information about
signals arriving elsewhere. A conclusion is that the presence of strong chaos
in recurrent networks does not prohibit precise stimulus encoding.

While recent recordings from neural populations show beyondpairwise, or
higherorder correlations (HOC), we have little understanding of how HOC arise
from network interactions and of how they impact encoded information. Here, we
show that input nonlinearities imply HOC in spinglasstype statistical models.
We then discuss one such model with parameterized pairwise and higherorder
interactions, revealing conditions under which beyondpairwise interactions
increase the mutual information between a given stimulus type and the
population responses. For jointly Gaussian stimuli, coding performance is
improved by shaping output HOC only when neural firing rates are constrained to
be low. For stimuli with skewed probability distributions (like natural image
luminances), performance improves for all firing rates. Our work suggests
surprising connections between nonlinear integration of neural inputs, stimulus
statistics, and normative theories of population coding. Moreover, it suggests
that the inclusion of beyondpairwise interactions could improve the
performance of Boltzmann machines for machine learning and signal processing
applications.

Which statistical features of spiking activity matter for how stimuli are
encoded in neural populations? A vast body of work has explored how firing
rates in individual cells and correlations in the spikes of cell pairs impact
coding. But little is known about how higherorder correlations, which describe
simultaneous firing in triplets and larger ensembles of cells, impact encoded
stimulus information. Here, we take a first step toward closing this gap. We
vary triplet correlations in small (~10 cell) neural populations while keeping
single cell and pairwise statistics fixed at typically reported values. For
each value of triplet correlations, we estimate the performance of the neural
population on a twostimulus discrimination task. We identify a predominant way
that such triplet correlations can strongly enhance coding: if triplet
correlations differ for the two stimuli, they skew the response distributions
of the two stimuli apart from each other, separating them and making them
easier to distinguish. This coding benefit does not occur when both stimuli
elicit similar triplet correlations. These results indicate that higherorder
correlations could have a strong effect on population coding. Finally, we
calculate how many samples are necessary to accurately measure spiking
correlations of this type, providing an estimate of the necessary recording
times in experiments.

Large networks of sparsely coupled, excitatory and inhibitory cells occur
throughout the brain. A striking feature of these networks is that they are
chaotic. How does this chaos manifest in the neural code? Specifically, how
variable are the spike patterns that such a network produces in response to an
input signal? To answer this, we derive a bound for the entropy of multicell
spike pattern distributions in large recurrent networks of spiking neurons
responding to fluctuating inputs. The analysis is based on results from random
dynamical systems theory and is complimented by detailed numerical simulations.
We find that the spike pattern entropy is an order of magnitude lower than what
would be extrapolated from single cells. This holds despite the fact that
network coupling becomes vanishingly sparse as network size grows  a
phenomenon that depends on ``extensive chaos," as previously discovered for
balanced networks without stimulus drive. Moreover, we show how spike pattern
entropy is controlled by temporal features of the inputs. Our findings provide
insight into how neural networks may encode stimuli in the presence of
inherently chaotic dynamics.

Over repeat presentations of the same stimulus, sensory neurons show variable
responses. This "noise" is typically correlated between pairs of cells, and a
question with rich history in neuroscience is how these noise correlations
impact the population's ability to encode the stimulus. Here, we consider a
very general setting for population coding, investigating how information
varies as a function of noise correlations, with all other aspects of the
problem  neural tuning curves, etc.  held fixed. This work yields unifying
insights into the role of noise correlations. These are summarized in the form
of theorems, and illustrated with numerical examples involving neurons with
diverse tuning curves. Our main contributions are as follows.
(1) We generalize previous results to prove a sign rule (SR)  if noise
correlations between pairs of neurons have opposite signs vs. their signal
correlations, then coding performance will improve compared to the independent
case. This holds for three different metrics of coding performance, and for
arbitrary tuning curves and levels of heterogeneity. This generality is true
for our other results as well.
(2) As also pointed out in the literature, the SR does not provide a
necessary condition for good coding. We show that a diverse set of correlation
structures can improve coding. Many of these violate the SR, as do
experimentally observed correlations. There is structure to this diversity: we
prove that the optimal correlation structures must lie on boundaries of the
possible set of noise correlations.
(3) We provide a novel set of necessary and sufficient conditions, under
which the coding performance (in the presence of noise) will be as good as it
would be if there were no noise present at all.

How does connectivity impact network dynamics? We address this question by
linking network characteristics on two scales. On the global scale we consider
the coherence of overall network dynamics. We show that such \emph{global
coherence} in activity can often be predicted from the \emph{local structure}
of the network. To characterize local network structure we use "motif
cumulants," a measure of the deviation of pathway counts from those expected in
a minimal probabilistic network model.
We extend previous results in three ways. First, we give a new combinatorial
formulation of motif cumulants that relates to the allied concept in
probability theory. Second, we show that the link between global network
dynamics and local network architecture is strongly affected by heterogeneity
in network connectivity. However, we introduce a networkpartitioning method
that recovers a tight relationship between architecture and dynamics. Third,
for a particular set of models we generalize the underlying theory to treat
dynamical coherence at arbitrary orders (i.e. triplet correlations, and
beyond). We show that at any order only a highly restricted set of motifs
impact dynamical correlations.

The collective dynamics of neural populations are often characterized in
terms of correlations in the spike activity of different neurons. Open
questions surround the basic nature of these correlations. In particular, what
leads to higherorder correlations  correlations in the population activity
that extend beyond those expected from cell pairs? Here, we examine this
question for a simple, but ubiquitous, circuit feature: common fluctuating
input arriving to spiking neurons of integrateandfire type. We show that
leads to strong higherorder correlations, as for earlier work with discrete
threshold crossing models. Moreover, we find that the same is true for another
widely used, doublystochastic model of neural spiking, the linearnonlinear
cascade. We explain the surprisingly strong connection between the collective
dynamics produced by these models, and conclude that higherorder correlations
are both broadly expected and possible to capture with surprising accuracy by
simplified (and tractable) descriptions of neural spiking.

Emerging technologies are revealing the spiking activity in ever larger
neural ensembles. Frequently, this spiking is far from independent, with
correlations in the spike times of different cells. Understanding how such
correlations impact the dynamics and function of neural ensembles remains an
important open problem. Here we describe a new, generative model for correlated
spike trains that can exhibit many of the features observed in data. Extending
prior work in mathematical finance, this generalized thinning and shift (GTaS)
model creates marginally Poisson spike trains with diverse temporal correlation
structures. We give several examples which highlight the model's flexibility
and utility. For instance, we use it to examine how a neural network responds
to highly structured patterns of inputs. We then show that the GTaS model is
analytically tractable, and derive cumulant densities of all orders in terms of
model parameters. The GTaS framework can therefore be an important tool in the
experimental and theoretical exploration of neural dynamics.

Biological information processing is often carried out by complex networks of
interconnected dynamical units. A basic question about such networks is that of
reliability: if the same signal is presented many times with the network in
different initial states, will the system entrain to the signal in a repeatable
way? Reliability is of particular interest in neuroscience, where large,
complex networks of excitatory and inhibitory cells are ubiquitous. These
networks are known to autonomously produce strongly chaotic dynamics  an
obvious threat to reliability. Here, we show that such chaos persists in the
presence of weak and strong stimuli, but that even in the presence of chaos,
intermittent periods of highly reliable spiking often coexist with unreliable
activity. We elucidate the local dynamical mechanisms involved in this
intermittent reliability, and investigate the relationship between this
phenomenon and certain timedependent attractors arising from the dynamics. A
conclusion is that chaotic dynamics do not have to be an obstacle to precise
spike responses, a fact with implications for signal coding in large networks.

Recent experimental and computational evidence suggests that several
dynamical properties may characterize the operating point of functioning neural
networks: critical branching, neutral stability, and production of a wide range
of firing patterns. We seek the simplest setting in which these properties
emerge, clarifying their origin and relationship in random, feedforward
networks of McCullochsPitts neurons. Two key parameters are the thresholds at
which neurons fire spikes, and the overall level of feedforward connectivity.
When neurons have low thresholds, we show that there is always a connectivity
for which the properties in question all occur: that is, these networks
preserve overall firing rates from layer to layer and produce broad
distributions of activity in each layer. This fails to occur, however, when
neurons have high thresholds. A key tool in explaining this difference is
eigenstructure of the resulting meanfield Markov chain, as this reveals which
activity modes will be preserved from layer to layer. We extend our analysis
from purely excitatory networks to more complex models that include inhibition
and 'local' noise, and find that both of these features extend the parameter
ranges over which networks produce the properties of interest.

Stimulus from the environment that guides behavior and informs decisions is
encoded in the firing rates of neural populations. Each neuron in the
populations, however, does not spike independently: spike events are correlated
from cell to cell. To what degree does this apparent redundancy impact the
accuracy with which decisions can be made, and the computations that are
required to optimally decide? We explore these questions for two illustrative
models of correlation among cells. Each model is statistically identical at the
level of pairs cells, but differs in higherorder statistics that describe the
simultaneous activity of larger cell groups. We find that the presence of
correlations can diminish the performance attained by an ideal decision maker
to either a small or large extent, depending on the nature of the higherorder
interactions. Moreover, while this optimal performance can in some cases be
obtained via the standard integrationtobound operation, in others it requires
a nonlinear computation on incoming spikes. Overall, we conclude that a given
level of pairwise correlationseven when restricted to identical neural
populationsmay not always indicate redundancies that diminish decision making
performance.

Motifs are patterns of subgraphs of complex networks. We studied the impact
of such patterns of connectivity on the level of correlated, or synchronized,
spiking activity among pairs of cells in a recurrent network model of integrate
and fire neurons. For a range of network architectures, we find that the
pairwise correlation coefficients, averaged across the network, can be closely
approximated using only three statistics of network connectivity. These are the
overall network connection probability and the frequencies of two secondorder
motifs: diverging motifs, in which one cell provides input to two others, and
chain motifs, in which two cells are connected via a third intermediary cell.
Specifically, the prevalence of diverging and chain motifs tends to increase
correlation. Our method is based on linear response theory, which enables us to
express spiking statistics using linear algebra, and a resumming technique,
which extrapolates from second order motifs to predict the overall effect of
coupling on network correlation. Our motifbased results seek to isolate the
effect of network architecture perturbatively from a known network state.

Describing the collective activity of neural populations is a daunting task:
the number of possible patterns grows exponentially with the number of cells,
resulting in practically unlimited complexity. Recent empirical studies,
however, suggest a vast simplification in how multineuron spiking occurs: the
activity patterns of some circuits are nearly completely captured by pairwise
interactions among neurons. Why are such pairwise models so successful in some
instances, but insufficient in others? Here, we study the emergence of
higherorder interactions in simple circuits with different architectures and
inputs. We quantify the impact of higherorder interactions by comparing the
responses of mechanistic circuit models vs. "null" descriptions in which all
higherthanpairwise correlations have been accounted for by lower order
statistics, known as pairwise maximum entropy models.
We find that bimodal input signals produce larger deviations from pairwise
predictions than unimodal inputs for circuits with local and global
connectivity. Moreover, recurrent coupling can accentuate these deviations, if
coupling strengths are neither too weak nor too strong. A circuit model based
on intracellular recordings from ON parasol retinal ganglion cells shows that a
broad range of light signals induce unimodal inputs to spike generators, and
that coupling strengths produce weak effects on higherorder interactions. This
provides a novel explanation for the success of pairwise models in this system.
Overall, our findings identify circuitlevel mechanisms that produce and fail
to produce higherorder spiking statistics in neural ensembles.

Modelbased studies of auditory nerve responses to electrical stimulation can
provide insight into the functioning of cochlear implants. Ideally, these
studies can identify limitations in sound processing strategies and lead to
improved methods for providing sound information to cochlear implant users. To
accomplish this, models must accurately describe auditory nerve spiking while
avoiding excessive complexity that would preclude largescale simulations of
populations of auditory nerve fibers and obscure insight into the mechanisms
that influence neural encoding of sound information. In this spirit, we develop
a point process model of the auditory nerve that provides a compact and
accurate description of neural responses to electric stimulation. Inspired by
the framework of generalized linear models, the proposed model consists of a
cascade of linear and nonlinear stages. We show how each of these stages can be
associated with biophysical mechanisms and related to models of neuronal
dynamics. Moreover, we derive a semianalytical procedure that uniquely
determines each parameter in the model on the basis of fundamental statistics
from recordings of single fiber responses to electric stimulation, including
threshold, relative spread, jitter, and chronaxie. The model also accounts for
refractory and summation effects that influence the responses of auditory nerve
fibers to high pulse rate stimulation. Throughout, we compare model predictions
to published physiological data and explain differences in auditory nerve
responses to high and low pulse rate stimulation. We close by performing an
ideal observer analysis of simulated spike trains in response to sinusoidally
amplitude modulated stimuli and find that carrier pulse rate does not affect
modulation detection thresholds.

A key step in many perceptual decision tasks is the integration of sensory
inputs over time, but fundamental questions remain about how this is
accomplished in neural circuits. One possibility is to balance decay modes of
membranes and synapses with recurrent excitation. To allow integration over
long timescales, however, this balance must be precise; this is known as the
fine tuning problem. The need for fine tuning can be overcome via a
ratchetlike mechanism, in which momentary inputs must be above a preset limit
to be registered by the circuit. The degree of this ratcheting embodies a
tradeoff between sensitivity to the input stream and robustness against
parameter mistuning.
The goal of our study is to analyze the consequences of this tradeoff for
decision making performance. For concreteness, we focus on the wellstudied
random dot motion discrimination task. For stimulus parameters constrained by
experimental data, we find that loss of sensitivity to inputs has surprisingly
little cost for decision performance. This leads robust integrators to
performance gains when feedback becomes mistuned. Moreover, we find that
substantially robust and mistuned integrator models remain consistent with
chronometric and accuracy functions found in experiments. We explain our
findings via sequential analysis of the momentary and integrated signals, and
discuss their implication: robust integrators may be surprisingly wellsuited
to subserve the basic function of evidence integration in many cognitive tasks.

Novel experimental techniques reveal the simultaneous activity of larger and
larger numbers of neurons. As a result there is increasing interest in the
structure of cooperative  or correlated  activity in neural populations,
and in the possible impact of such correlations on the neural code. A
fundamental theoretical challenge is to understand how the architecture of
network connectivity along with the dynamical properties of single cells shape
the magnitude and timescale of correlations. We provide a general approach to
this problem by extending prior techniques based on linear response theory. We
consider networks of general integrateandfire cells with arbitrary
architecture, and provide explicit expressions for the approximate
crosscorrelation between constituent cells. These correlations depend strongly
on the operating point (input mean and variance) of the neurons, even when
connectivity is fixed. Moreover, the approximations admit an expansion in
powers of the matrices that describe the network architecture. This expansion
can be readily interpreted in terms of paths between different cells. We apply
our results to large excitatoryinhibitory networks, and demonstrate first how
precise balance  or lack thereof  between the strengths and timescales of
excitatory and inhibitory synapses is reflected in the overall correlation
structure of the network. We then derive explicit expressions for the average
correlation structure in randomly connected networks. These expressions help to
identify the important factors that shape coordinated neural activity in such
networks.