• In physics, biology and engineering, network systems abound. How does the connectivity of a network system combine with the behavior of its individual components to determine its collective function? We approach this question for networks with linear time-invariant dynamics by relating internal network feedbacks to the statistical prevalence of connectivity motifs, a set of surprisingly simple and local statistics of connectivity. This results in a reduced order model of the network input-output dynamics in terms of motifs structures. As an example, the new formulation dramatically simplifies the classic Erdos-Renyi graph, reducing the overall network behavior to one proportional feedback wrapped around the dynamics of a single node. For general networks, higher-order motifs systematically provide further layers and types of feedback to regulate the network response. Thus, the local connectivity shapes temporal and spectral processing by the network as a whole, and we show how this enables robust, yet tunable, functionality such as extending the time constant with which networks remember past signals. The theory also extends to networks composed from heterogeneous nodes with distinct dynamics and connectivity, and patterned input to (and readout from) subsets of nodes. These statistical descriptions provide a powerful theoretical framework to understand the functionality of real-world network systems, as we illustrate with examples including the mouse brain connectome.
  • This paper presents a randomized algorithm for computing the near-optimal low-rank dynamic mode decomposition (DMD). Randomized algorithms are emerging techniques to compute low-rank matrix approximations at a fraction of the cost of deterministic algorithms, easing the computational challenges arising in the area of `big data'. The idea is to derive a small matrix from the high-dimensional data, which is then used to efficiently compute the dynamic modes and eigenvalues. The algorithm is presented in a modular probabilistic framework, and the approximation quality can be controlled via oversampling and power iterations. The effectiveness of the resulting randomized DMD (rDMD) algorithm is demonstrated on several benchmark examples of increasing complexity, providing an accurate and efficient approach to extract spatiotemporal coherent structures from big data in a framework that scales with the intrinsic rank of the data, rather than the ambient measurement dimension.
  • The problem of optimally placing sensors under a cost constraint arises naturally in the design of industrial and commercial products, as well as in scientific experiments. We consider a relaxation of the full optimization formulation of this problem and then extend a well-established QR-based greedy algorithm for the optimal sensor placement problem without cost constraints. We demonstrate the effectiveness of this algorithm on data sets related to facial recognition, climate science, and fluid mechanics. This algorithm is scalable and often identifies sparse sensors with near optimal reconstruction performance, while dramatically reducing the overall cost of the sensors. We find that the cost-error landscape varies by application, with intuitive connections to the underlying physics. Additionally, we include experiments for various pre-processing techniques and find that a popular technique based on the singular value decomposition is often sub-optimal.
  • Nonnegative matrix factorization (NMF) is a powerful tool for data mining. However, the emergence of `big data' has severely challenged our ability to compute this fundamental decomposition using deterministic algorithms. This paper presents a randomized hierarchical alternating least squares (HALS) algorithm to compute the NMF. By deriving a smaller matrix from the nonnegative input data, a more efficient nonnegative decomposition can be computed. Our algorithm scales to big data applications while attaining a near-optimal factorization. The proposed algorithm is evaluated using synthetic and real world data and shows substantial speedups compared to deterministic HALS.
  • Identifying coordinate transformations that make strongly nonlinear dynamics approximately linear is a central challenge in modern dynamical systems. These transformations have the potential to enable prediction, estimation, and control of nonlinear systems using standard linear theory. The Koopman operator has emerged as a leading data-driven embedding, as eigenfunctions of this operator provide intrinsic coordinates that globally linearize the dynamics. However, identifying and representing these eigenfunctions has proven to be mathematically and computationally challenging. This work leverages the power of deep learning to discover representations of Koopman eigenfunctions from trajectory data of dynamical systems. Our network is parsimonious and interpretable by construction, embedding the dynamics on a low-dimensional manifold that is of the intrinsic rank of the dynamics and parameterized by the Koopman eigenfunctions. In particular, we identify nonlinear coordinates on which the dynamics are globally linear using a modified auto-encoder. We also generalize Koopman representations to include a ubiquitous class of systems that exhibit continuous spectra, ranging from the simple pendulum to nonlinear optics and broadband turbulence. Our framework parametrizes the continuous frequency using an auxiliary network, enabling a compact and efficient embedding at the intrinsic rank, while connecting our models to half a century of asymptotics. In this way, we benefit from the power and generality of deep learning, while retaining the physical interpretability of Koopman embeddings.
  • Data-driven transformations that reformulate nonlinear systems in a linear framework have the potential to enable the prediction, estimation, and control of strongly nonlinear dynamics using linear systems theory. The Koopman operator has emerged as a principled linear embedding of nonlinear dynamics, and its eigenfunctions establish intrinsic coordinates along which the dynamics behave linearly. In this work, we demonstrate a data-driven control architecture, termed Koopman Reduced Order Nonlinear Identification and Control (KRONIC), that utilizes Koopman eigenfunctions to manipulate nonlinear systems using linear systems theory. We approximate these eigenfunctions with data-driven regression and power series expansions, based on the partial differential equation governing the infinitesimal generator of the Koopman operator. Although previous regression-based methods may identify spurious dynamics, we show that lightly damped eigenfunctions may be faithfully extracted using sparse regression. These lightly damped eigenfunctions are particularly relevant for control, as they correspond to nearly conserved quantities that are associated with persistent dynamics, such as the Hamiltonian. We derive the form of control in these intrinsic eigenfunction coordinates and design nonlinear controllers using standard linear control theory. KRONIC is then demonstrated on a number of relevant examples, including 1) a nonlinear system with a known linear embedding, 2) a variety of Hamiltonian systems, and 3) a high-dimensional double-gyre model for ocean mixing.
  • We consider a new class of periodic solutions to the Lugiato-Lefever equations (LLE) that govern the electromagnetic field in a microresonator cavity. Specifically, we rigorously characterize the stability and dynamics of the Jacobi elliptic function solutions of LLE and show that the dn solution is stabilized by the pumping of the microresonator. In analogy with soliton perturbation theory, we also derive a microcomb perturbation theory that allows one to consider the effects of physically realizable perturbations on the comb line stability, including effects of Raman scattering and stimulated emission. Our results are verified through full numerical simulations of the LLE cavity dynamics. The perturbation theory gives a simple analytic platform for potentially engineering new resonator designs.
  • Sparse principal component analysis (SPCA) has emerged as a powerful technique for modern data analysis. We discuss a robust and scalable algorithm for computing sparse principal component analysis. Specifically, we model SPCA as a matrix factorization problem with orthogonality constraints, and develop specialized optimization algorithms that partially minimize a subset of the variables (variable projection). The framework incorporates a wide variety of sparsity-inducing regularizers for SPCA. We also extend the variable projection approach to robust SPCA, for any robust loss that can be expressed as the Moreau envelope of a simple function, with the canonical example of the Huber loss. Finally, randomized methods for linear algebra are used to extend the approach to the large-scale (big data) setting. The proposed algorithms are demonstrated using both synthetic and real world data.
  • Matrix decompositions are fundamental tools in the area of applied mathematics, statistical computing, and machine learning. In particular, low-rank matrix decompositions are vital, and widely used for data analysis, dimensionality reduction, and data compression. Massive datasets, however, pose a computational challenge for traditional algorithms, placing significant constraints on both memory and processing power. Recently, the powerful concept of randomness has been introduced as a strategy to ease the computational load. The essential idea of probabilistic algorithms is to employ some amount of randomness in order to derive a smaller matrix from a high-dimensional data matrix. The smaller matrix is then used to compute the desired low-rank approximation. Such algorithms are shown to be computationally efficient for approximating matrices with low-rank structure. We present the R package rsvd, and provide a tutorial introduction to randomized matrix decompositions. Specifically, randomized routines for the singular value decomposition, (robust) principal component analysis, interpolative decomposition, and CUR decomposition are discussed. Several examples demonstrate the routines, and show the computational advantage over other methods implemented in R.
  • Big data has become a critically enabling component of emerging mathematical methods aimed at the automated discovery of dynamical systems, where first principles modeling may be intractable. However, in many engineering systems, abrupt changes must be rapidly characterized based on limited, incomplete, and noisy data. Many leading automated learning techniques rely on unrealistically large data sets and it is unclear how to leverage prior knowledge effectively to re-identify a model after an abrupt change. In this work, we propose a conceptual framework to recover parsimonious models of a system in response to abrupt changes in the low-data limit. First, the abrupt change is detected by comparing the estimated Lyapunov time of the data with the model prediction. Next, we apply the sparse identification of nonlinear dynamics (SINDy) regression to update a previously identified model with the fewest changes, either by addition, deletion, or modification of existing model terms. We demonstrate this sparse model recovery on several examples for abrupt system change detection in periodic and chaotic dynamical systems. Our examples show that sparse updates to a previously identified model perform better with less data, have lower runtime complexity, and are less sensitive to noise than identifying an entirely new model. The proposed abrupt-SINDy architecture provides a new paradigm for the rapid and efficient recovery of a system model after abrupt changes.
  • We develop a biophysically realistic model of the nematode C. elegans that includes: (i) its muscle structure and activation, (ii) key connectomic activation circuitry, and (iii) a weighted and time-dynamic proprioception. In combination, we show that these model components can reproduce the complex waveforms exhibited in C. elegans locomotive behaviors, chiefly omega turns. This is achieved via weighted, time-dependent suppression of the proprioceptive signal. Though speculative, such dynamics are biologically plausible due to the presence of neuromodulators which have recently been experimentally implicated in the escape response, which includes an omega turn. This is the first integrated neuromechanical model to reveal a mechanism capable of generating the complex waveforms observed in the behavior of C. elegans, thus contributing to a mathematical framework for understanding how control decisions can be executed at the connectome level in order to produce the full repertoire of observed behaviors.
  • Diffusion maps are an emerging data-driven technique for non-linear dimensionality reduction, which are especially useful for the analysis of coherent structures and nonlinear embeddings of dynamical systems. However, the computational complexity of the diffusion maps algorithm scales with the number of observations. Thus, long time-series data presents a significant challenge for fast and efficient embedding. We propose integrating the Nystr\"om method with diffusion maps in order to ease the computational demand. We achieve a speedup of roughly two to four times when approximating the dominant diffusion map components.
  • We seek to (i) characterize the learning architectures exploited in biological neural networks for training on very few samples, and (ii) port these algorithmic structures to a machine learning context. The Moth Olfactory Network is among the simplest biological neural systems that can learn, and its architecture includes key structural elements widespread in biological neural nets, such as cascaded networks, competitive inhibition, high intrinsic noise, sparsity, reward mechanisms, and Hebbian plasticity. The interactions of these structural elements play a critical enabling role in rapid learning. We assign a computational model of the Moth Olfactory Network the task of learning to read the MNIST digits. This model, MothNet, is closely aligned with the moth's known biophysics and with in vivo electrode data, including data collected from moths learning new odors. We show that MothNet successfully learns to read given very few training samples (1 to 20 samples per class). In this few-samples regime, it substantially outperforms standard machine learning methods such as nearest-neighbors, support-vector machines, and convolutional neural networks (CNNs). The MothNet architecture illustrates how our proposed algorithmic structures, derived from biological brains, can be used to build alternative deep neural nets (DNNs) that may potentially avoid some of DNNs current learning rate limitations. This novel, bio-inspired neural network architecture offers a valuable complementary approach to DNN design.
  • The insect olfactory system, which includes the antennal lobe (AL), mushroom body (MB), and ancillary structures, is a relatively simple neural system capable of learning. Its structural features, which are widespread in biological neural systems, process olfactory stimuli through a cascade of networks where large dimension shifts occur from stage to stage and where sparsity and randomness play a critical role in coding. Learning is partly enabled by a neuromodulatory reward mechanism of octopamine stimulation of the AL, whose increased activity induces rewiring of the MB through Hebbian plasticity. Enforced sparsity in the MB focuses Hebbian growth on neurons that are the most important for the representation of the learned odor. Based upon current biophysical knowledge, we have constructed an end-to-end computational model of the Manduca sexta moth olfactory system which includes the interaction of the AL and MB under octopamine stimulation. Our model is able to robustly learn new odors, and our simulations of integrate-and-fire neurons match the statistical features of in-vivo firing rate data. From a biological perspective, the model provides a valuable tool for examining the role of neuromodulators, like octopamine, in learning, and gives insight into critical interactions between sparsity, Hebbian growth, and stimulation during learning. Our simulations also inform predictions about structural details of the olfactory system that are not currently well-characterized. From a machine learning perspective, the model yields bio-inspired mechanisms that are potentially useful in constructing neural nets for rapid learning from very few samples. These mechanisms include high-noise layers, sparse layers as noise filters, and a biologically-plausible optimization method to train the network based on octopamine stimulation, sparse layers, and Hebbian growth.
  • Kerr soliton frequency comb generation in monolithic microresonators recently attracted great interests as it enables chip-scale few-cycle pulse generation at microwave rates with smooth octave-spanning spectra for self-referencing. Such versatile platform finds significant applications in dual-comb spectroscopy, low-noise optical frequency synthesis, coherent communication systems, etc. However, it still remains challenging to straightforwardly and deterministically generate and sustain the single-soliton state in microresonators. In this paper, we propose and theoretically demonstrate the excitation of single-soliton Kerr frequency comb by seeding the continuous-wave driven nonlinear microcavity with a pulsed trigger. Unlike the mostly adopted frequency tuning scheme reported so far, we show that an energetic single shot pulse can trigger the single-soliton state deterministically without experiencing any unstable or chaotic states. Neither the pump frequency nor the cavity resonance is required to be tuned. The generated mode-locked single-soliton Kerr comb is robust and insensitive to perturbations. Even when the thermal effect induced by the absorption of the intracavity light is taken into account, the proposed single pulse trigger approach remains valid without requiring any thermal compensation means.
  • Self-tuning optical systems are of growing importance in technological applications such as mode-locked fiber lasers. Such self-tuning paradigms require {\em intelligent} algorithms capable of inferring approximate models of the underlying physics and discovering appropriate control laws in order to maintain robust performance for a given objective. In this work, we demonstrate the first integration of a {\em deep learning} (DL) architecture with {\em model predictive control} (MPC) in order to self-tune a mode-locked fiber laser. Not only can our DL-MPC algorithmic architecture approximate the unknown fiber birefringence, it also builds a dynamical model of the laser and appropriate control law for maintaining robust, high-energy pulses despite a stochastically drifting birefringence. We demonstrate the effectiveness of this method on a fiber laser which is mode-locked by nonlinear polarization rotation. The method advocated can be broadly applied to a variety of optical systems that require robust controllers.
  • Optimal sensor placement is a central challenge in the design, prediction, estimation, and control of high-dimensional systems. High-dimensional states can often leverage a latent low-dimensional representation, and this inherent compressibility enables sparse sensing. This article explores optimized sensor placement for signal reconstruction based on a tailored library of features extracted from training data. Sparse point sensors are discovered using the singular value decomposition and QR pivoting, which are two ubiquitous matrix computations that underpin modern linear dimensionality reduction. Sparse sensing in a tailored basis is contrasted with compressed sensing, a universal signal recovery method in which an unknown signal is reconstructed via a sparse representation in a universal basis. Although compressed sensing can recover a wider class of signals, we demonstrate the benefits of exploiting known patterns in data with optimized sensing. In particular, drastic reductions in the required number of sensors and improved reconstruction are observed in examples ranging from facial images to fluid vorticity fields. Principled sensor placement may be critically enabling when sensors are costly and provides faster state estimation for low-latency, high-bandwidth control. MATLAB code is provided for all examples.
  • High frequency fluctuation in the optical signal generated in Fourier-Domain Mode Locked fiber laser (FDML-FL), which is the major problem and degrades the laser performance, is not yet fully analyzed or studied. The basic theory which is causing this high frequency fluctuation is required to clearly understand its dynamics and to control it for various applications. In this letter, by analyzing the signal and system dynamics of FDML-FL, we theoretically demonstrate that the high frequency fluctuation is induced by the intrinsic instability of frequency offset of the signal in cavity with nonlinear gain and spectral filter. Unlike the instabilities observed in other laser cavities this instability is very unique to FDML-FL as the central frequency of the optical signal continuously shifts away from the center frequency of the filter due to the effects like dispersion and/or nonlinearity. This instability is none other than the Eckhaus instability reported and well studied in fluid dynamics governed by real Ginzburg-Landau equation.
  • We present a traveling wave model for a semiconductor diode laser based on quantum wells. The gain model is carefully derived from first principles and implemented with as few phenomeno- logical constants as possible. The transverse energies of the quantum well confined electrons are discretized to automatically capture the effects of spectral and spatial hole burning, gain asym- metry, and the linewidth enhancement factor. We apply this model to semiconductor optical amplifiers and single-section phase-locked lasers. We are able to reproduce the experimental re- sults. The calculated frequency modulated comb shows potential to be a compact, chip-scale comb source without additional external components.
  • The dynamic mode decomposition (DMD) has become a leading tool for data-driven modeling of dynamical systems, providing a regression framework for fitting linear dynamical models to time-series measurement data. We present a simple algorithm for computing an optimized version of the DMD for data which may be collected at unevenly spaced sample times. By making use of the variable projection method for nonlinear least squares problems, the algorithm is capable of solving the underlying nonlinear optimization problem efficiently. We explore the performance of the algorithm with some numerical examples for synthetic and real data from dynamical systems and find that the resulting decomposition displays less bias in the presence of noise than standard DMD algorithms. Because of the flexibility of the algorithm, we also present some interesting new options for DMD-based analysis.
  • The CANDECOMP/PARAFAC (CP) tensor decomposition is a popular dimensionality-reduction method for multiway data. Dimensionality reduction is often sought since many high-dimensional tensors have low intrinsic rank relative to the dimension of the ambient measurement space. However, the emergence of `big data' poses significant computational challenges for computing this fundamental tensor decomposition. Leveraging modern randomized algorithms, we demonstrate that the coherent structure can be learned from a smaller representation of the tensor in a fraction of the time. Moreover, the high-dimensional signal can be faithfully approximated from the compressed measurements. Thus, this simple but powerful algorithm enables one to compute the approximate CP decomposition even for massive tensors. The approximation error can thereby be controlled via oversampling and the computation of power iterations. In addition to theoretical results, several empirical results demonstrate the performance of the proposed algorithm.
  • We demonstrate the application of the Dynamic Mode Decomposition (DMD) for the diagnostic analysis of the nonlinear dynamics of a magnetized plasma in resistive magnetohydrodynamics. The DMD method is an ideal spatio-temporal matrix decomposition that correlates spatial features of computational or experimental data while simultaneously associating the spatial activity with periodic temporal behavior. DMD can produce low-rank, reduced order surrogate models that can be used to reconstruct the state of the system and produce high-fidelity future state predictions. This allows for a reduction in the computational cost, and, at the same time, accurate approximations of the problem, even if the data are sparsely sampled. We demonstrate the use of the method on both numerical and experimental data, showing that it is a successful mathematical architecture for characterizing the HIT-IS magnetohydrodynamics. Importantly, the DMD decomposition produces interpretable, dominant mode structures, including the spheromak mode and a pair of injector-driven modes. In combination, the 3-mode DMD model produces excellent dynamic reconstructions that can be used for future state prediction and/or control strategies.
  • This paper addresses the problem of identifying different flow environments from sparse data collected by wing strain sensors. Insects regularly perform this feat using a sparse ensemble of noisy strain sensors on their wing. First, we obtain strain data from numerical simulation of a Manduca sexta hawkmoth wing undergoing different flow environments. Our data-driven method learns low-dimensional strain features originating from different aerodynamic environments using proper orthogonal decomposition (POD) modes in the frequency domain, and leverages sparse approximation to classify a set of strain frequency signatures using a dictionary of POD modes. This bio-inspired machine learning architecture for dictionary learning and sparse classification permits fewer costly physical strain sensors while being simultaneously robust to sensor noise. A measurement selection algorithm identifies frequencies that best discriminate the different aerodynamic environments in low-rank POD feature space. In this manner, sparse and noisy wing strain data can be exploited to robustly identify different aerodynamic environments encountered in flight, providing insight into the stereotyped placement of neurons that act as strain sensors on a Manduca sexta hawkmoth wing.
  • We develop an algorithm for model selection which allows for the consideration of a combinatorially large number of candidate models governing a dynamical system. The innovation circumvents a disadvantage of standard model selection which typically limits the number candidate models considered due to the intractability of computing information criteria. Using a recently developed sparse identification of nonlinear dynamics algorithm, the sub-selection of candidate models near the Pareto frontier allows for a tractable computation of AIC (Akaike information criteria) or BIC (Bayes information criteria) scores for the remaining candidate models. The information criteria hierarchically ranks the most informative models, enabling the automatic and principled selection of the model with the strongest support in relation to the time series data. Specifically, we show that AIC scores place each candidate model in the {\em strong support}, {\em weak support} or {\em no support} category. The method correctly identifies several canonical dynamical systems, including an SEIR (susceptible-exposed-infectious-recovered) disease model and the Lorenz equations, giving the correct dynamical system as the only candidate model with strong support.
  • Local climate conditions play a major role in the development of the mosquito population responsible for transmitting Dengue Fever. Since the {\em Aedes Aegypti} mosquito is also a primary vector for the recent Zika and Chikungunya epidemics across the Americas, a detailed monitoring of periods with favorable climate conditions for mosquito profusion may improve the timing of vector-control efforts and other urgent public health strategies. We apply dimensionality reduction techniques and machine-learning algorithms to climate time series data and analyze their connection to the occurrence of Dengue outbreaks for seven major cities in Brazil. Specifically, we have identified two key variables and a period during the annual cycle that are highly predictive of epidemic outbreaks. The key variables are the frequency of precipitation and temperature during an approximately two month window of the winter season preceding the outbreak. Thus simple climate signatures may be influencing Dengue outbreaks even months before their occurrence. Some of the more challenging datasets required usage of compressive-sensing procedures to estimate missing entries for temperature and precipitation records. Our results indicate that each Brazilian capital considered has a unique frequency of precipitation and temperature signature in the winter preceding a Dengue outbreak. Such climate contributions on vector populations are key factors in dengue dynamics which could lead to more accurate prediction models and early warning systems. Finally, we show that critical temperature and precipitation signatures may vary significantly from city to city, suggesting that the interplay between climate variables and dengue outbreaks is more complex than generally appreciated.