-
Feature selection is a standard approach to understanding and modeling
high-dimensional classification data, but the corresponding statistical methods
hinge on tuning parameters that are difficult to calibrate. In particular,
existing calibration schemes in the logistic regression framework lack any
finite sample guarantees. In this paper, we introduce a novel calibration
scheme for $\ell_1$-penalized logistic regression. It is based on simple tests
along the tuning parameter path and is equipped with optimal guarantees for
feature selection. It is also amenable to easy and efficient implementations,
and it rivals or outmatches existing methods in simulations and real data
applications.
-
Maximum regularized likelihood estimators (MRLEs) are arguably the most
established class of estimators in high-dimensional statistics. In this paper,
we derive guarantees for MRLEs in Kullback-Leibler divergence, a general
measure of prediction accuracy. We assume only that the densities have a convex
parametrization and that the regularization is definite and positive
homogenous. The results thus apply to a very large variety of models and
estimators, such as tensor regression and graphical models with convex and
non-convex regularized methods. A main conclusion is that MRLEs are broadly
consistent in prediction - regardless of whether restricted eigenvalues or
similar conditions hold.
-
The abundance of high-dimensional data in the modern sciences has generated
tremendous interest in penalized estimators such as the lasso, scaled lasso,
square-root lasso, elastic net, and many others. In this paper, we establish a
general oracle inequality for prediction in high-dimensional linear regression
with such methods. Since the proof relies only on convexity and continuity
arguments, the result holds irrespective of the design matrix and applies to a
wide range of penalized estimators. Overall, the bound demonstrates that
generic estimators can provide consistent prediction with any design matrix.
From a practical point of view, the bound can help to identify the potential of
specific estimators, and they can help to get a sense of the prediction
accuracy in a given application.
-
This paper concerns statistical inference for the components of a
high-dimensional regression parameter despite possible endogeneity of each
regressor. Given a first-stage linear model for the endogenous regressors and a
second-stage linear model for the dependent variable, we develop a novel
adaptation of the parametric one-step update to a generic second-stage
estimator. We provide conditions under which the scaled update is
asymptotically normal. We then introduce a two-stage Lasso procedure and show
that the second-stage Lasso estimator satisfies the aforementioned conditions.
Using these results, we construct asymptotically valid confidence intervals for
the components of the second-stage regression coefficients. We complement our
asymptotic theory with simulation studies, which demonstrate the performance of
our method in finite samples.
-
The TREX is a recently introduced approach to sparse linear regression. In
contrast to most well-known approaches to penalized regression, the TREX can be
formulated without the use of tuning parameters. In this paper, we establish
the first known prediction error bounds for the TREX. Additionally, we
introduce extensions of the TREX to a more general class of penalties, and we
provide a bound on the prediction error in this generalized setting. These
results deepen the understanding of TREX from a theoretical perspective and
provide new insights into penalized regression in general.
-
We introduce a general framework for undirected graphical models. It
generalizes Gaussian graphical models to a wide range of continuous, discrete,
and combinations of different types of data. The models in the framework,
called exponential trace models, are amenable to estimation based on maximum
likelihood. We introduce a sampling-based approximation algorithm for computing
the maximum likelihood estimator, and we apply this pipeline to learn
simultaneous neural activities from spike data.
-
High-dimensional prediction typically comprises two steps: variable selection
and subsequent least-squares refitting on the selected variables. However, the
standard variable selection procedures, such as the lasso, hinge on tuning
parameters that need to be calibrated. Cross-validation, the most popular
calibration scheme, is computationally costly and lacks finite sample
guarantees. In this paper, we introduce an alternative scheme, easy to
implement and both computationally and theoretically efficient.
-
In applications of graphical models, we typically have more information than
just the samples themselves. A prime example is the estimation of brain
connectivity networks based on fMRI data, where in addition to the samples
themselves, the spatial positions of the measurements are readily available.
With particular regard for this application, we are thus interested in ways to
incorporate additional knowledge most effectively into graph estimation. Our
approach to this is to make neighborhood selection receptive to additional
knowledge by strengthening the role of the tuning parameters. We demonstrate
that this concept (i) can improve reproducibility, (ii) is computationally
convenient and efficient, and (iii) carries a lucid Bayesian interpretation. We
specifically show that the approach provides effective estimations of brain
connectivity graphs from fMRI data. However, providing a general scheme for the
inclusion of additional knowledge, our concept is expected to have applications
in a wide range of domains.
-
We introduce a novel scheme for choosing the regularization parameter in
high-dimensional linear regression with Lasso. This scheme, inspired by
Lepski's method for bandwidth selection in non-parametric regression, is
equipped with both optimal finite-sample guarantees and a fast algorithm. In
particular, for any design matrix such that the Lasso has low sup-norm error
under an "oracle choice" of the regularization parameter, we show that our
method matches the oracle performance up to a small constant factor, and show
that it can be implemented by performing simple tests along a single Lasso
path. By applying the Lasso to simulated and real data, we find that our novel
scheme can be faster and more accurate than standard schemes such as
Cross-Validation.
-
Although the Lasso has been extensively studied, the relationship between its
prediction performance and the correlations of the covariates is not fully
understood. In this paper, we give new insights into this relationship in the
context of multiple linear regression. We show, in particular, that the
incorporation of a simple correlation measure into the tuning parameter can
lead to a nearly optimal prediction performance of the Lasso even for highly
correlated covariates. However, we also reveal that for moderately correlated
covariates, the prediction performance of the Lasso can be mediocre
irrespective of the choice of the tuning parameter. We finally show that our
results also lead to near-optimal rates for the least-squares estimator with
total variation penalty.
-
Due to technological advances, large and high-dimensional data have become
the rule rather than the exception. Methods that allow for feature selection
with such data are thus highly sought after, in particular, since standard
methods, such as cross-validated lasso and group-lasso, can be challenging both
computationally and mathematically. In this paper, we propose a novel approach
to feature selection and group feature selection in linear regression. It
consists of simple optimization steps and tests, which makes it computationally
more efficient than standard approaches and suitable even for very large data
sets. Moreover, it satisfies sharp guarantees for estimation and feature
selection in terms of oracle inequalities. We thus expect that our contribution
can help to leverage the increasing volume of data in Biology, Public Health,
Astronomy, Economics, and other fields.
-
The TREX is a recently introduced method for performing sparse
high-dimensional regression. Despite its statistical promise as an alternative
to the lasso, square-root lasso, and scaled lasso, the TREX is computationally
challenging in that it requires solving a non-convex optimization problem. This
paper shows a remarkable result: despite the non-convexity of the TREX problem,
there exists a polynomial-time algorithm that is guaranteed to find the global
minimum. This result adds the TREX to a very short list of non-convex
optimization problems that can be globally optimized (principal components
analysis being a famous example). After deriving and developing this new
approach, we demonstrate that (i) the ability of the preexisting TREX heuristic
to reach the global minimum is strongly dependent on the difficulty of the
underlying statistical problem, (ii) the new polynomial-time algorithm for TREX
permits a novel variable ranking and selection scheme, (iii) this scheme can be
incorporated into a rule that controls the false discovery rate (FDR) of
included features in the model. To achieve this last aim, we provide an
extension of the results of Barber & Candes (2015) to establish that the
knockoff filter framework can be applied to the TREX. This investigation thus
provides both a rare case study of a heuristic for non-convex optimization and
a novel way of exploiting non-convexity for statistical inference.
-
Sparse Filtering is a popular feature learning algorithm for image
classification pipelines. In this paper, we connect the performance of Sparse
Filtering with spectral properties of the corresponding feature matrices. This
connection provides new insights into Sparse Filtering; in particular, it
suggests early stopping of Sparse Filtering. We therefore introduce the Optimal
Roundness Criterion (ORC), a novel stopping criterion for Sparse Filtering. We
show that this stopping criterion is related with pre-processing procedures
such as Statistical Whitening and demonstrate that it can make image
classification with Sparse Filtering considerably faster and more accurate.
-
Lasso is a seminal contribution to high-dimensional statistics, but it hinges
on a tuning parameter that is difficult to calibrate in practice. A partial
remedy for this problem is Square-Root Lasso, because it inherently calibrates
to the noise variance. However, Square-Root Lasso still requires the
calibration of a tuning parameter to all other aspects of the model. In this
study, we introduce TREX, an alternative to Lasso with an inherent calibration
to all aspects of the model. This adaptation to the entire model renders TREX
an estimator that does not require any calibration of tuning parameters. We
show that TREX can outperform cross-validated Lasso in terms of variable
selection and computational efficiency. We also introduce a bootstrapped
version of TREX that can further improve variable selection. We illustrate the
promising performance of TREX both on synthetic data and on a recent
high-dimensional biological data set that considers riboflavin production in B.
subtilis.
-
We introduce Graphical TREX (GTREX), a novel method for graph estimation in
high-dimensional Gaussian graphical models. By conducting neighborhood
selection with TREX, GTREX avoids tuning parameters and is adaptive to the
graph topology. We compare GTREX with standard methods on a new simulation
set-up that is designed to assess accurately the strengths and shortcomings of
different methods. These simulations show that a neighborhood selection scheme
based on Lasso and an optimal (in practice unknown) tuning parameter
outperforms other standard methods over a large spectrum of scenarios.
Moreover, we show that GTREX can rival this scheme and, therefore, can provide
competitive graph estimation without the need for tuning parameter calibration.
-
While effective concentration inequalities for suprema of empirical processes
exist under boundedness or strict tail assumptions, no comparable results have
been available under considerably weaker assumptions. In this paper, we derive
concentration inequalities assuming only low moments for an envelope of the
empirical process. These concentration inequalities are beneficial even when
the envelope is much larger than the single functions under consideration.
-
We introduce a robust and fully adaptive method for pointwise estimation in
heteroscedastic regression. We allow for noise and design distributions that
are unknown and fulfill very weak assumptions only. In particular, we do not
impose moment conditions on the noise distribution. Moreover, we do not require
a positive density for the design distribution. In a first step, we study the
consistency of locally polynomial M-estimators that consist of a contrast and a
kernel. Afterwards, minimax results are established over unidimensional
H\"older spaces for degenerate design. We then choose the contrast and the
kernel that minimize an empirical variance term and demonstrate that the
corresponding M-estimator is adaptive with respect to the noise and design
distributions and adaptive (Huber) minimax for contamination models. In a
second step, we additionally choose a data-driven bandwidth via Lepski's
method. This leads to an M-estimator that is adaptive with respect to the noise
and design distributions and, additionally, adaptive with respect to the
smoothness of an isotropic, multivariate, locally polynomial target function.
These results are also extended to anisotropic, locally constant target
functions. Our data-driven approach provides, in particular, a level of
robustness that adapts to the noise, contamination, and outliers.
-
We introduce and study the Group Square-Root Lasso (GSRL) method for
estimation in high dimensional sparse regression models with group structure.
The new estimator minimizes the square root of the residual sum of squares plus
a penalty term proportional to the sum of the Euclidean norms of groups of the
regression parameter vector. The net advantage of the method over the existing
Group Lasso (GL)-type procedures consists in the form of the proportionality
factor used in the penalty term, which for GSRL is independent of the variance
of the error terms. This is of crucial importance in models with more
parameters than the sample size, when estimating the variance of the noise
becomes as difficult as the original problem. We show that the GSRL estimator
adapts to the unknown sparsity of the regression vector, and has the same
optimal estimation and prediction accuracy as the GL estimators, under the same
minimal conditions on the model. This extends the results recently established
for the Square-Root Lasso, for sparse regression without group structure.
Moreover, as a new type of result for Square-Root Lasso methods, with or
without groups, we study correct pattern recovery, and show that it can be
achieved under conditions similar to those needed by the Lasso or
Group-Lasso-type methods, but with a simplified tuning strategy. We implement
our method via a new algorithm, with proved convergence properties, which,
unlike existing methods, scales well with the dimension of the problem. Our
simulation studies support strongly our theoretical findings.
-
Least-squares refitting is widely used in high dimensional regression to
reduce the prediction bias of l1-penalized estimators (e.g., Lasso and
Square-Root Lasso). We present theoretical and numerical results that provide
new insights into the benefits and pitfalls of least-squares refitting. In
particular, we consider both prediction and estimation, and we pay close
attention to the effects of correlations in the design matrices of linear
regression models, since these correlations - although often neglected - are
crucial in the context of linear regression, especially in high dimensional
contexts. First, we demonstrate that the benefit of least-squares refitting
strongly depends on the setting and task under consideration: least-squares
refitting can be beneficial even for settings with highly correlated design
matrices but is not advisable for all settings, and least-squares refitting can
be beneficial for estimation but performs better for prediction. Finally, we
introduce a criterion that indicates whether least-squares refitting is
advisable for a specific setting and task under consideration, and we conduct a
thorough simulation study involving the Lasso to show the usefulness of this
criterion.
-
We introduce two new concepts designed for the study of empirical processes.
First, we introduce a new Orlicz norm which we call the Bernstein-Orlicz norm.
This new norm interpolates sub-Gaussian and sub-exponential tail behavior. In
particular, we show how this norm can be used to simplify the derivation of
deviation inequalities for suprema of collections of random variables.
Secondly, we introduce chaining and generic chaining along a tree. These
simplify the well-known concepts of chaining and generic chaining. The supremum
of the empirical process is then studied as a special case. We show that
chaining along a tree can be done using entropy with bracketing. Finally, we
establish a deviation inequality for the empirical process for the unbounded
case.
-
We study high-dimensional linear models and the $\ell_1$-penalized least
squares estimator, also known as the Lasso estimator. In literature, oracle
inequalities have been derived under restricted eigenvalue or compatibility
conditions. In this paper, we complement this with entropy conditions which
allow one to improve the dual norm bound, and demonstrate how this leads to new
oracle inequalities. The new oracle inequalities show that a smaller choice for
the tuning parameter and a trade-off between $\ell_1$-norms and small
compatibility constants are possible. This implies, in particular for
correlated design, improved bounds for the prediction error of the Lasso
estimator as compared to the methods based on restricted eigenvalue or
compatibility conditions only.