
Feature selection is a standard approach to understanding and modeling
highdimensional classification data, but the corresponding statistical methods
hinge on tuning parameters that are difficult to calibrate. In particular,
existing calibration schemes in the logistic regression framework lack any
finite sample guarantees. In this paper, we introduce a novel calibration
scheme for $\ell_1$penalized logistic regression. It is based on simple tests
along the tuning parameter path and is equipped with optimal guarantees for
feature selection. It is also amenable to easy and efficient implementations,
and it rivals or outmatches existing methods in simulations and real data
applications.

Maximum regularized likelihood estimators (MRLEs) are arguably the most
established class of estimators in highdimensional statistics. In this paper,
we derive guarantees for MRLEs in KullbackLeibler divergence, a general
measure of prediction accuracy. We assume only that the densities have a convex
parametrization and that the regularization is definite and positive
homogenous. The results thus apply to a very large variety of models and
estimators, such as tensor regression and graphical models with convex and
nonconvex regularized methods. A main conclusion is that MRLEs are broadly
consistent in prediction  regardless of whether restricted eigenvalues or
similar conditions hold.

The abundance of highdimensional data in the modern sciences has generated
tremendous interest in penalized estimators such as the lasso, scaled lasso,
squareroot lasso, elastic net, and many others. In this paper, we establish a
general oracle inequality for prediction in highdimensional linear regression
with such methods. Since the proof relies only on convexity and continuity
arguments, the result holds irrespective of the design matrix and applies to a
wide range of penalized estimators. Overall, the bound demonstrates that
generic estimators can provide consistent prediction with any design matrix.
From a practical point of view, the bound can help to identify the potential of
specific estimators, and they can help to get a sense of the prediction
accuracy in a given application.

This paper concerns statistical inference for the components of a
highdimensional regression parameter despite possible endogeneity of each
regressor. Given a firststage linear model for the endogenous regressors and a
secondstage linear model for the response variable, we develop a novel
adaptation of the parametric onestep update to a generic secondstage
estimator. We provide highlevel conditions under which the scaled update is
asymptotically normal. We introduce a twostage Lasso procedure and show that,
under a subGaussian noise regime, the secondstage Lasso estimator satisfies
the aforementioned conditions. Using these results, we construct asymptotically
valid confidence intervals for the components of the secondstage regression
vector. We complement our asymptotic theory with empirical studies, which
demonstrate the relevance of our method in finite samples.

The TREX is a recently introduced approach to sparse linear regression. In
contrast to most wellknown approaches to penalized regression, the TREX can be
formulated without the use of tuning parameters. In this paper, we establish
the first known prediction error bounds for the TREX. Additionally, we
introduce extensions of the TREX to a more general class of penalties, and we
provide a bound on the prediction error in this generalized setting. These
results deepen the understanding of TREX from a theoretical perspective and
provide new insights into penalized regression in general.

We introduce a general framework for undirected graphical models. It
generalizes Gaussian graphical models to a wide range of continuous, discrete,
and combinations of different types of data. The models in the framework,
called exponential trace models, are amenable to estimation based on maximum
likelihood. We introduce a samplingbased approximation algorithm for computing
the maximum likelihood estimator, and we apply this pipeline to learn
simultaneous neural activities from spike data.

Highdimensional prediction typically comprises two steps: variable selection
and subsequent leastsquares refitting on the selected variables. However, the
standard variable selection procedures, such as the lasso, hinge on tuning
parameters that need to be calibrated. Crossvalidation, the most popular
calibration scheme, is computationally costly and lacks finite sample
guarantees. In this paper, we introduce an alternative scheme, easy to
implement and both computationally and theoretically efficient.

In applications of graphical models, we typically have more information than
just the samples themselves. A prime example is the estimation of brain
connectivity networks based on fMRI data, where in addition to the samples
themselves, the spatial positions of the measurements are readily available.
With particular regard for this application, we are thus interested in ways to
incorporate additional knowledge most effectively into graph estimation. Our
approach to this is to make neighborhood selection receptive to additional
knowledge by strengthening the role of the tuning parameters. We demonstrate
that this concept (i) can improve reproducibility, (ii) is computationally
convenient and efficient, and (iii) carries a lucid Bayesian interpretation. We
specifically show that the approach provides effective estimations of brain
connectivity graphs from fMRI data. However, providing a general scheme for the
inclusion of additional knowledge, our concept is expected to have applications
in a wide range of domains.

We introduce a novel scheme for choosing the regularization parameter in
highdimensional linear regression with Lasso. This scheme, inspired by
Lepski's method for bandwidth selection in nonparametric regression, is
equipped with both optimal finitesample guarantees and a fast algorithm. In
particular, for any design matrix such that the Lasso has low supnorm error
under an "oracle choice" of the regularization parameter, we show that our
method matches the oracle performance up to a small constant factor, and show
that it can be implemented by performing simple tests along a single Lasso
path. By applying the Lasso to simulated and real data, we find that our novel
scheme can be faster and more accurate than standard schemes such as
CrossValidation.

Although the Lasso has been extensively studied, the relationship between its
prediction performance and the correlations of the covariates is not fully
understood. In this paper, we give new insights into this relationship in the
context of multiple linear regression. We show, in particular, that the
incorporation of a simple correlation measure into the tuning parameter can
lead to a nearly optimal prediction performance of the Lasso even for highly
correlated covariates. However, we also reveal that for moderately correlated
covariates, the prediction performance of the Lasso can be mediocre
irrespective of the choice of the tuning parameter. We finally show that our
results also lead to nearoptimal rates for the leastsquares estimator with
total variation penalty.

Driven by the advances in technology, large and highdimensional data have
become the rule rather than the exception. Approaches that allow for feature
selection with such data are thus highly sought after, in particular, since
standard methods, like crossvalidated Lasso, can be computationally
intractable and, in any case, lack theoretical guarantees. In this paper, we
propose a novel approach to feature selection in regression. Consisting of
simple optimization steps and tests, it is computationally more efficient than
existing methods and, therefore, suited even for very large data sets.
Moreover, in contrast to standard methods, it is equipped with sharp
statistical guarantees. We thus expect that our algorithm can help to leverage
the increasing volume of data in Biology, Public Health, Astronomy, Economics,
and other fields.

The TREX is a recently introduced method for performing sparse
highdimensional regression. Despite its statistical promise as an alternative
to the lasso, squareroot lasso, and scaled lasso, the TREX is computationally
challenging in that it requires solving a nonconvex optimization problem. This
paper shows a remarkable result: despite the nonconvexity of the TREX problem,
there exists a polynomialtime algorithm that is guaranteed to find the global
minimum. This result adds the TREX to a very short list of nonconvex
optimization problems that can be globally optimized (principal components
analysis being a famous example). After deriving and developing this new
approach, we demonstrate that (i) the ability of the preexisting TREX heuristic
to reach the global minimum is strongly dependent on the difficulty of the
underlying statistical problem, (ii) the new polynomialtime algorithm for TREX
permits a novel variable ranking and selection scheme, (iii) this scheme can be
incorporated into a rule that controls the false discovery rate (FDR) of
included features in the model. To achieve this last aim, we provide an
extension of the results of Barber & Candes (2015) to establish that the
knockoff filter framework can be applied to the TREX. This investigation thus
provides both a rare case study of a heuristic for nonconvex optimization and
a novel way of exploiting nonconvexity for statistical inference.

Sparse Filtering is a popular feature learning algorithm for image
classification pipelines. In this paper, we connect the performance of Sparse
Filtering with spectral properties of the corresponding feature matrices. This
connection provides new insights into Sparse Filtering; in particular, it
suggests early stopping of Sparse Filtering. We therefore introduce the Optimal
Roundness Criterion (ORC), a novel stopping criterion for Sparse Filtering. We
show that this stopping criterion is related with preprocessing procedures
such as Statistical Whitening and demonstrate that it can make image
classification with Sparse Filtering considerably faster and more accurate.

Lasso is a seminal contribution to highdimensional statistics, but it hinges
on a tuning parameter that is difficult to calibrate in practice. A partial
remedy for this problem is SquareRoot Lasso, because it inherently calibrates
to the noise variance. However, SquareRoot Lasso still requires the
calibration of a tuning parameter to all other aspects of the model. In this
study, we introduce TREX, an alternative to Lasso with an inherent calibration
to all aspects of the model. This adaptation to the entire model renders TREX
an estimator that does not require any calibration of tuning parameters. We
show that TREX can outperform crossvalidated Lasso in terms of variable
selection and computational efficiency. We also introduce a bootstrapped
version of TREX that can further improve variable selection. We illustrate the
promising performance of TREX both on synthetic data and on a recent
highdimensional biological data set that considers riboflavin production in B.
subtilis.

We introduce Graphical TREX (GTREX), a novel method for graph estimation in
highdimensional Gaussian graphical models. By conducting neighborhood
selection with TREX, GTREX avoids tuning parameters and is adaptive to the
graph topology. We compare GTREX with standard methods on a new simulation
setup that is designed to assess accurately the strengths and shortcomings of
different methods. These simulations show that a neighborhood selection scheme
based on Lasso and an optimal (in practice unknown) tuning parameter
outperforms other standard methods over a large spectrum of scenarios.
Moreover, we show that GTREX can rival this scheme and, therefore, can provide
competitive graph estimation without the need for tuning parameter calibration.

While effective concentration inequalities for suprema of empirical processes
exist under boundedness or strict tail assumptions, no comparable results have
been available under considerably weaker assumptions. In this paper, we derive
concentration inequalities assuming only low moments for an envelope of the
empirical process. These concentration inequalities are beneficial even when
the envelope is much larger than the single functions under consideration.

We introduce a robust and fully adaptive method for pointwise estimation in
heteroscedastic regression. We allow for noise and design distributions that
are unknown and fulfill very weak assumptions only. In particular, we do not
impose moment conditions on the noise distribution. Moreover, we do not require
a positive density for the design distribution. In a first step, we study the
consistency of locally polynomial Mestimators that consist of a contrast and a
kernel. Afterwards, minimax results are established over unidimensional
H\"older spaces for degenerate design. We then choose the contrast and the
kernel that minimize an empirical variance term and demonstrate that the
corresponding Mestimator is adaptive with respect to the noise and design
distributions and adaptive (Huber) minimax for contamination models. In a
second step, we additionally choose a datadriven bandwidth via Lepski's
method. This leads to an Mestimator that is adaptive with respect to the noise
and design distributions and, additionally, adaptive with respect to the
smoothness of an isotropic, multivariate, locally polynomial target function.
These results are also extended to anisotropic, locally constant target
functions. Our datadriven approach provides, in particular, a level of
robustness that adapts to the noise, contamination, and outliers.

We introduce and study the Group SquareRoot Lasso (GSRL) method for
estimation in high dimensional sparse regression models with group structure.
The new estimator minimizes the square root of the residual sum of squares plus
a penalty term proportional to the sum of the Euclidean norms of groups of the
regression parameter vector. The net advantage of the method over the existing
Group Lasso (GL)type procedures consists in the form of the proportionality
factor used in the penalty term, which for GSRL is independent of the variance
of the error terms. This is of crucial importance in models with more
parameters than the sample size, when estimating the variance of the noise
becomes as difficult as the original problem. We show that the GSRL estimator
adapts to the unknown sparsity of the regression vector, and has the same
optimal estimation and prediction accuracy as the GL estimators, under the same
minimal conditions on the model. This extends the results recently established
for the SquareRoot Lasso, for sparse regression without group structure.
Moreover, as a new type of result for SquareRoot Lasso methods, with or
without groups, we study correct pattern recovery, and show that it can be
achieved under conditions similar to those needed by the Lasso or
GroupLassotype methods, but with a simplified tuning strategy. We implement
our method via a new algorithm, with proved convergence properties, which,
unlike existing methods, scales well with the dimension of the problem. Our
simulation studies support strongly our theoretical findings.

Leastsquares refitting is widely used in high dimensional regression to
reduce the prediction bias of l1penalized estimators (e.g., Lasso and
SquareRoot Lasso). We present theoretical and numerical results that provide
new insights into the benefits and pitfalls of leastsquares refitting. In
particular, we consider both prediction and estimation, and we pay close
attention to the effects of correlations in the design matrices of linear
regression models, since these correlations  although often neglected  are
crucial in the context of linear regression, especially in high dimensional
contexts. First, we demonstrate that the benefit of leastsquares refitting
strongly depends on the setting and task under consideration: leastsquares
refitting can be beneficial even for settings with highly correlated design
matrices but is not advisable for all settings, and leastsquares refitting can
be beneficial for estimation but performs better for prediction. Finally, we
introduce a criterion that indicates whether leastsquares refitting is
advisable for a specific setting and task under consideration, and we conduct a
thorough simulation study involving the Lasso to show the usefulness of this
criterion.

We introduce two new concepts designed for the study of empirical processes.
First, we introduce a new Orlicz norm which we call the BernsteinOrlicz norm.
This new norm interpolates subGaussian and subexponential tail behavior. In
particular, we show how this norm can be used to simplify the derivation of
deviation inequalities for suprema of collections of random variables.
Secondly, we introduce chaining and generic chaining along a tree. These
simplify the wellknown concepts of chaining and generic chaining. The supremum
of the empirical process is then studied as a special case. We show that
chaining along a tree can be done using entropy with bracketing. Finally, we
establish a deviation inequality for the empirical process for the unbounded
case.

We study highdimensional linear models and the $\ell_1$penalized least
squares estimator, also known as the Lasso estimator. In literature, oracle
inequalities have been derived under restricted eigenvalue or compatibility
conditions. In this paper, we complement this with entropy conditions which
allow one to improve the dual norm bound, and demonstrate how this leads to new
oracle inequalities. The new oracle inequalities show that a smaller choice for
the tuning parameter and a tradeoff between $\ell_1$norms and small
compatibility constants are possible. This implies, in particular for
correlated design, improved bounds for the prediction error of the Lasso
estimator as compared to the methods based on restricted eigenvalue or
compatibility conditions only.