• ### TBD: Benchmarking and Analyzing Deep Neural Network Training(1803.06905)

April 14, 2018 cs.LG, stat.ML
The recent popularity of deep neural networks (DNNs) has generated a lot of research interest in performing DNN-related computation efficiently. However, the primary focus is usually very narrow and limited to (i) inference -- i.e. how to efficiently execute already trained models and (ii) image classification networks as the primary benchmark for evaluation. Our primary goal in this work is to break this myopic view by (i) proposing a new benchmark for DNN training, called TBD (TBD is short for Training Benchmark for DNNs), that uses a representative set of DNN models that cover a wide range of machine learning applications: image classification, machine translation, speech recognition, object detection, adversarial networks, reinforcement learning, and (ii) by performing an extensive performance analysis of training these different applications on three major deep learning frameworks (TensorFlow, MXNet, CNTK) across different hardware configurations (single-GPU, multi-GPU, and multi-machine). TBD currently covers six major application domains and eight different state-of-the-art models. We present a new toolchain for performance analysis for these models that combines the targeted usage of existing performance analysis tools, careful selection of new and existing metrics and methodologies to analyze the results, and utilization of domain specific characteristics of DNN training. We also build a new set of tools for memory profiling in all three major frameworks; much needed tools that can finally shed some light on precisely how much memory is consumed by different data structures (weights, activations, gradients, workspace) in DNN training. By using our tools and methodologies, we make several important observations and recommendations on where the future research and optimization of DNN training should be focused.
• ### Relativistic asymmetries in the galaxy cross-correlation function(1709.07854)

We study the asymmetry in the two-point cross-correlation function of two populations of galaxies focusing in particular on the relativistic effects that include the gravitational redshift. We derive the cross-correlation function on small and large scales using two different approaches: General Relativistic and Newtonian perturbation theory. Following recent work by Bonvin et al., Gaztanaga et al. and Croft, we calculate the dipole and the shell estimator with the two procedures and we compare our results. We find that while General Relativistic Perturbation Theory (GRPT) is able to make predictions of relativistic effects on very large, obviously linear scales (r > 50 Mpc/h), the presence of non-linearities physically occurring on much smaller scales (down to those describing galactic potential wells) can strongly affect the asymmetry estimators. These can lead to cancellations of the relativistic terms, and sign changes in the estimators on scales up to r ~ 50 Mpc/h. On the other hand, with an appropriate non-linear gravitational potential, the results obtained using Newtonian theory can successfully describe the asymmetry on smaller, non-linear scales (r < 20 Mpc/h) where gravitational redshift is the dominant term. On larger scales the asymmetry is much smaller in magnitude, and measurement is not within reach of current observations. This is in agreement with the observational results obtained by Gaztnaga et al. and the first detection of relativistic effects (on (r < 20 Mpc/h) scales) by Alam et al.
• ### Relativistic distortions in the large-scale clustering of SDSS-III BOSS CMASS galaxies(1709.07855)

General relativistic effects have long been predicted to subtly influence the observed large-scale structure of the universe. The current generation of galaxy redshift surveys have reached a size where detection of such effects is becoming feasible. In this paper, we report the first detection of the redshift asymmetry from the cross-correlation function of two galaxy populations which is consistent with relativistic effects. The dataset is taken from the Sloan Digital Sky Survey DR12 CMASS galaxy sample, and we detect the asymmetry at the $2.7\sigma$ level by applying a shell-averaged estimator to the cross-correlation function. Our measurement dominates at scales around $10$ h$^{-1}$Mpc, larger than those over which the gravitational redshift profile has been recently measured in galaxy clusters, but smaller than scales for which linear perturbation theory is likely to be accurate. The detection significance varies by 0.5$\sigma$ with the details of our measurement and tests for systematic effects. We have also devised two null tests to check for various survey systematics and show that both results are consistent with the null hypothesis. We measure the dipole moment of the cross-correlation function, and from this the asymmetry is also detected, at the $2.8 \sigma$ level. The amplitude and scale-dependence of the clustering asymmetries are approximately consistent with the expectations of General Relativity and a biased galaxy population, within large uncertainties. We explore theoretical predictions using numerical simulations in a companion paper.
• ### Relativistic Effects on Galaxy Redshift Samples due to Target Selection(1709.07856)

In a galaxy redshift survey the objects to be targeted for spectra are selected from a photometrically observed sample. The observed magnitudes and colours of galaxies in this parent sample will be affected by their peculiar velocities, through relativistic Doppler and relativistic beaming effects. In this paper we compute the resulting expected changes in galaxy photometry. The magnitudes of the relativistic effects are a function of redshift, stellar mass, galaxy velocity and velocity direction. We focus on the CMASS sample from the Sloan Digital Sky Survey (SDSS), Baryon Oscillation Spectroscopic Survey (BOSS), which is selected on the basis of colour and magnitude. We find that 0.10\% of the sample ($\sim 585$ galaxies) has been scattered into the targeted region of colour-magnitude space by relativistic effects, and conversely 0.09\% of the sample ($\sim 532$ galaxies) has been scattered out. Observational consequences of these effects include an asymmetry in clustering statistics, which we explore in a companion paper. Here we compute a set of weights which can be used to remove the effect of modulations introduced into the density field inferred from a galaxy sample. We conclude by investigating the possible effects of these relativistic modulation on large scale clustering of the galaxy sample.
• ### N-body simulations of gravitational redshifts and other relativistic distortions of galaxy clustering(1709.07859)

Large redshift surveys of galaxies and clusters are providing the first opportunities to search for distortions in the observed pattern of large-scale structure due to such effects as gravitational redshift. We focus on non-linear scales and apply a quasi-Newtonian approach using N-body simulations to predict the small asymmetries in the cross-correlation function of two galaxy different populations. Following recent work by Bonvin et al., Zhao and Peacock and Kaiser on galaxy clusters, we include effects which enter at the same order as gravitational redshift: the transverse Doppler effect, light-cone effects, relativistic beaming, luminosity distance perturbation and wide-angle effects. We find that all these effects cause asymmetries in the cross-correlation functions. Quantifying these asymmetries, we find that the total effect is dominated by the gravitational redshift and luminosity distance perturbation at small and large scales, respectively. By adding additional subresolution modelling of galaxy structure to the large-scale structure information, we find that the signal is significantly increased, indicating that structure on the smallest scales is important and should be included. We report on comparison of our simulation results with measurements from the SDSS/BOSS galaxy redshift survey in a companion paper.